Add information on stat, vmstat and strace commands

shell_tools_101.rst

@@ -244,17 +244,102 @@ Further documentation on either can be found in the ``man`` pages.
 stat
 ----
 
-.. todo:: stat command
+``stat`` gives you information about a file, including the file size, what permissions the file has, and when the file was created, last modified, and last accessed.
+
+.. code-block:: console
+
+ $ stat /etc/passwd
+ File: /etc/passwd
+ Size: 3988 Blocks: 8 IO Block: 4096 regular file
+ Device: 1bh/27d Inode: 2370069 Links: 1
+ Access: (0644/-rw-r--r--) Uid: ( 0/ root) Gid: ( 0/ root)
+ Access: 2019-03-25 18:53:16.313335636 -0400
+ Modify: 2019-03-16 13:25:39.173332827 -0400
+ Change: 2019-03-16 13:25:39.179999493 -0400
+ Birth: -
+
+To use ``stat`` in a shell script, you can pass the ``--format=`` argument, which accepts ``%``-based escape sequences.
+For example, ``stat --format="%s %u"`` will print the file size in bytes and the UID of the owner of the file.
+A full list of supported escape sequences is in the man page, ``man stat``.
+
 
 vmstat
 ------
 
-.. todo:: vmstat command
+The ``vmstat`` command gives general statistics about the system.
+Running ``vmstat`` with no arguments will report statistics accumulated since system boot.
+You can run ``vmstat 1`` instead to report statistics each second.
+The first line of output will still be statistics since boot, but then it will print a new line each second, containing the average statistics since the last line of output.
+An example run on a mostly idle system might look like this (headers have been manually realigned):
+
+.. code-block:: console
+ $ vmstat 1
+ procs ------------memory----------- ---swap-- -----io---- -system-- ------cpu-----
+ r b swpd free buff cache si so bi bo in cs us sy id wa st
+ 1 0 435112 1694040 84688 1802748 2 6 171 232 191 142 22 10 67 0 0
+ 0 0 435112 1693340 84688 1806028 0 0 0 12 2977 3579 3 3 95 0 0
+ 0 0 435112 1694584 84688 1803984 0 0 0 0 2826 3052 2 2 96 0 0
+ 1 0 435112 1693836 84688 1803984 0 0 0 44 2062 2645 2 2 97 0 0
+
+For the processes section, the first number, under ``r`` is the number of runnable processes.
+
+For the memory section section, the ``swpd`` column refers to how much total virtual memory is in use by the system.
+``free`` refers to the amount of free memory, and ``cache`` refers to the amount of memory being used for filesystem caching, which can be reclaimed if free memory gets too low.
+
+Under the swap section, ``si`` and ``so`` display the number of pages swapped in from the disk and swapped out to the disk, per second.
+Lots of swapping will seriously decrease performance, and indicates that there is not enough physical memory available on the system.
+
+Under ``io``, the ``bi`` and ``bo`` columns list the average number of blocks read and written to the filesystem per second.
+
+Under ``system``, the ``in`` and ``cs`` columns list the average number of interrupts per second and context switches per second.
+
+Under ``cpu``, all values are percentages of CPU time.
+The ``us`` section is the percentage of time spent running userspace code.
+The ``sy`` section is the percentage of time spent running kernel code.
+``id`` lists time spent idle, and ``wa`` lists time spent waiting for IO to complete.
+
 
 strace
 ------
 
-.. todo:: strace command
+``strace`` is a command that can be used to monitor which syscalls a process is executing.
+You can run a program under strace by passing the program name and all of its arguments to strace, like ``strace ping google.com``.
+You can also attach strace to a running program by calling ``strace -p`` with a process ID.
+WARNING: This will seriously degrade performance, by as much as a 20x slowdown.
+
+``strace`` generates a lot of output, even for simple programs.
+You can log strace's output to a file instead of the terminal by running ``-o`` followed by the name of file to create.
+You can also limit the syscalls that ``strace`` will look for by passing a comma seperated list to ``-e``, for example, ``strace -e openat,write,read,close ls``
+
+``strace`` also truncates strings that it feels are too long to print to the console.
+If you need the full contents of what your program is reading or writing, pass ``-s`` and a length, such as ``-s 9000``
+
+An example ``strace`` output of a simple statically linked hello world program in C is included below:
+
+.. code-block:: console
+ execve("./hello_world", ["./hello_world"], 0x7ffcadb2ed00 /* 90 vars */) = 0
+ arch_prctl(0x3001 /* ARCH_??? */, 0x7ffcea3e3080) = -1 EINVAL (Invalid argument)
+ brk(NULL) = 0x1079000
+ brk(0x107a1c0) = 0x107a1c0
+ arch_prctl(ARCH_SET_FS, 0x1079880) = 0
+ uname({sysname="Linux", nodename="Cyprus", ...}) = 0
+ readlink("/proc/self/exe", "/home/niles/hello_world", 4096) = 23
+ brk(0x109b1c0) = 0x109b1c0
+ brk(0x109c000) = 0x109c000
+ fstat(1, {st_mode=S_IFCHR|0600, st_rdev=makedev(0x88, 0x4), ...}) = 0
+ write(1, "Hello, world\n", 13) = 13
+ exit_group(0) = ?
+ +++ exited with 0 +++
+
+
+Syscalls and their arguments appear on the left, and the return value of the syscall apears on the right after the equals sign.
+If the syscall doesn't return, like ``execve`` or ``exit_group``, it is printed as a question mark.
+Common syscalls to recognize and look for when troubleshooting a program are ``open``, ``close``, ``read``, and ``write`` for files and sockets; ``fork`` and ``execve`` for creating subprocesses; and ``connect``, ``sendto``, and ``recvfrom`` for network traffic.
+If you see a syscall you don't recognize, it has a ``man`` page in section ``2``.
+So for example, to see more information about ``fstat``, including what arguments it takes and what it returns, you can do ``man 2 fstat``.
+
+As mentioned above, ``strace`` incurs a significant performance overhead.
+``perf trace`` uses a newer tracing system that may be much faster, but can miss some events. 
 
 ulimit
 ------