I recently needed to figure out what process was generating a bunch of I/O requests on a Linux system. On Solaris, there are a ton of tools available in the DTraceToolkit that can pin down i/o performance consumers.
Using iostat in Linux, I can see that the drives are spinning like mad, but I really want to know which process on the machine is driving the disk. Using collectl, we can view this. The collectl utility has a “top” like function, which you can direct separate subsystems at.
The following is a list of --top's sort types which apply to either process or slab data. In some cases you may be allowed to sort by a field that is not part of the display if you so desire TOP PROCESS SORT FIELDS Memory vsz virtual memory rss resident (physical) memory Time syst system time usrt user time time total time I/O rkb KB read wkb KB written iokb total I/O KB rkbc KB read from pagecache wkbc KB written to pagecache iokbc total pagecacge I/O ioall total I/O KB (iokb+iokbc) rsys read system calls wsys write system calls iosys total system calls iocncl Cancelled write bytes Page Faults majf major page faults minf minor page faults flt total page faults Miscellaneous (best when used with --procfilt) cpu cpu number pid process pid thread total process threads (not counting main) TOP SLAB SORT FIELDS numobj total number of slab objects actobj active slab objects objsize sizes of slab objects numslab number of slabs objslab number of objects in a slab totsize total memory sizes taken by slabs totchg change in memory sizes totpct percent change in memory sizes name slab names
So, i’m really interested in total I/O KB per process.
collectl --top iokb
waiting for 1 second sample... # TOP PROCESSES sorted by iokb (counters are /sec) 15:52:17 # PID User PR PPID THRD S VSZ RSS CP SysT UsrT Pct AccuTime RKB WKB MajF MinF Command 3751 mysql 15 3698 25 S 2G 2G 6 0.00 0.02 2 88:09.79 96 12 0 3 /usr/libexec/mysqld
Cool, so MySQL is driving the I/O subsystem. This is a pretty cool utility, and even more documentation on the project’s website here.