I recently needed to figure out what process was generating a bunch of I/O requests on a Linux system. On Solaris, there are a ton of tools available in the DTraceToolkit that can pin down i/o performance consumers.
Using iostat in Linux, I can see that the drives are spinning like mad, but I really want to know which process on the machine is driving the disk. Using collectl, we can view this. The collectl utility has a “top” like function, which you can direct separate subsystems at.
$ collectl --showtopopts
The following is a list of --top's sort types which apply to either
process or slab data. In some cases you may be allowed to sort
by a field that is not part of the display if you so desire
TOP PROCESS SORT FIELDS
Memory
vsz virtual memory
rss resident (physical) memory
Time
syst system time
usrt user time
time total time
I/O
rkb KB read
wkb KB written
iokb total I/O KB
rkbc KB read from pagecache
wkbc KB written to pagecache
iokbc total pagecacge I/O
ioall total I/O KB (iokb+iokbc)
rsys read system calls
wsys write system calls
iosys total system calls
iocncl Cancelled write bytes
Page Faults
majf major page faults
minf minor page faults
flt total page faults
Miscellaneous (best when used with --procfilt)
cpu cpu number
pid process pid
thread total process threads (not counting main)
TOP SLAB SORT FIELDS
numobj total number of slab objects
actobj active slab objects
objsize sizes of slab objects
numslab number of slabs
objslab number of objects in a slab
totsize total memory sizes taken by slabs
totchg change in memory sizes
totpct percent change in memory sizes
name slab names
So, i’m really interested in total I/O KB per process.
$ collectl --top iokb
waiting for 1 second sample...
# TOP PROCESSES sorted by iokb (counters are /sec) 15:52:17
# PID User PR PPID THRD S VSZ RSS CP SysT UsrT Pct
AccuTime RKB WKB MajF MinF Command
3751 mysql 15 3698 25 S 2G 2G 6 0.00 0.02 2
88:09.79 96 12 0 3 /usr/libexec/mysqld
Cool, so MySQL is driving the I/O subsystem. This is a pretty cool utility, and even more documentation on the project’s website here.