Using collectl on Linux to view system performance

I recently needed to figure out what process was generating a bunch of I/O requests on a Linux system. On Solaris, there are a ton of tools available in the DTraceToolkit that can pin down i/o performance consumers.

Using iostat in Linux, I can see that the drives are spinning like mad, but I really want to know which process on the machine is driving the disk. Using collectl, we can view this. The collectl utility has a “top” like function, which you can direct separate subsystems at.

$ collectl --showtopopts

The following is a list of --top's sort types which apply to either
process or slab data. In some cases you may be allowed to sort
by a field that is not part of the display if you so desire


vsz virtual memory
rss resident (physical) memory

syst system time
usrt user time
time total time

rkb KB read
wkb KB written
iokb total I/O KB

rkbc KB read from pagecache
wkbc KB written to pagecache
iokbc total pagecacge I/O
ioall total I/O KB (iokb+iokbc)

rsys read system calls
wsys write system calls
iosys total system calls

iocncl Cancelled write bytes

Page Faults
majf major page faults
minf minor page faults
flt total page faults

Miscellaneous (best when used with --procfilt)
cpu cpu number
pid process pid
thread total process threads (not counting main)


numobj total number of slab objects
actobj active slab objects
objsize sizes of slab objects
numslab number of slabs
objslab number of objects in a slab
totsize total memory sizes taken by slabs
totchg change in memory sizes
totpct percent change in memory sizes
name slab names

So, i’m really interested in total I/O KB per process.

$ collectl --top iokb

waiting for 1 second sample...

# TOP PROCESSES sorted by iokb (counters are /sec) 15:52:17
# PID  User     PR  PPID THRD S   VSZ   RSS CP  SysT  UsrT Pct 
AccuTime  RKB  WKB MajF MinF Command
3751  mysql    15  3698   25 S    2G    2G  6  0.00  0.02   2 
88:09.79   96   12    0    3 /usr/libexec/mysqld

Cool, so MySQL is driving the I/O subsystem. This is a pretty cool utility, and even more documentation on the project’s website here.

This article was posted by Matty on 2011-10-05 11:54:00 -0400 -0400