Tracing block I/O operations on Linux hosts with blktrace

I’ve been spending a bunch of time with Linux lately, and have found some really nifty tools to help me better manage the systems I support. One of these tools is blktrace, which allows you to view block I/O operations to the devices connected to your system. To see just how useful this utility is, I present to you with the following example:

disarm:~# blktrace -d /dev/sda -o – | blkparse -i –

  8,0    0        1     0.000000000   783  A   W 36744519 + 8 <- (8,1) 36744456
  8,0    0        2     0.000000472   783  Q   W 36744519 + 8 [kjournald]
  8,0    0        3     0.000012413   783  G   W 36744519 + 8 [kjournald]
  8,0    0        4     0.000016614   783  P   N [kjournald]
  8,0    0        5     0.000018272   783  I   W 36744519 + 8 [kjournald]
  8,0    0        6     0.000026492   783  A   W 36744527 + 8 <- (8,1) 36744464
  8,0    0        7     0.000026784   783  Q   W 36744527 + 8 [kjournald]
  8,0    0        8     0.000028877   783  M   W 36744527 + 8 [kjournald]
  8,0    0        9     0.000031539   783  A   W 36744535 + 8 <- (8,1) 36744472
  8,0    0       10     0.000031752   783  Q   W 36744535 + 8 [kjournald]
  8,0    0       11     0.000032234   783  M   W 36744535 + 8 [kjournald]
  8,0    0       12     0.000033151   783  A   W 36744543 + 8 <- (8,1) 36744480
  8,0    0       13     0.000033343   783  Q   W 36744543 + 8 [kjournald]
  8,0    0       14     0.000033758   783  M   W 36744543 + 8 [kjournald]
  8,0    0       15     0.000034477   783  A   W 36744551 + 8 <- (8,1) 36744488
  8,0    0       16     0.000034670   783  Q   W 36744551 + 8 [kjournald]
  8,0    0       17     0.000035082   783  M   W 36744551 + 8 [kjournald]
  8,0    0       18     0.000035786   783  A   W 36744559 + 8 <- (8,1) 36744496
  8,0    0       19     0.000035972   783  Q   W 36744559 + 8 [kjournald]
  8,0    0       20     0.000036381   783  M   W 36744559 + 8 [kjournald]
  8,0    0       21     0.000037124   783  A   W 36744567 + 8 <- (8,1) 36744504

  < ..... >

^CCPU0 (8,0):
 Reads Queued:           0,        0KiB  Writes Queued:          13,       52KiB
 Read Dispatches:        0,        0KiB  Write Dispatches:        2,       52KiB
 Reads Requeued:         0               Writes Requeued:         0
 Reads Completed:        0,        0KiB  Writes Completed:        2,       52KiB
 Read Merges:            0,        0KiB  Write Merges:           11,       44KiB
 Read depth:             0               Write depth:             1
 IO unplugs:             2               Timer unplugs:           0

Throughput (R/W): 0KiB/s / 52000KiB/s
Events (8,0): 49 entries
Skips: 0 forward (0 -   0.0%)



As you can see from the above output, each I/O is printed along with a summary of the operations and how they were processed by the I/O scheduler. This is super information, which can be used to figure out I/O patterns (random reads, random writes, etc.), the size of the I/O operations hitting physical devices in a system and the type of workload on a system. I used to think that Solaris (w/ DTrace) had a serious leg up on Linux, but I’m starting to realize that you can understand Linux performance just as well if you install one or more third party tools. I’ve come across a slew of nifty utilities over the past month, and will start posting regularly with example output and links to each utility.

7 thoughts on “Tracing block I/O operations on Linux hosts with blktrace”

  1. Great find!! I’ll be using blktrace this week. I’m looking forward to what other tools you have come across.

  2. Thanks for the pointer! I’ve got a mix of solaris and linux, and i’m working on trying to understand the workload on one of my big linux farms at this point….hoping some flash disks might help with a bottleneck.

    Did you get my mail about the nfstrace ‘bug’?

  3. You probably want to look at seekwatcher that creates some nice plots/movies from blktrace.

  4. I have installed the package, whenever I execute the command you posted abovw, I have the following output:

    /sys/kernel/debug does not appear to be a debug filesystem

    Any clue?

  5. luqmanux, you need to mount the debug filesystem:

    mount -t debugfs debugfs /sys/kernel/debug

Leave a Reply

Your email address will not be published. Required fields are marked *