I’ve been spending a bunch of time with Linux lately, and have found some really nifty tools to help me better manage the systems I support. One of these tools is blktrace, which allows you to view block I/O operations to the devices connected to your system. To see just how useful this utility is, I present to you with the following example:
$ blktrace -d /dev/sda -o - | blkparse -i -
8,0 0 1 0.000000000 783 A W 36744519 + 8 <- (8,1) 36744456
8,0 0 2 0.000000472 783 Q W 36744519 + 8 [kjournald]
8,0 0 3 0.000012413 783 G W 36744519 + 8 [kjournald]
8,0 0 4 0.000016614 783 P N [kjournald]
8,0 0 5 0.000018272 783 I W 36744519 + 8 [kjournald]
8,0 0 6 0.000026492 783 A W 36744527 + 8 <- (8,1) 36744464
8,0 0 7 0.000026784 783 Q W 36744527 + 8 [kjournald]
8,0 0 8 0.000028877 783 M W 36744527 + 8 [kjournald]
8,0 0 9 0.000031539 783 A W 36744535 + 8 <- (8,1) 36744472
8,0 0 10 0.000031752 783 Q W 36744535 + 8 [kjournald]
8,0 0 11 0.000032234 783 M W 36744535 + 8 [kjournald]
8,0 0 12 0.000033151 783 A W 36744543 + 8 <- (8,1) 36744480
8,0 0 13 0.000033343 783 Q W 36744543 + 8 [kjournald]
8,0 0 14 0.000033758 783 M W 36744543 + 8 [kjournald]
8,0 0 15 0.000034477 783 A W 36744551 + 8 <- (8,1) 36744488
8,0 0 16 0.000034670 783 Q W 36744551 + 8 [kjournald]
8,0 0 17 0.000035082 783 M W 36744551 + 8 [kjournald]
8,0 0 18 0.000035786 783 A W 36744559 + 8 <- (8,1) 36744496
8,0 0 19 0.000035972 783 Q W 36744559 + 8 [kjournald]
8,0 0 20 0.000036381 783 M W 36744559 + 8 [kjournald]
8,0 0 21 0.000037124 783 A W 36744567 + 8 <- (8,1) 36744504
^CCPU0 (8,0):
Reads Queued: 0, 0KiB Writes Queued: 13, 52KiB
Read Dispatches: 0, 0KiB Write Dispatches: 2, 52KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 0, 0KiB Writes Completed: 2, 52KiB
Read Merges: 0, 0KiB Write Merges: 11, 44KiB
Read depth: 0 Write depth: 1
IO unplugs: 2 Timer unplugs: 0
Throughput (R/W): 0KiB/s / 52000KiB/s
Events (8,0): 49 entries
Skips: 0 forward (0 - 0.0%)
As you can see from the above output, each I/O is printed along with a summary of the operations and how they were processed by the I/O scheduler. This is super information, which can be used to figure out I/O patterns (random reads, random writes, etc.), the size of the I/O operations hitting physical devices in a system and the type of workload on a system. I used to think that Solaris (w/ DTrace) had a serious leg up on Linux, but I’m starting to realize that you can understand Linux performance just as well if you install one or more third party tools. I’ve come across a slew of nifty utilities over the past month, and will start posting regularly with example output and links to each utility.