Ben Rockwood is awesome. If you haven’t had a chance to check out his blog, its a must read for any Solaris Admin.
He gave a presentation at the Open Storage Summit about ZFS. (Video Here)Its worth the read / view for some indepth ZFS concepts.
Joerg assisted in turningsome of this into plain English so the rest of us can understand.
Kudos to Ben for giving props to Matty in the pdf!
While transferring a slew of data to one of my Linux boxes a few weeks back, I was curious how many interrupts were being generated each second. To answer my question, I installed the itop utility. Itop displays the number of interrupts per second that occur per device, and provides output similar to top. Here is a sample run:
$ itop -a
INT NAME RATE MAX
0 [ timer] 0 Ints/s (max: 0)
1 [ i8042] 0 Ints/s (max: 0)
6 [ floppy] 0 Ints/s (max: 0)
7 [ parport0] 0 Ints/s (max: 0)
8 [ rtc0] 0 Ints/s (max: 0)
9 [ acpi] 0 Ints/s (max: 0)
12 [ i8042] 0 Ints/s (max: 0)
15 [ ide0] 0 Ints/s (max: 0)
16 [ ehci_hcd:usb2] 0 Ints/s (max: 0)
17 [ ioc0] 0 Ints/s (max: 24)
18 [ eth0] 1400 Ints/s (max: 1959)
19 [ uhci_hcd:usb1, ] 0 Ints/s (max: 0)
This is a useful tool, and one I want to remember for the future!
While perusing the catonmat blog, I came across a reference to the pv utility. Pv allows you to monitor the amount of data that is written to a pipe, which is incredibly useful for monitoring how much data is transferred between pipe endpoints. The following example shows this super useful tool in action:
$ dd if=/dev/zero | pv > foo
522MB 0:00:06 [ 109MB/s] [ <=> ]
When pv is added to the pipeline, you get a continuous display of the amount of data that is being transferred between two pipe endpoints. I really dig this utility, and I am stoked that I found the catonmat website! Niiiiiiiiice!
UPDATE:
Tryusing pv when sending stuff over the network using dd. Neato.
[root@machine2 ~]# ssh machine1 “dd if=/dev/VolGroup00/domU2migrate”|pv -s 8G -petr|dd of=/dev/xen02vg/domU2migrate 0:00:30 [11.2MB/s] [====> ] 4% ETA :10:13
Want to rate limit the transfer so you don’t flood the pipe?
-L RATE, --rate-limit RATE
Limit the transfer to a maximum of RATE bytes per second. A suffix of
"k", "m", "g", or "t" can be added to denote kilobytes (*1024),
megabytes, and so on.
-B BYTES, --buffer-size BYTES
Use a transfer buffer size of BYTES bytes. A suffix of "k", "m", "g",
or "t" can be added to denote kilobytes (*1024), megabytes, and so on.
The default buffer size is the block size of the input file’s filesystem
multiplied by 32 (512kb max), or 400kb if the block size cannot be
determined.
Already have a transfer in progress and want to rate limit it without restarting?
-R PID, --remote PID
If PID is an instance of pv that is already running, -R PID will cause
that instance to act as though it had been given this instance’s command
line instead. For example, if pv -L 123k is running with process ID
9876, then running pv -R 9876 -L 321k will cause it to start using a
rate limit of 321k instead of 123k. Note that some options cannot be
changed while running, such as -c, -l, and -f.
I read through the bash tips on the hacktux website, which brought to light the fact that you can do basic integer math in your bash scripts. This is easily accomplished by using dual parenthesis similar to this:
four=$(( 2 + 2 ))
echo $four
This is good stuff, and I need to replace some old bc ... and expr .. statements with this.
I’ve been spending a bunch of time with Linux lately, and have found some really nifty tools to help me better manage the systems I support. One of these tools is blktrace, which allows you to view block I/O operations to the devices connected to your system. To see just how useful this utility is, I present to you with the following example:
$ blktrace -d /dev/sda -o - | blkparse -i -
8,0 0 1 0.000000000 783 A W 36744519 + 8 <- (8,1) 36744456
8,0 0 2 0.000000472 783 Q W 36744519 + 8 [kjournald]
8,0 0 3 0.000012413 783 G W 36744519 + 8 [kjournald]
8,0 0 4 0.000016614 783 P N [kjournald]
8,0 0 5 0.000018272 783 I W 36744519 + 8 [kjournald]
8,0 0 6 0.000026492 783 A W 36744527 + 8 <- (8,1) 36744464
8,0 0 7 0.000026784 783 Q W 36744527 + 8 [kjournald]
8,0 0 8 0.000028877 783 M W 36744527 + 8 [kjournald]
8,0 0 9 0.000031539 783 A W 36744535 + 8 <- (8,1) 36744472
8,0 0 10 0.000031752 783 Q W 36744535 + 8 [kjournald]
8,0 0 11 0.000032234 783 M W 36744535 + 8 [kjournald]
8,0 0 12 0.000033151 783 A W 36744543 + 8 <- (8,1) 36744480
8,0 0 13 0.000033343 783 Q W 36744543 + 8 [kjournald]
8,0 0 14 0.000033758 783 M W 36744543 + 8 [kjournald]
8,0 0 15 0.000034477 783 A W 36744551 + 8 <- (8,1) 36744488
8,0 0 16 0.000034670 783 Q W 36744551 + 8 [kjournald]
8,0 0 17 0.000035082 783 M W 36744551 + 8 [kjournald]
8,0 0 18 0.000035786 783 A W 36744559 + 8 <- (8,1) 36744496
8,0 0 19 0.000035972 783 Q W 36744559 + 8 [kjournald]
8,0 0 20 0.000036381 783 M W 36744559 + 8 [kjournald]
8,0 0 21 0.000037124 783 A W 36744567 + 8 <- (8,1) 36744504
^CCPU0 (8,0):
Reads Queued: 0, 0KiB Writes Queued: 13, 52KiB
Read Dispatches: 0, 0KiB Write Dispatches: 2, 52KiB
Reads Requeued: 0 Writes Requeued: 0
Reads Completed: 0, 0KiB Writes Completed: 2, 52KiB
Read Merges: 0, 0KiB Write Merges: 11, 44KiB
Read depth: 0 Write depth: 1
IO unplugs: 2 Timer unplugs: 0
Throughput (R/W): 0KiB/s / 52000KiB/s
Events (8,0): 49 entries
Skips: 0 forward (0 - 0.0%)
As you can see from the above output, each I/O is printed along with a summary of the operations and how they were processed by the I/O scheduler. This is super information, which can be used to figure out I/O patterns (random reads, random writes, etc.), the size of the I/O operations hitting physical devices in a system and the type of workload on a system. I used to think that Solaris (w/ DTrace) had a serious leg up on Linux, but I’m starting to realize that you can understand Linux performance just as well if you install one or more third party tools. I’ve come across a slew of nifty utilities over the past month, and will start posting regularly with example output and links to each utility.