Displaying device multi-path statistics on Linux hosts


Linux provides a number of useful utilities for viewing I/O statistics. Atop, iostat, iotop, etc. are extremely useful utilities but they don’t allow you to easily see I/O statistics for each path to a mapper device. To illustrate this lets say you have a mapper device named mpatha with four paths (represented by sdcl, sdnf, sdeq and sdpk in the output below):

$ multipath -ll mpatha

mpatha (12345456789) dm-95 MYARRAY, FastBizzle
size=4.0T features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 1:0:1:25 sdcl 69:144 active ready running
|- 3:0:2:25 sdnf 71:272 active ready running
|- 1:0:2:25 sdeq 129:32 active ready running
`- 3:0:3:25 sdpk 130:416 active ready running

To visualize the amount of I/O traversing down each path to a mapper device we need to combine iostat and grep:

$ iostat -xN 1 | egrep '(mpatha|sdcl|sdnf|sdeq|sdpk)'

sdcl 0.00 0.00 0.20 0.04 2.06 7.35 39.19 0.00 0.88 0.56 2.57 0.86 0.02
sdnf 0.00 0.00 0.20 0.04 2.43 7.09 39.66 0.00 0.81 0.45 2.68 0.79 0.02
sdeq 0.00 0.00 0.20 0.04 1.99 7.06 37.68 0.00 0.90 0.59 2.57 0.88 0.02
sdpk 0.00 0.00 0.20 0.04 1.93 7.03 37.33 0.00 0.79 0.45 2.63 0.77 0.02
mpatha 0.06 0.00 0.52 0.15 5.96 28.53 51.28 0.00 0.98 0.49 2.67 0.86 0.06

This solution requires you to look up the device you want to monitor and the default output is terrible! I wanted a cleaner solution so I wrote mpathstat. Mpathstat parses ‘multipath -ll’ to get the mapper devices on a system as well as the block devices that represent the paths to this device. The script then iterates over an iostat sample and combines everything in one location:

$ mpathstat.py 1

Device Name Reads Writes KBytesR/S KBytesW/S Await
mpatha 0.00 44.00 0.00 912.00 0.82
|- sdcj 0.00 11.00 0.00 256.00 0.73
|- sdpi 0.00 11.00 0.00 224.00 0.91
|- sdeo 0.00 11.00 0.00 192.00 0.73
|- sdnd 0.00 11.00 0.00 240.00 0.91
mpathb 0.00 3.00 0.00 384.00 1.00
|- sdmv 0.00 1.00 0.00 128.00 1.00
|- sdcb 0.00 0.00 0.00 0.00 0.00
|- sdpa 0.00 1.00 0.00 128.00 1.00
|- sdeg 0.00 1.00 0.00 128.00 1.00

This has helped me track down a couple of issues with one of our Enterprise storage arrays. If you make any enhancements to the script please send me a pull request on github!

This article was posted by Matty on 2016-10-24 19:12:00 -0400 EDT