Getting the number of bytes read and written by your Linux NFS kernel threads (nfsd)

Linux NFS server implementations export a number of statistics through the /proc file system. The nfsstat utility can parse this file and display various performance counters, and the data that is displayed comes from the /proc/net/rpc/nfsd proc entry:

$ cat /proc/net/rpc/nfsd

rc 0 2585 290
fh 0 0 0 0 0
io 1069882 10485760
th 8 0 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
ra 32 16 0 0 0 0 0 0 0 0 0 1
net 2880 0 2865 14
rpc 2875 0 0 0 0
proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 6 50 1 16 37 0 0 2560 13 1 0 0 10 0 0 0 0 16 3 6 3 10
proc4 2 3 124
proc4ops 59 0 0 0 12 2 0 0 0 0 88 17 0 0 0 0 12 0 0 4 0 2 0 104 0 7 17 1 0 0 0 11 2 4 0 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0

So what do these values mean? To figure this out we can surf over to fs/nfsd/stats.c in the kernel source:

* Format:
 *      rc   
 *                      Statistsics for the reply cache
 *      fh     
 *                      statistics for filehandle lookup
 *      io  
 *                      statistics for IO throughput
 *      th   <10%-20%> <20%-30%> ... <90%-100%> <100%> 
 *                      time (seconds) when nfsd thread usage above thresholds
 *                      and number of times that all threads were in use
 *      ra cache-size  <10%  <20%  <30% ... <100% not-found
 *                      number of times that read-ahead entry was found that deep in
 *                      the cache.
 *      plus generic RPC stats (see net/sunrpc/stats.c)

and then read through the nfsstat manual pages. While debugging an NFS issue a few weeks back I noticed that nfsstat doesn't have an option to print the number of bytes read and written (there is an io line in the output above, but for some reason nfsstat doesn't process it). The io line along with the number of reads and writes is useful for understanding how many VFS write operations are performed and the average size of these operations. To help me understand the workload pattern of the nfsd threads on one of my systems I developed the nfsrwstat.sh script:

$ nfsrwstat.sh

Date       Bytes_Read Bytes_Written
19:55:02   2139764    25187130  
19:55:07   0          3145728   
19:55:12   0          3358720   
19:55:18   0          3497984   
19:55:23   0          3629056   
19:55:28   0          3145728   

This script will help you better understand how much I/O your NFS server threads are performing, and you should be able to figure out if you have a mixed workload or a workload that is read or write intensive. When you compare this data along side the output from iostat you can start to see how the block I/O layer and I/O scheduler are handling the data as it's pushed down the stack . If you haven't read the NFS RFC of the Linux NFS FAQ I would highly recommend you do so. There is a plethora of information in those two documents, and it will make managing NFS a breeze!

2 thoughts on “Getting the number of bytes read and written by your Linux NFS kernel threads (nfsd)”

  1. what is the option -if ? when I run it, it wants to run against eth3, i’d like it to run on another iface. the only way I can get it going is using the -allif

Leave a Reply

Your email address will not be published. Required fields are marked *