Linux NFS server implementations export a number of statistics through the /proc file system. The nfsstat utility can parse this file and display various performance counters, and the data that is displayed comes from the /proc/net/rpc/nfsd proc entry:
$ cat /proc/net/rpc/nfsd
rc 0 2585 290
fh 0 0 0 0 0
io 1069882 10485760
th 8 0 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000 0.000
ra 32 16 0 0 0 0 0 0 0 0 0 1
net 2880 0 2865 14
rpc 2875 0 0 0 0
proc2 18 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
proc3 22 6 50 1 16 37 0 0 2560 13 1 0 0 10 0 0 0 0 16 3 6 3 10
proc4 2 3 124
proc4ops 59 0 0 0 12 2 0 0 0 0 88 17 0 0 0 0 12 0 0 4 0 2 0 104 0 7 17 1 0 0 0 11 2 4 0 0 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0
So what do these values mean? To figure this out we can surf over to fs/nfsd/stats.c in the kernel source:
* Format:
* rc
* Statistsics for the reply cache
* fh
* statistics for filehandle lookup
* io
* statistics for IO throughput
* th <10%-20%> <20%-30%> ... <90%-100%> <100%>
* time (seconds) when nfsd thread usage above thresholds
* and number of times that all threads were in use
* ra cache-size <10% <20% <30% ... <100% not-found
* number of times that read-ahead entry was found that deep in
* the cache.
* plus generic RPC stats (see net/sunrpc/stats.c)
and then read through the nfsstat manual pages. While debugging an NFS issue a few weeks back I noticed that nfsstat doesn’t have an option to print the number of bytes read and written (there is an io line in the output above, but for some reason nfsstat doesn’t process it). The io line along with the number of reads and writes is useful for understanding how many VFS write operations are performed and the average size of these operations. To help me understand the workload pattern of the nfsd threads on one of my systems I developed the nfsrwstat.sh script:
$ nfsrwstat.sh
Date Bytes_Read Bytes_Written
19:55:02 2139764 25187130
19:55:07 0 3145728
19:55:12 0 3358720
19:55:18 0 3497984
19:55:23 0 3629056
19:55:28 0 3145728
This script will help you better understand how much I/O your NFS server threads are performing, and you should be able to figure out if you have a mixed workload or a workload that is read or write intensive. When you compare this data along side the output from iostat you can start to see how the block I/O layer and I/O scheduler are handling the data as it’s pushed down the stack . If you haven’t read the NFS RFC of the Linux NFS FAQ I would highly recommend you do so. There is a plethora of information in those two documents, and it will make managing NFS a breeze!