Most admins have probobably encountered a situation where someone says “hey this really bizarre thing <fill_in_the_blank> is occurring.” Whenever I am approached to look at these types of issues I will typically start by jumping on my systems and reviewing system, network and performance data. Once I’ve verified those are within normal levels I will begin reviewing the client server communications to see what is going on.
I’ve encountered some NFS issues that fit into the “bizarre” category, and it’s amazing how much information you can glean by reviewing the NFS traffic between the client and server. I like to visualize problems, so I will frequently grab the network traffic with tcpdump:
$ tcpdump -w /var/tmp/nfs.dmp -i eth0 host 192.168.56.102 and not port 22
Re-create the issue and then feed the resulting dump into wireshark:
$ wireshark nfs.dmp
Wireshark will decode all of the NFS procedures for you, which makes comparing the captured packet against the pertinent RFC super super easy. The following picture shows how wireshark decodes an NFS WRITE request from a client:
And the response sent back from the server:
I’m not sure how NFS debugging can get any easier, and the capabilities built into wireshark are absolutely amazing! I reach for it weekly, and if you need to debug NFS protocol issues wireshark will soon become your best friend!