Yet another good reason to switch to NFSv4!

There are currently two main NFS protocol versions in mainstream use. The first is version 3, which was introduced in 1995 as part of RFC 1813. NFSv3 implementations use separate daemons to implement the locking (rpc.lockd), status (rpc.statd), mount (rpc.mountd) and port assignments (portmap) features required by the protocol. NFSv4, which was introduced in 2003 as part of RFC 3530, takes a different approach. The locking and stat functions are now part of the core NFS implementation, so you no longer need to ensure that the portmap, rpc.lockd and rpc.statd daemons are running on your Linux NFSv4 clients and servers. There is a blurb regarding this in the Redhat Linux 5.5 administration guide:

Because protocol support has been incorporated into the v4 protocol, NFSv4 has no interaction with the portmap, rpc.lockd, and rpc.statd daemons. NFSv4 listens on the well-known TCP port 2049, which eliminates the need for portmap interaction. The mounting and locking protocols have been incorporated into the V4 protocol which eliminates the need for interaction with rpc.lockd and rpc.statd. The rpc.mountd daemon is still required on the server, but is not involved in any over-the-wire operations.

This is an interesting tidbit that I wasn’t aware of, and I’m learning a lot of new things as I dig through some of the NFS implementation details. If you are interested in learning how NFS truly works, check out the NFSv3 and NFSv4 RFCs! You will be glad you did!!

Documentation on the Solaris NFS client and server implementations

My good friend Clay sent me a link to Vallish Guru’s blog, which contains a number of PDFs that describe the Solaris NFS client and server implementations. I read through the NFSv3 RFC and Brent Callaghan’s awesome book on NFS a while back, and both of these resources came in handy while I was reading through some of the PDFs. The fact that NFS has been alive for over 20 years amazes me, and it’s awesome to see that the NFS folks are taking it in new and interesting directions.

Monitoring NFS operations with nfswatch

I support a number of NFS clients, and periodically need to see which types of NFS operations are being performed. The nfsstat utility works pretty well for this, but sometimes I want to get a broader view of what is going on. When these situations arise, I like to fire up the nfswatch utility which displays network traffic along with a listing of NFS operations:

$ nfswatch

monty         Tue May 12 20:47:21 2009   Elapsed time:   00:01:29
Interval packets:        34 (network)         34 (to host)          0 (dropped)
Total packets:         4482 (network)       4482 (to host)          0 (dropped)
                     Monitoring packets from interface bge0
                     int   pct    total                      int   pct    total
NFS3 Read             22   65%     3459 TCP Packets           33   97%     4292
NFS3 Write             5   15%      422 UDP Packets            0    0%       21
NFS Read               0    0%        0 ICMP Packets           1    3%       62
NFS Write              0    0%        0 Routing Control        0    0%        0
NFS Mount              0    0%        0 Addr Resolution        0    0%      107
Port Mapper            0    0%        1 Rev Addr Resol         0    0%        0
RPC Authorization      0    0%        0 Ether/FDDI Bdcst       0    0%      107
Other RPC Packets      0    0%        0 Other Packets          0    0%        0
                                22 NFS Procedures
 Procedure           int   pct   total  completed  avg(msec)  std dev   max resp
  ACCESS               4   15%      284    
  CREATE               0    0%        0    
  GETATTR             18   67%     2991    
  LINK                 0    0%        0    
  LOOKUP               0    0%      149    
  MKDIR                0    0%        0    
  MKNOD                0    0%        0         
  NULLPROC             0    0%        0
  READ                 0    0%       21
  READDIR              0    0%       12
  READDIRPLUS          0    0%        2
  READLINK             0    0%        0
  REMOVE               0    0%        0
  RENAME               0    0%        0
  RMDIR                0    0%        0
  SETATTR              5   19%      422
  SYMLINK              0    0%        0
nfswatch> 



This is an awesome tool, and runs awesome on both Linux and Solaris hosts.

Interesting NFSv3 errors

While reading through RFC 1813 (NFSv3 RFC), I came across the following interesting NFS error:

NFS3ERR_JUKEBOX

     The server initiated the request, but was not able to
     complete it in a timely fashion. The client should wait
     and then try the request with a new RPC transaction ID.
     For example, this error should be returned from a server
     that supports hierarchical storage and receives a request
     to process a file that has been migrated. In this case,
     the server should start the immigration process and
     respond to client with this error.

This is nifty, and leads me to wonder if any NFS-based HSM solutions are utilizing this.

Viewing NFS client operations

While reading through the NFS RFCs this week, I decided to write nfsclientstats.pl to view NFS operations by process:

$ nfsclientstats.pl

process    read write readdir getattr setattr lookup access create remove rename mkdir rmdir
mkdir         0     0       0     380       0    190      0      0      0      0   190     0
mv            0     0       0     189       0   1890   2079      0      0    189     0     0
orca       3328   194       0    5496       6   6882   8246     12      0      0     0     0
rm            0     0     760     950       0   2850   5320      0    190      0     0   190
touch         0     0       0     378     189   1512   1323    189      0      0     0     0

process    read write readdir getattr setattr lookup access create remove rename mkdir rmdir
mkdir         0     0       0     386       0    193      0      0      0      0   193     0
mv            0     0       0     195       0   1950   2145      0      0    195     0     0
orca       3110   169       0    8312      22  10434  12476     44      0      0     0     0
rm            0     0     780     975       0   2925   5460      0    195      0     0   195
touch         0     0       0     388     194   1552   1358    194      0      0     0     0

The script uses the DTrace FBT provider, and produces output that is greater than 80-characters. If you are an 80-column purist, you might want to refrain from downloading the script. ;) Hope folks find this useful!