Archive
Posts in Networking
Using rpcdebug to debug Linux NFS client and server issues
Debugging NFS issues can sometimes be a chore, especially when you are dealing with busy NFS servers. Tools like nfswatch and nfsst at can help you understand what operations your server is servicing, but sometimes you need to get into the protocol guts to find out more info. There are a couple of ways you can do this. First you can capture NFS traffic with tcpdump and use the NFS protocol decoding features that are built into wireshark…
$ read more →Using wireshark's protocol decoding to debug NFS problems
Most admins have probobably encountered a situation where someone says "hey this really bizarre thing is occurring." Whenever I am approached to look at these types of issues I will typically start by jumping on my systems and reviewing system, network and performance data. Once I've verified those are within normal levels I will begin reviewing the client server communications to see what is going on. I've encountered some NFS issues that fit into the "bizarre" category, and it's amazing how much information you can glean by reviewing the NFS traffic between the client and server. I like to visualize problems, so I will frequently grab the network traffic with tcpdump: Re-create the issue and then feed the resulting dump into wireshark: Wireshark will decode all of the NFS procedures for you, which makes comparing the captured packet against the pertinent RFC super super easy…
$ read more →Getting the number of bytes read and written by your Linux NFS kernel threads (nfsd)
Linux NFS server implementations export a number of statistics through the /proc file system. The nfsstat utility can parse this file and display various performance counters, and the data that is displayed comes from the /proc/net/rpc/nfsd proc entry: So what do these values mean? To figure this out we can surf over to fs/nfsd/stats.c in the kernel source: and then read through the nfsstat manual pages. While debugging an NFS issue a few weeks back I noticed that nfsstat doesn't have an option to print the number of bytes read and written (there is an io line in the output above, but for some reason nfsstat doesn't process it)…
$ read more →Good write up Linux consistent network device naming
In RHEL 6.1 the default names assigned to Dell server network interfaces changed from ethX to emX and pXpX. The new names describe where a network interface physically resides in the system, and will have the following format: emX the X (first, second, etc.) onboard interface pXpY PCI device X port Y Dell wrote a really good white paper on this, and the following text from the document summarizes how the pieces fit together: "A naming mechanism that can impart meaning to the network > interface‟s name based on the physical location of a network port in > concordance to the intended system design is necessary. To achieve that, > the system firmware has the ability to communicate the intended order > for network devices on the mother board to the Operating System via > standard mechanisms such as SMBIOS and ACPI. > > The new naming scheme uses 'biosdevname' udev helper utility , developed > by Dell and released under GPL, suggests new names based on the location > of the network adapters on the system as suggested by system BIOS." I like the new format, and this will definitely be a nice addition to hardware provisioning systems…
$ read more →Speeding up SSH (SCP) data transfers
I'll be the first to admit that I'm an SCP addict. It doesn't matter what kind of data I'm working with, if it can be turned into an object that I move around with scp I'm in! One thing I've always noticed with scp is the dismal out of the box performance. I read quite some time back on the openssh mailing list that there were some fixed buffers inside openssh that prevented copy operations from fully utilizing high speed network links…
$ read more →