Archive
Posts in Storage
Creating clustered file systems with glusterfs on CentOS and Fedora Linux servers
I've been using gluster for the past few months, and so far I am really impressed with what I'm seeing. For those that haven't used gluster, it is an open source clustered file system that can provides scalable storage on commodity hardware. As with all file systems and applications, gluster comes with it's own vernacular. Here are some terms you will need to know if you are going to gluster it up: brick - a unit of storage which consists of a server and directory path (i.e., server:/export) translator - modules that are chained together to move data from point a to point b volume - a collection of bricks At the simplest level, a brick contains the name of a server and a directory on that server where stuff will be stored…
$ read more →Defragmenting EXT4 file systems with e4defrag (coming soon to a distribution near you)
If you have been around the systems engineering field you have probably read about file system fragmentation at some point. This typically occurs when files are randomly updated over time, and the blocks that comprise the file get scattered over different areas of the disk. This causes the drives to perform more work since the drive heads have to perform additional seeks vs. being able to sequentially read data off of a given platter…
$ read more →Using rpcdebug to debug Linux NFS client and server issues
Debugging NFS issues can sometimes be a chore, especially when you are dealing with busy NFS servers. Tools like nfswatch and nfsst at can help you understand what operations your server is servicing, but sometimes you need to get into the protocol guts to find out more info. There are a couple of ways you can do this. First you can capture NFS traffic with tcpdump and use the NFS protocol decoding features that are built into wireshark…
$ read more →Using wireshark's protocol decoding to debug NFS problems
Most admins have probobably encountered a situation where someone says "hey this really bizarre thing is occurring." Whenever I am approached to look at these types of issues I will typically start by jumping on my systems and reviewing system, network and performance data. Once I've verified those are within normal levels I will begin reviewing the client server communications to see what is going on. I've encountered some NFS issues that fit into the "bizarre" category, and it's amazing how much information you can glean by reviewing the NFS traffic between the client and server. I like to visualize problems, so I will frequently grab the network traffic with tcpdump: Re-create the issue and then feed the resulting dump into wireshark: Wireshark will decode all of the NFS procedures for you, which makes comparing the captured packet against the pertinent RFC super super easy…
$ read more →Getting the number of bytes read and written by your Linux NFS kernel threads (nfsd)
Linux NFS server implementations export a number of statistics through the /proc file system. The nfsstat utility can parse this file and display various performance counters, and the data that is displayed comes from the /proc/net/rpc/nfsd proc entry: So what do these values mean? To figure this out we can surf over to fs/nfsd/stats.c in the kernel source: and then read through the nfsstat manual pages. While debugging an NFS issue a few weeks back I noticed that nfsstat doesn't have an option to print the number of bytes read and written (there is an io line in the output above, but for some reason nfsstat doesn't process it)…
$ read more →