I mentioned in my previous post creating clustered file systems with glusterfs that gluster self-healing has a few issues. To see what I’m referring to we can access a gluster client and cp a bunch of data to the glusterfs file system located at /gluster:
$ cd /gluster
$ mkdir etc
$ cd etc
$ cp -rp /etc/* .
Once I started the copy operation I immediately reboot a brick to stop I/O from going to it. When the box came back up there was a discrepancy between the contents of the two bricks:
Gluster brick #1:
$ pwd
/gluster/vol01/etc
[root@fedora-cluster01 etc]# **du -sh . ..
35M .
Gluster brick #2:
$ pwd
/gluster/vol01/etc
[root@fedora-cluster02 etc]# **du -sh . ..
4.6M .
Bizarre, ey? This isn’t fixed in due time, you need to follow the instructions in the official documentation and stat the contents of the file system to make them consistent again:
$ find /gluster -noleaf -print0 | xargs --null stat >/dev/null
This return things to normal on the brick that was rebooted:
$ pwd
/gluster/vol01/etc
$ du -sh .
35M .
There are numerous reasons you could encounter this situation. A server could panic and reboot. You may need to reboot the server to apply updates. Or you may need to perform hardware maintenance (disk replacement, rack maintenance, etc.) which requires the server to be offline for a while. Jeff Darcy posted a nice reply to my original gluster post, and it looks like he’s in the midst of working on a permanent fix for this. It will surely be welcomed from folks who want to put gluster into production. :)