Remounting Linux read-only file systems read-write

I’ve been paged countless times to investigate systems that have gone off the air for one reason or another. Typically when look into these issues the server has crashed unexpectedly (bug, hardware, etc.) and rebooted in an attempt to get a fresh start on life. On occassion the box won’t boot due to a misconfiguration or inconsistent file system, and the server will be left with a read-only root file system and no network connectivity.

Fixing this is pretty easy. If possible I will add a 1 to the end of the boot line to boot the kernel into single user mode. If that works I can fix the issue and then allow the box to boot up normally. If the root file system is mounted read-only, I will use the mount command’s remount option to mount it read-write:

$ mount -o remount,rw /

Once you can write to the file system you should be able to write out changes to the file system to correct the issue that prevented the server from booting. Viva la remount!

Scanning for new LUNs on Linux servers

Storage management, the joys! Storage administration is one of those tasks that every admin does, and depending on your growth it may be something that consumes a lot of your time. If your servers are connected to a Fibre channel or iSCSI SAN, you probably need to periodically poke your systems to see new storage that is made available to them. There are several tools that can be used to do this. QLogic provides the scli utility and a rescan shell script, the sg3utils package comes with the rescan-scsi-bus.sh script and you can always roll your own script to do this.

I use a mix of all of the tools above. On older systems I use the QLogic scli utility, on servers that have Emulex adapters or recent Operating Systems I use rescan-scsi-bun.sh, and in other cases I use the following script to prod the sysfs scan and issue_lip entries directly:

#!/bin/bash

SLEEP_INTERVAL=300

echo "Scanning all fibre channel host adapters"

for i in `ls /sys/class/fc_host`
do
    echo "Rescanning /sys/class/fc_host/${i}:"

    echo "  Issuing a loop initialization on ${i}:" 
    echo "1" > /sys/class/fc_host/${i}/issue_lip

    echo "  Scanning ${i} for new devices:"
    echo "- - -" > "/sys/class/scsi_host/${i}/scan"

    echo "Sleeping for ${SLEEP_INTERVAL} seconds"
    sleep ${SLEEP_INTERVAL}
done

This is the nice thing about Linux. There is a tool to do just about everything, and if there isn’t one available you can usually cobble something together in short order to do what you need. Isn’t life as a Linux admin grand! :)

KVM makes its way to Illumos

I’ve been a fan of KVM for quite some time, and gave a talk on it a while back. The KVM design always made sense to me, and I tossed Xen out the window the minute I started using it. Today I was pleasantly surprised to see that several folks from Joyent ported KVM to Illumos! This is way cool, and I can’t wait to give this a spin! Hopefully libvirt integration and various goodies are there. :)

Update on my quest to find the perfect NAS device for home use

A few months back I started looking into NAS solutions that would be ideal for home use. I jotted down my initial research in the post building your own nas and the post making sense of the various nas solutions. My original intent was to purchase an HP microserver from Amazon and test out all of the freely available NAS solutions. Due to some time constraints I cancelled my microserver order and re-used a server I had at home. I’m still planning to order a microserver at some point, since they are killer boxes for building home NAS devices with.

I first loaded up openfiler and went to town creating devices and mapping them out to hosts on my network. If you haven’t used openfiler before, it’s a Linux-based NAS distribution that provides a graphical interface on top of a RHEL derived distribution. The fact that it runs Linux gave it a few cool points, but I was less than impressed with the graphical interface. Tasks such as creating logical volumes and shares weren’t as intuitive as I would have thought, and IMHO the GUI didn’t really provide all that much value over running the LVM utilities from the command line. It did manage all of the iSCSI, NFS and Samba work behind the scenes, which is kinda nice if you don’t want to dig into these services and see how they work.

Craving a bit more out of my NAS device, I wiped the server that I installed openfiler on and installed FreeNAS. For those who haven’t used FreeNAS, it is a FreeBSD-based NAS distribution that makes heavy use of ZFS. The FreeNAS installation process is about as painless as can be, and after a 10 minute install I was able to fire up the web interface and start poking around. I was impressed with the initial graphical interface, and after a few minutes of clicking around I had a working ZFS pool and a number of NFS and iSCSI targets provisioned. Everything seemed quite intuitive with the FreeNAS interface, and all of the options were in a place you would normally think to look.

I’m still using FreeNAS, though it doesn’t offer all of the items I would like in a true NAS appliance. Here are the items that would make FreeNAS a slam dunk for me:

– System and network service performance counters and graphs. These may be coming in 8.1.
– Built-in DLNA/UPnP support for streaming videos. This may be coming in 8.1.
– Better hardware monitoring and reporting.
– Energy conservation when the NAS device isn’t in use.
– Ability to act as a print server.

These things are relatively minor, and I suspect they will come in due time. Being a huge fan of ZFS I was stoked to see that this was part of FreeNAS. I’m curious to see how well the file system is supported now that Oracle has cut off the ZFS source code (I suspect it will thrive since the FreeBSD team has some crazy smart chaps working on it). As things stand now FreeNAS works, and it doesn’t cost you a penny to try out. I would still love to buy a synology diskstation to test out, but it’s kinda hard justifying one when what you have a working solution. It’s also nice to know that I can gain access to the FreeNAS source code anytime I want.

If anyone has read Gary Sim’s book Learning FreeNAS: Configure and manage a network attached storage solution please let me know what you thought of it in the comments section. I’m thinking about ordering a copy for a not so technically savy friend since his Youtube FreeNAS videos were done so well.

Integrating Linux with LDAP presentation

I gave a talk tonight at the local UNIX uers group on integrating Linux servers with LDAP. The slide desk is now available, and I want to thank everyone for coming out! The talk was a bunch of fun, and I appreciate all of the questions I received. Hopefully I can come back out later this year to talk about FreeIPA.