I’ve been paged countless times to investigate systems that have gone off the air for one reason or another. Typically when look into these issues the server has crashed unexpectedly (bug, hardware, etc.) and rebooted in an attempt to get a fresh start on life. On occassion the box won’t boot due to a misconfiguration or inconsistent file system, and the server will be left with a read-only root file system and no network connectivity.
Fixing this is pretty easy. If possible I will add a 1 to the end of the boot line to boot the kernel into single user mode. If that works I can fix the issue and then allow the box to boot up normally. If the root file system is mounted read-only, I will use the mount command’s remount option to mount it read-write:
$ mount -o remount,rw /
Once you can write to the file system you should be able to write out changes to the file system to correct the issue that prevented the server from booting. Viva la remount!