Dreamhost outage

I had my first extended downtime with Dreamhost today. According to my cron job that runs content-check every minute, my site was down for close to 30 minutes. I hope this is an isolated event, and plan to evaluate other providers in case I need to move my domain again. If anyone has recommendations on stable and reliable shared hosting providers, please feel free to leave your feedback in the comments section.

Cron format

I’ve been cleaning out a bunch of super old notes this week, and am going to post them on my blog for future reference. In case I incur a head injury and need to memorize the crontab format again, here it is as ascii art:

minute (0-59),
|      hour (0-23),
|      |       day of the month (1-31),
|      |       |       month of the year (1-12),
|      |       |       |       day of the week (0-6 with 0=Sunday).
|      |       |       |       |       command
0      2       *       *       0,4     /etc/cron.d/logchecker

You will probably see a few of these posts hit my blog in the days to come.

Creating a Linux ramdisk

While performing some testing a few weeks ago, I needed to create a ramdisk on one of my redhat AS 4.0 servers. I knew Solaris supported tmpfs, and after a bit of googling was surprised to find that Linux supported the tmpfs pseudo-file system as well. To create a ramdisk on a Linux host, you first need to find a suitable place to mount the tmpfs file system. For my tests, I used mkdir to create a directory valled /var/ramdisk:

$ mkdir /var/ramdisk

Once the mount point is identified, you can use the mount command to mount a tmpfs file system on top of that mount point:

$ mount -t tmpfs none /var/ramdisk -o size=28m

Now each time you access /var/ramdisk, your reads and writes will be coming directly from memory. Nice!

Rebuilding the RPM database indexes

The RPM indexes on one of my CentOS 4.4 machines got corrupted last weekend, which caused some issues on one of the servers I was trying to update. To fix this issue, I removed the files with two underscores in /var/lib/rpm:

$ rm -f /var/lib/rpm/__*

And then I recreated the indexes by running rpm with the “–rebuild” option:

$ rpm –rebuild

This experience taught me a few things:

1. The package header information is stored in /var/lib/rpm/Packages

2. RPM has an “–initdb” option to initialize the RPM database

3. I really don’t care for RPM

Monitoring logfiles with logsentry

I manage a fair number of servers, and use several tools to monitor the health of my systems. One such tool is logsentry (formerly known as logcheck), which is a shell script that can be used to monitor logfiles for anomalies. Logsentry consists of a single shell script and one or more violation files, and installing it as simple as extracting the package and modifying the paths in the shell script. If your using OpenBSD, you can use the pkg_add utility to add the logsenty package to your system:

$ export PKG_PATH=”ftp://ftp.openbsd.org/pub/OpenBSD/3.9/packages/i386/”

$ pkg_add logsentry

logsentry-1.1.1p2: complete

--- logsentry-1.1.1p2 -------------------
The logsentry configuration files have been installed at
/etc/logsentry.
Please view these files and change the configuration to meet your needs.

Currently logsentry will check the following files:

/var/log/messages
/var/log/maillog
/var/log/authlog
/var/log/secure
/var/log/daemon
/var/log/xferlog

Edit /etc/logsentry/logsentry.sh
if you want to add more files.

Be sure to configure your crontab as indicated by
/usr/local/share/doc/logsentry/INSTALL
so that logsentry is run regularly.

After logsentry is installed, you can add a cron job similar to the following to enable it:

$ echo ”
# Check system logfiles
0 * * * * /bin/sh /etc/logsentry/logsentry.sh” >> /var/spool/cron/crontabs/root

Once logsentry is enabled, you will get email similar to the following each time an anomaly is detected:

from	     Charlie Root
to	 	     root
date		Nov 25, 2006 1:00 PM	 
subject		yappy 11/25/06:13.00 system check	 

Security Violations
=-=-=-=-=-=-=-=-=-=
Nov 25 16:25:57 yappy su: matty to root on /dev/ttyp0
Nov 26 05:18:40 yappy su: matty to root on /dev/ttyp0
Nov 26 05:22:10 yappy su: BAD SU matty to root on /dev/ttyp0
Nov 26 05:22:14 yappy su: matty to root on /dev/ttyp0

If logsentry emails you about an anomaly that your not interested in, you can add a string that matches the error to logsentry.ignore (this is used to filter out messages from the Unusual system events section) or logsentry.violations.ignore (this is used to filter out events from the security section). Logsentry works pretty well, and once the ignore files are adjusted to match the personality of the server, it can be a life saver (I like the fact that logsentry will send an email notifications when a hardware error is written to the system logfiles).

Monitoring connections on OpenBSD hosts

I received a comment from a reader regarding the trafshow utility. Trafshow is definitely a cool piece of software, and I use it one some of my Linux hosts. On my OpenBSD systems, I have been using the pktstat utility, which provides connection statistics for all traffic on the system:

$ pktstat

interface: sis0    total: 13.8Mb (1m34s)
cur: 147.0k (72%) [115.5k 39.4k 14.5k] min: 94.4k max: 202.3k avg: 145.6k bps

   bps    %      b desc                                                                     
134.4k  66%  12.9M tcp 192.168.1.10:8010 <-> 192.168.1.100:58720
 105.6   0%  528.0 tcp 192.168.1.10:www <-> 192.168.1.100:54947
 12.5k   6%  62.5k tcp 192.168.1.10:www <-> 192.168.1.100:64475
                   - 200 GET /index.html

Since trafshow contains numerous features that aren’t present in pktstat, I reckon I should upgrade my OpenBSD image to use trafshow. Thanks for the comment!