Blog O’ Matty

I had my first extended downtime with Dreamhost today. According to my cron job that runs content-check every minute, my site was down for close to 30 minutes. I hope this is an isolated event, and plan to evaluate other providers in case I need to move my domain again. If anyone has recommendations on stable and reliable shared hosting providers, please feel free to leave your feedback in the comments section.

I’ve been cleaning out a bunch of super old notes this week, and am going to post them on my blog for future reference. In case I incur a head injury and need to memorize the crontab format again, here it is as ascii art:

minute (0-59),
|      hour (0-23),
|      |       day of the month (1-31),
|      |       |       month of the year (1-12),
|      |       |       |       day of the week (0-6 with 0=Sunday).
|      |       |       |       |       command
0      2       *       *       0,4     /etc/cron.d/logchecker

You will probably see a few of these posts hit my blog in the days to come.

While performing some testing a few weeks ago, I needed to create a ramdisk on one of my redhat AS 4.0 servers. I knew Solaris supported tmpfs, and after a bit of googling was surprised to find that Linux supported the tmpfs pseudo-file system as well. To create a ramdisk on a Linux host, you first need to find a suitable place to mount the tmpfs file system. For my tests, I used mkdir to create a directory valled /var/ramdisk:

$ mkdir /var/ramdisk

Once the mount point is identified, you can use the mount command to mount a tmpfs file system on top of that mount point:

$ mount -t tmpfs none /var/ramdisk -o size=28m

Now each time you access /var/ramdisk, your reads and writes will be coming directly from memory. Nice!

The RPM indexes on one of my CentOS 4.4 machines got corrupted last weekend, which caused some issues on one of the servers I was trying to update. To fix this issue, I removed the files with two underscores in /var/lib/rpm:

$ rm -f /var/lib/rpm/__*

And then I recreated the indexes by running rpm with the “–rebuild” option:

$ rpm –rebuild

This experience taught me a few things:

1. The package header information is stored in /var/lib/rpm/Packages

2. RPM has an “–initdb” option to initialize the RPM database

3. I really don’t care for RPM

I manage a fair number of servers, and use several tools to monitor the health of my systems. One such tool is logsentry (formerly known as logcheck), which is a shell script that can be used to monitor logfiles for anomalies. Logsentry consists of a single shell script and one or more violation files, and installing it as simple as extracting the package and modifying the paths in the shell script. If your using OpenBSD, you can use the pkg_add utility to add the logsenty package to your system:

$ export PKG_PATH=”ftp://ftp.openbsd.org/pub/OpenBSD/3.9/packages/i386/”

$ pkg_add logsentry

logsentry-1.1.1p2: complete

Be sure to configure your crontab as indicated by
/usr/local/share/doc/logsentry/INSTALL
so that logsentry is run regularly.

After logsentry is installed, you can add a cron job similar to the following to enable it:

$ echo ”
# Check system logfiles
0 * * * * /bin/sh /etc/logsentry/logsentry.sh” >> /var/spool/cron/crontabs/root

Once logsentry is enabled, you will get email similar to the following each time an anomaly is detected:

from	     Charlie Root
to	 	     root
date		Nov 25, 2006 1:00 PM
subject		yappy 11/25/06:13.00 system check	 

Security Violations
=-=-=-=-=-=-=-=-=-=
Nov 25 16:25:57 yappy su: matty to root on /dev/ttyp0
Nov 26 05:18:40 yappy su: matty to root on /dev/ttyp0
Nov 26 05:22:10 yappy su: BAD SU matty to root on /dev/ttyp0
Nov 26 05:22:14 yappy su: matty to root on /dev/ttyp0

If logsentry emails you about an anomaly that your not interested in, you can add a string that matches the error to logsentry.ignore (this is used to filter out messages from the Unusual system events section) or logsentry.violations.ignore (this is used to filter out events from the security section). Logsentry works pretty well, and once the ignore files are adjusted to match the personality of the server, it can be a life saver (I like the fact that logsentry will send an email notifications when a hardware error is written to the system logfiles).

I received a comment from a reader regarding the trafshow utility. Trafshow is definitely a cool piece of software, and I use it one some of my Linux hosts. On my OpenBSD systems, I have been using the pktstat utility, which provides connection statistics for all traffic on the system:

$ pktstat

interface: sis0    total: 13.8Mb (1m34s)
cur: 147.0k (72%) [115.5k 39.4k 14.5k] min: 94.4k max: 202.3k avg: 145.6k bps

   bps    %      b desc
134.4k  66%  12.9M tcp 192.168.1.10:8010 <-> 192.168.1.100:58720
 105.6   0%  528.0 tcp 192.168.1.10:www <-> 192.168.1.100:54947
 12.5k   6%  62.5k tcp 192.168.1.10:www <-> 192.168.1.100:64475
                   - 200 GET /index.html

Since trafshow contains numerous features that aren’t present in pktstat, I reckon I should upgrade my OpenBSD image to use trafshow. Thanks for the comment!

While performing some testing a few weeks back, I needed to enable jumbo frames on one of our Fujitsu 250s. This was accomplished with the following three steps:

1. Add the following line to /etc/system:
$ echo “set fjgi:fjgi_jumbo=1″ >> /etc/system

2. If the default 9000 byte MTU isn’t ideal, add the preferred MTU to the /etc/fjmtu.fjgiX file
$ echo “8192″ >> /etc/fjmtu.fjgi0

3. Reboot the system.

If everything works correctly, ifconfig will report the larger mtu.

While debugging a DNS problem a few weeks back, I needed a way to measure the time it took a name server to respond to a DNS request. After poking around the OpenBSD ports collection, I came across the nsping utility. Nsping queries a DNS server passed on the command line, and reports the time it took the server to resolve a name. The following example shows how to use nsping to measure the time it takes to resolve the name prefetch.net on the name server ns2.dreamhost.com:

$ nsping -t 5 -h prefetch.net ns2.dreamhost.com

NSPING ns2.dreamhost.com (66.201.54.66): Hostname = "prefetch.net", Type = "IN A"
+ [   0 ]    46 bytes from 66.201.54.66:   76.224 ms [    0.000 san-avg ]
+ [   1 ]    46 bytes from 66.201.54.66:   79.862 ms [   78.043 san-avg ]
+ [   3 ]    46 bytes from 66.201.54.66:   79.902 ms [   78.663 san-avg ]
+ [   4 ]    46 bytes from 66.201.54.66:   79.912 ms [   78.975 san-avg ]
+ [   6 ]    46 bytes from 66.201.54.66:   79.920 ms [   79.164 san-avg ]
^C
Total Sent: [   7 ] Total Received: [   5 ] Missed: [   2 ] Lagged [   0 ]
Ave/Max/Min:   79.164 /   79.920 /   76.224

Each line contains the size of the response, the time it took to complete the request, and a sequence number. The summary line contains the numer of requests that were sent to the server, the number that were missing, and the average, maximum and minimum response times. If you want to use a resource record type other than “A,” (the default resource record type) you can invoke nsping with the “-T” option the resource record type to use:

$ nsping -t 5 -h prefetch.net -T mx ns1.dreamhost.com

NSPING ns1.dreamhost.com (66.33.206.206): Hostname = "prefetch.net", Type = "IN MX"
+ [   0 ]   136 bytes from 66.33.206.206:   73.875 ms [    0.000 san-avg ]
+ [   1 ]   136 bytes from 66.33.206.206:   79.905 ms [   76.890 san-avg ]
+ [   2 ]   136 bytes from 66.33.206.206:   80.476 ms [   78.085 san-avg ]
+ [   3 ]   136 bytes from 66.33.206.206:   80.030 ms [   78.572 san-avg ]
+ [   6 ]   136 bytes from 66.33.206.206:   80.004 ms [   78.858 san-avg ]
^C
Total Sent: [   7 ] Total Received: [   5 ] Missed: [   2 ] Lagged [   0 ]
Ave/Max/Min:   78.858 /   80.476 /   73.875

Now to figure out why DNS responses are missing!