Updating OpenBSD packages with pkg_add

One nifty feature that recently made it’s way into OpenBSD is the ability to remotely update packages with the pkg_add utility. This is accomplished by adding the URL of a remote repository to the PKG_PATH variable, and then running pkg_add with the “-u” (update packages) and optional “-v” (verbose output) and “-i” (interactice installation) options:

$ export PKG_PATH=”ftp://ftp.openbsd.org/pub/OpenBSD/3.9/packages/i386/”

$ pkg_add -uvi

Candidates for updating curl-7.15.3 -> curl-7.15.1 curl-7.15.3                                          
Ambiguous: curl-7.15.3 could be curl-7.15.1 curl-7.15.3
Choose one package
         0: 
         1: curl-7.15.1
         2: curl-7.15.3
Your choice: 2
Looking for updates: complete                                                                           
Running the equivalent of pkg_add -r curl-7.15.3
parsing curl-7.15.3
Already installed: curl-7.15.3

This is a super useful feature for busy admins, and will definitely make my life easier!

Checking swap usage on Solaris, Linux and OpenBSD hosts

Each and every operating systemI support has a different utility to report on swap usage. On my Soalris hosts, I use the swap and vmstat utilities to check utilization:

$ swap -s

total: 36176k bytes allocated + 4672k reserved = 40848k used, 1189004k available

On Linux hosts, I use teh free and top utilities:

$ free

             total       used       free     shared    buffers     cached
Mem:       2055340    1427696     627644          0     179124     876300
-/+ buffers/cache:     372272    1683068
Swap:      1004052          0    1004052

And on my OpenBSD servers, I use the swapctl and systat utilities:

$ swapctl -l

Device      512-blocks     Used    Avail Capacity  Priority
swap_device     262068        0   262068     0%    0

Oh how I wish there was an administrator tool naming standard. :)

Debugging OpenBSD passwd problems

I recently had to manually add a few users to /etc/passwd and /etc/master.passwd on an OpenBSD 3.9 server. After I added the entries, the accounts were still unable to login. I started poking around with ktrace, and noticed that during a normal account creation session the files /etc/pwd.db and /etc/spwd.db were modified:

$ ls -la /etc/*.db

-rw-r--r--  1 root  wheel    40960 Nov 23 05:38 /etc/pwd.db
-rw-r-----  1 root  _shadow  40960 Nov 23 05:38 /etc/spwd.db

After seeing this, I went and read up on both of these files. It turns out that /etc/passwd and /etc/master.passwd get converted to database files by pwd_mkdb, and then the database files are used for actual authentication. Once I ran pwd_mkdb by hand:

$ pwd_mkdb /etc/master.passwd

Everything worked as expected. I reckon other operating systems use database files as well, so I will have to keep this in mind the next time I try to muck with a credential repository manually.

Monitoring logfiles with logsentry

I manage a fair number of servers, and use several tools to monitor the health of my systems. One such tool is logsentry (formerly known as logcheck), which is a shell script that can be used to monitor logfiles for anomalies. Logsentry consists of a single shell script and one or more violation files, and installing it as simple as extracting the package and modifying the paths in the shell script. If your using OpenBSD, you can use the pkg_add utility to add the logsenty package to your system:

$ export PKG_PATH=”ftp://ftp.openbsd.org/pub/OpenBSD/3.9/packages/i386/”

$ pkg_add logsentry

logsentry-1.1.1p2: complete

--- logsentry-1.1.1p2 -------------------
The logsentry configuration files have been installed at
/etc/logsentry.
Please view these files and change the configuration to meet your needs.

Currently logsentry will check the following files:

/var/log/messages
/var/log/maillog
/var/log/authlog
/var/log/secure
/var/log/daemon
/var/log/xferlog

Edit /etc/logsentry/logsentry.sh
if you want to add more files.

Be sure to configure your crontab as indicated by
/usr/local/share/doc/logsentry/INSTALL
so that logsentry is run regularly.

After logsentry is installed, you can add a cron job similar to the following to enable it:

$ echo ”
# Check system logfiles
0 * * * * /bin/sh /etc/logsentry/logsentry.sh” >> /var/spool/cron/crontabs/root

Once logsentry is enabled, you will get email similar to the following each time an anomaly is detected:

from	     Charlie Root
to	 	     root
date		Nov 25, 2006 1:00 PM	 
subject		yappy 11/25/06:13.00 system check	 

Security Violations
=-=-=-=-=-=-=-=-=-=
Nov 25 16:25:57 yappy su: matty to root on /dev/ttyp0
Nov 26 05:18:40 yappy su: matty to root on /dev/ttyp0
Nov 26 05:22:10 yappy su: BAD SU matty to root on /dev/ttyp0
Nov 26 05:22:14 yappy su: matty to root on /dev/ttyp0

If logsentry emails you about an anomaly that your not interested in, you can add a string that matches the error to logsentry.ignore (this is used to filter out messages from the Unusual system events section) or logsentry.violations.ignore (this is used to filter out events from the security section). Logsentry works pretty well, and once the ignore files are adjusted to match the personality of the server, it can be a life saver (I like the fact that logsentry will send an email notifications when a hardware error is written to the system logfiles).

Monitoring interface throughput on OpenBSD systems

While persuing the OpenBSD ports collection a few weeks ago, I came across the ifstat utility. This nifty utility allows you to view bandwidth totals for each interface in a server, and at specific intervals. Here is a sample run showing the bandwidth in and out of the sis0 and sis1 Ethernet interfaces, and the total bandwidth in and out of the system:

$ ifstat -TAb 5

       sis0                sis1               Total       
 Kbps in  Kbps out   Kbps in  Kbps out   Kbps in  Kbps out
  129.96      4.37      3.91    131.71    133.87    136.09
  130.48      5.43      4.77    131.98    135.25    137.41
  132.21      4.24      3.60    133.71    135.81    137.95

This is a nifty utility, and one I plan to add to my stock OpenBSD builds!

Viewing OpenBSD server utilization with systat

OpenBSD has a number of nifty utilities, and I happened to come across the systat(1) utility this weekend while looking for an executable in /usr/bin. Systat prints out performance data in an ncurses display, and can be used to view CPU saturation, I/O statistics, swap utilization, netstat data, and MBUF and network interface utilization. The utility takes the metric to display as an argument, and allows an interval to be used to control how often data is displayed:

$ systat iostat 5

                    /0   /1   /2   /3   /4   /5   /6   /7   /8   /9   /10
     Load Average   >

          /0   /10  /20  /30  /40  /50  /60  /70  /80  /90  /100
cpu  user|
     nice|
   system|
interrupt|X
     idle|XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

          /0   /10  /20  /30  /40  /50  /60  /70  /80  /90  /100
 wd0  Kps|
      tps|

I absolutely love UNIX, BSD and Linux systems. There are so many nifty tools available for these operating systems, and it’s a h00t when you come across a new utility that you didn’t previously know about. Shibby!