While attempting to run the smartctl utility a few weeks back on an x86 Solaris 10 host with IDE disk drives, I received the following error:
$ smartctl -a /dev/dsk/c1d0s0
smartctl version 5.36 [i386-pc-solaris2.10] Copyright (C) 2002-6 Bruce Allen
Home page is http://smartmontools.sourceforge.net/
#######################################################################
ATA command routine ata_command_interface() NOT IMPLEMENTED under Solaris.
Please contact smartmontools-support@lists.sourceforge.net if
you want to help in porting smartmontools to Solaris.
#######################################################################
Smartctl: Device Read Identity Failed (not an ATA/ATAPI device)
A mandatory SMART command failed: exiting. To continue, add one or more '-T permissive' options.
It turns out this error is caused by several missing ioctls in the x86 Solaris cmdk IDE device driver. Since I have always wanted to develop (or modify) a kernel device driver, I decided to start reviewing the source code in cmdk.c to see what would be needed to make smartmontools happy. After reading through the cmdkioctl routine in cmdk.c, it dawned on me that the SPARC IDE driver, dad, contained the ioctls that are used by smartmontools. If the dad device driver used the underlying DDI framework, it would be relatively trivial to port the missing pieces from the dad driver (the SPARC IDE driver) to the cmdk driver (the x86 Solaris IDE driver). Well – it turns out the dad driver is closed source (so much for “open” solaris, ey?), so that dashed my hopes of porting code. :( Since there were two open bugs related to SMART support:
Bug ID: 4665068 SMART support in IDE driver
Bug ID: 6280687 Collect SMART data from disks and deliver info to FMA
I decided to call the Sun support center to get the current status of these bugs (I had already pinged my sales team to get the status of these bugs, but my inquiries got silently routed to /dev/null). The support folks weren’t able to tell me if and when these bugs would be fixed, but they did inform me that I could add my employers name to the bugs. They also mentioned that bug fixes are drive by customer demand, so hopefully folks who read my blog can help me out here. Since disk drives are the single component most likely to fail in a system, and SMART can help to proactively detect when a disk drive will fail, it seems only logical that Sun would devote resources to adding SMART support (or at least fixing the existing bugs so people can use third party packages). If you happen to have a support contract with Sun, I would greatly appreciate it if you could open a case (either online or by phone) and request the status of these bugs, and request that your companies name be attached to the bug. Hopefully the customers voice will be heard. :)
I logged into one of my Solaris 10 hosts today to add some additional firewall rules, and noticed that the ipfilter service was in the maintenance state:
$ svcs -x ipfilter
svc:/network/ipfilter:default (IP Filter)
State: maintenance since Sat Oct 28 15:56:30 2006
Reason: Start method failed repeatedly, last exited with status 2.
See: http://sun.com/msg/SMF-8000-KS
See: ipfilter(5)
See: /etc/svc/volatile/network-ipfilter:default.log
Impact: This service is not running.
This is odd, considering this was working the last time I had checked up on the server. When I dumped out the logfile mentioned in the service state listed above, I noticed that the shell script that starts ipfilter was bombing out at line 180:
$ cat /etc/svc/volatile/network-ipfilter:default.log
[ Oct 28 15:56:16 Enabled. ]
[ Oct 28 15:56:27 Executing start method ("/lib/svc/method/ipfilter start") ]
/lib/svc/method/ipfilter: syntax error at line 180: `end of file' unexpected
[ Oct 28 15:56:27 Method "start" exited with status 2 ]
[ Oct 28 15:56:27 Executing start method ("/lib/svc/method/ipfilter start") ]
/lib/svc/method/ipfilter: syntax error at line 180: `end of file' unexpected
[ Oct 28 15:56:28 Method "start" exited with status 2 ]
[ Oct 28 15:56:28 Executing start method ("/lib/svc/method/ipfilter start") ]
/lib/svc/method/ipfilter: syntax error at line 180: `end of file' unexpected
Since I didn’t modify /lib/svc/method/ipfilter, I started to wonder why ipfilter all of a sudden quit working. The erorr message above indicated that there was an error in the script at line 180, which is a bit misleading considering the script only has 179 lines:
$ cat /lib/svc/method/ipfilter | wc -l
179
To find the actual line that was causing the issue, I decided to change the shell in /lib/svc/method/ipfilter from /sbin/sh to /bin/bash ( As a side note – I still don’t quite understand why anyone would use /sbin/sh on Solaris hosts, considering zsh, tsch and bash are available. If the reason is because of dependencies, Sun should consider moving the shells folks actually use into one of the core packages!). Once I made this change and invoked the script with the start option, bash notified me that line 123 was actually to blame:
$ /lib/svc/method/ipfilter start
/lib/svc/method/ipfilter: line 123: unexpected EOF while looking for
matching ``'
/lib/svc/method/ipfilter: line 180: syntax error: unexpected end of
file
Upon inspecting the ipfitler script in more detail, I noticed that a “`” character was missing on line 123:
case "$1" in
start)
[ ! -f ${IPFILCONF} ] && exit 0
[ -n "$pfildpid" ] && kill -TERM $pfildpid 2>/dev/null
[ -n "$pid" ] && kill -TERM $pid 2>/dev/null
/usr/sbin/pfild >/dev/null
if load_ippool && load_ipf && load_ipnat ; then
ipmon -Dsv` <------- PROBLEM
else
Once I removed the “`” from line 123, everything worked as expected. I am still not certain what caused this to happen in the first place, and the sunsolve and opensolaris bug database are not much help. If anyone else happens to experience this issue, please let me know!
When debugging problems with web applications, it is often useful to display the HTTP request and response headers along with the HTTP entity bodies. There are numerous ways to do this, and I covered several client centric solutions in my SysAdmin article Debugging Web Applications. Client side tools are extremely useful for pinpointing problems, but periodically it is helpful to see the raw requests from the client, and the responses from the server. If you happen to be running Apache, you can use the dumpio module to dump the request and response data to the Apache error log.
Setting up dumpio is super easy. To enable the dumpio module during the Apache build process, you can pass the “–enable-dumpio=shared” option to the configure script:
$ ./configure --enable-mods-shared=most --enable-dumpio-shared --with-mpm=worker \ --enable-deflate=shared --prefix=/opt/apache-2.2.3
To enable the module, you will first need to load the module with “LoadModule” directive. Once the module is configured to load, you can add the “DumpIOInput On” directive to to dump incoming data, and “DumpIOOutput On” to dump data sent by Apache. This will result in messages similar to the following appearing in the Apache error logfile:
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 16 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): GET / HTTP/1.1rn
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 13 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): Accept: */*rn
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 21 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): Accept-Language: enrn
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 32 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): Accept-Encoding: gzip, deflatern
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 24 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): Connection: keep-alivern
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 20 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): Host: 192.168.9.24rn
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(103): mod_dumpio: dumpio_in [getline-blocking] 0 readbytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_in (data-HEAP): 2 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_in (data-HEAP): rn
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(129): mod_dumpio: dumpio_out
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(51): mod_dumpio: dumpio_out (data-HEAP): 281 bytes
[Sat Oct 28 10:52:31 2006] [debug] mod_dumpio.c(67): mod_dumpio: dumpio_out (data-HEAP): HTTP/1.1 200 OKrnDate: Sat, 28 Oct 2006 14:52:31 GMTrnServer: Apache/2.2.3 (Unix)rnLast-Modified: Sat, 20 Nov 2004 20:16:24 GMTrnETag: "45b81-2c-4c23b600"rnAccept-Ranges: bytesrnContent-Length: 44rnKeep-Alive: timeout=5, max=100rnConnection: Keep-AlivernContent-Type:Ttext/htmlrnrn:00-04:00
Dumpio is an extremely useful module, and is an invaluable tool for debugging web application.
In OpenBSD 3.9, the sensor framework was added to allow users to monitor systems that contained hardware sensors (e.g., temperature, voltage, fan speed, etc.). Sensor data is made available through one or more “hw.sensors” kernel state variables, which can be viewed with the sysctl utility:
$ sysctl -a | grep "hw.sensors"
hw.sensors.0=nsclpcsio0, TSENS1, temp, 127.00 degC / 260.60 degF
hw.sensors.1=nsclpcsio0, TSENS2, temp, 127.00 degC / 260.60 degF
hw.sensors.2=nsclpcsio0, TNSC, temp, 57.00 degC / 134.60 degF
hw.sensors.3=nsclpcsio0, VSENS0, volts_dc, 3.01 V
hw.sensors.4=nsclpcsio0, VSENS1, volts_dc, 2.02 V
hw.sensors.5=nsclpcsio0, VSENS2, volts_dc, 2.52 V
hw.sensors.6=nsclpcsio0, VSENS3, volts_dc, 0.60 V
hw.sensors.7=nsclpcsio0, VSENS4, volts_dc, 2.48 V
hw.sensors.8=nsclpcsio0, VSENS5, volts_dc, 2.49 V
hw.sensors.9=nsclpcsio0, VSENS6, volts_dc, 0.00 V
hw.sensors.10=nsclpcsio0, VSB, volts_dc, 3.28 V
hw.sensors.11=nsclpcsio0, VDD, volts_dc, 3.31 V
hw.sensors.12=nsclpcsio0, VBAT, volts_dc, 3.01 V
hw.sensors.13=nsclpcsio0, AVDD, volts_dc, 3.26 V
hw.sensors.14=nsclpcsio0, TS1, volts_dc, 1.63 V
hw.sensors.15=nsclpcsio0, TS2, volts_dc, 1.62 V
hw.sensors.16=nsclpcsio0, TS3, volts_dc, 1.51 V
In addition to kernel support, OpenBSD has a userland daemon, sensorsd, which can be used to monitor one or more sensor values. If one of the sensor values that sensorsd is monitoring goes above or below an administrator defined threshold, sensorsd will by default write an entry similar to the following to the system log:
Oct 27 18:27:42 [192.168.1.1.2.2] sensorsd[8513]: hw.sensors.2: exceed limits, value: 56.00C/132.80F
Sensor thresholds are defined in the file /etc/sensorsd.conf. Each entry in /etc/sensorsd.conf contains the name of a sensor to monitor, the value “high” or “low” to indicate a peak or valley for the sensor value, and the value the sensor value cannot go above or below. The following example shows one of the entries I use to monitor the CPU temperature in my Soekris net4801-60:
$ cat /etc/sensorsd.conf
hw.sensors.2:high=55C
If syslog entries aren’t sufficient for monitoring, you can also use the “command” option to run a program when sensorsd detects that a value has crossed one of the thresholds. This script can email an administrator, send an SNMP trap to a network monitoring station, or shutdown the system (this could be useful it a fan fails or if a voltage level takes a turn for the worst). This is a sweet feature, and super super valuable!
Social Distortion have been around for a long time, and they used to get a fair amount of airplay on the alternative radio stations I listened to back home. Mike Ness’ vocals and guitar skills always amazed me, and I became a devoted Social D fan after listening to their “White Light, White Heat, White Trash” CD. I got a chance to see the band play live at a relatively small venue last night, and the show turned out to be chaotic and fun!
The band took the stage around 10pm, with Mike Ness and company launching into an energized version of “reach for the sky” to start the show. The band sounded incredible live, and their music provided the necessary energy for hundreds of people to go crazy down in the mosh pits (it was a scene from a 1996 Metallica concert, and I thought my friend kenny was going to get killed!). In addition to playing “reach for the sky,” the band also cranked out hits such as “don’t drag me down,” “mommy’s little monster,” “ball and chain,” “highway 101,” “story of my life,” “prison break” and covered “under my thumb” and the classic hit “ring of fire” from Johnny Cash. This was definitly a night I will remember, and the amount of energy that was at the club was incredible! Long live Social D!