Monitoring hardware sensors with the OpenBSD sensor daemon

In OpenBSD 3.9, the sensor framework was added to allow users to monitor systems that contained hardware sensors (e.g., temperature, voltage, fan speed, etc.). Sensor data is made available through one or more “hw.sensors” kernel state variables, which can be viewed with the sysctl utility:

$ sysctl -a | grep “hw.sensors”

hw.sensors.0=nsclpcsio0, TSENS1, temp, 127.00 degC / 260.60 degF
hw.sensors.1=nsclpcsio0, TSENS2, temp, 127.00 degC / 260.60 degF
hw.sensors.2=nsclpcsio0, TNSC, temp, 57.00 degC / 134.60 degF
hw.sensors.3=nsclpcsio0, VSENS0, volts_dc, 3.01 V
hw.sensors.4=nsclpcsio0, VSENS1, volts_dc, 2.02 V
hw.sensors.5=nsclpcsio0, VSENS2, volts_dc, 2.52 V
hw.sensors.6=nsclpcsio0, VSENS3, volts_dc, 0.60 V
hw.sensors.7=nsclpcsio0, VSENS4, volts_dc, 2.48 V
hw.sensors.8=nsclpcsio0, VSENS5, volts_dc, 2.49 V
hw.sensors.9=nsclpcsio0, VSENS6, volts_dc, 0.00 V
hw.sensors.10=nsclpcsio0, VSB, volts_dc, 3.28 V
hw.sensors.11=nsclpcsio0, VDD, volts_dc, 3.31 V
hw.sensors.12=nsclpcsio0, VBAT, volts_dc, 3.01 V
hw.sensors.13=nsclpcsio0, AVDD, volts_dc, 3.26 V
hw.sensors.14=nsclpcsio0, TS1, volts_dc, 1.63 V
hw.sensors.15=nsclpcsio0, TS2, volts_dc, 1.62 V
hw.sensors.16=nsclpcsio0, TS3, volts_dc, 1.51 V

In addition to kernel support, OpenBSD has a userland daemon, sensorsd, which can be used to monitor one or more sensor values. If one of the sensor values that sensorsd is monitoring goes above or below an administrator defined threshold, sensorsd will by default write an entry similar to the following to the system log:

Oct 27 18:27:42 [192.168.1.1.2.2] sensorsd[8513]: hw.sensors.2: exceed limits, value: 56.00C/132.80F

Sensor thresholds are defined in the file /etc/sensorsd.conf. Each entry in /etc/sensorsd.conf contains the name of a sensor to monitor, the value “high” or “low” to indicate a peak or valley for the sensor value, and the value the sensor value cannot go above or below. The following example shows one of the entries I use to monitor the CPU temperature in my Soekris net4801-60:

$ cat /etc/sensorsd.conf
hw.sensors.2:high=55C

If syslog entries aren’t sufficient for monitoring, you can also use the “command” option to run a program when sensorsd detects that a value has crossed one of the thresholds. This script can email an administrator, send an SNMP trap to a network monitoring station, or shutdown the system (this could be useful it a fan fails or if a voltage level takes a turn for the worst). This is a sweet feature, and super super valuable!

Concert review: Social Distortion

Social Distortion have been around for a long time, and they used to get a fair amount of airplay on the alternative radio stations I listened to back home. Mike Ness’ vocals and guitar skills always amazed me, and I became a devoted Social D fan after listening to their “White Light, White Heat, White Trash” CD. I got a chance to see the band play live at a relatively small venue last night, and the show turned out to be chaotic and fun!

The band took the stage around 10pm, with Mike Ness and company launching into an energized version of “reach for the sky” to start the show. The band sounded incredible live, and their music provided the necessary energy for hundreds of people to go crazy down in the mosh pits (it was a scene from a 1996 Metallica concert, and I thought my friend kenny was going to get killed!). In addition to playing “reach for the sky,” the band also cranked out hits such as “don’t drag me down,” “mommy’s little monster,” “ball and chain,” “highway 101,” “story of my life,” “prison break” and covered “under my thumb” and the classic hit “ring of fire” from Johnny Cash. This was definitly a night I will remember, and the amount of energy that was at the club was incredible! Long live Social D!

Getting stack traces from Linux and Solaris processes

When an application is running, it is sometimes useful to see which stack frame the application is currently in. This is especially useful when debugging hung processes, since you can see which function the program is currently executing, and the code path taken to get to that point. Both Linux and Solaris provide utilities to capture live stack traces from processes. The Solaris tool to grab this information is called pstack. Pstack takes a process id as an argument, and will print stack traces for each thread in the process id passed as an argument:

$ pstack 539

539:    slapd -f /opt/openldap-common/etc/slapd.conf -u openldap -g openldap -
-----------------  lwp# 1 / thread# 1  --------------------
 fee412d0 lwp_wait (2, ffbffaac)
 fee3ccd8 _thrp_join (2, 0, 0, 1, ffbffaac, fee6cbc0) + 34
 00032f30 slapd_daemon (0, 130118, 0, 0, 0, 1) + cc
 00023bb8 main     (9, ffbffc2c, a0, 199920, 0, 199338) + ed4
 000228b0 _start   (0, 0, 0, 0, 0, 0) + 5c
-----------------  lwp# 2 / thread# 2  --------------------
 fee40e7c pollsys  (f83fd000, 3, 0, 0)
 fede25b0 pselect_large_fdset (f83fd018, 3, f83fd208, f83fd200, 20, f83fd20c) + 2ec
 fede29f8 select_large_fdset (13, f83ffd88, 0, 0, 0, 199500) + a0
 00032300 ???????? (134c00, f83ffb88, 0, 173c00, 170400, 176d80)
 fee400b0 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 3 / thread# 3  --------------------
 fee40150 lwp_park (0, 0, 0)
 fee3a1e4 cond_wait_queue (1acbc8, 1acbb0, 0, 0, 0, 0) + 28
 fee3a764 cond_wait (1acbc8, 1acbb0, 0, fe9f0000, 2280b8, 0) + 10
 fee3a7a0 pthread_cond_wait (1acbc8, 1acbb0, 0, 0, 0, 3) + 8
 000df5b4 ???????? (1acba8, f7c00000, 0, 0, df42c, 0)
 fee400b0 _lwp_start (0, 0, 0, 0, 0, 0)
-----------------  lwp# 4 / thread# 4  --------------------
 fee40150 lwp_park (0, 0, 0)
 fee3a1e4 cond_wait_queue (1acbc8, 1acbb0, 0, 0, 0, 0) + 28
 fee3a764 cond_wait (1acbc8, 1acbb0, 0, fe9f0400, 2280b8, 0) + 10
 fee3a7a0 pthread_cond_wait (1acbc8, 1acbb0, 0, 0, 0, 3) + 8
 000df5b4 ???????? (1acba8, f7400000, 0, 0, df42c, 0)
 fee400b0 _lwp_start (0, 0, 0, 0, 0, 0)

The Linux utility to grab comparable information is called gstack, and it works nearly identically to pstack:

$ gstack 2195

#0  0x00194402 in __kernel_vsyscall ()
#1  0x00469e7d in ___newselect_nocancel () from /lib/libc.so.6
#2  0x00cc1c5b in main () from /proc/2195/exe

These utilities have helped me time and time again, and are two extremely useful pieces of software.

Speeding up Solaris zone creation with cloning

One really neat feature that was recently added to Solaris is the ability to clone zones. Cloning allows you to create a new zone from an existing zone, which can reduce provisioning time, and ensure that all zones are created consistently (e.g., all zones that will act as web servers can be cloned from a zone that was setup to act as a web server). If the zone you are cloning from resides on a UFS or VxFS file system, the clone operation will copy all of the files from the source zone to the new zone. If the zone that you are cloning from lives on a ZFS file system, the clone operation will create a writeable ZFS snapshot, and use that as the backing store for the new zone. When zones are cloned with the ZFS method, the new zone is created almost instantaneously (it typically takes .5 seconds), and little to no storage is required to initially provision the zone.

To show you how this works, I created a Linux branded zone named “centostemplate.” The zone centostemplate lives on it’s own ZFS file system, as you can see from the output of the “zoneadm” and “zfs” utilities:

$ /usr/sbin/zoneadm list -vc

  ID NAME             STATUS         PATH                           BRAND     
   0 global           running        /                              native    
   - centostemplate   installed      /app/zones/centostemplate      lx        

$ zfs list

NAME                       USED  AVAIL  REFER  MOUNTPOINT
app                       3.05G  8.67G  27.5K  /app
app/zones                 1.05G  8.67G  26.5K  /app/zones
app/zones/centostemplate  1.05G  8.67G  1.05G  /app/zones/centostemplate

To create a new zone named centos_37_1 from a clone of the zone named centostemplate, the zone cetos_37_1 first needs to be configured in the zone configuration shell:

$ zonecfg -z centos_37_1

centos_37_1: No such zone configured
Use 'create' to begin configuring a new zone.
zonecfg:centos_37_1> create
zonecfg:centos_37_1> set zonepath=/app/zones/centos_37_1
zonecfg:centos_37_1> commit
zonecfg:centos_37_1> exit

Once the new zone is configured the way you want it, the zoneadm utilily can be run with the “-z” option, the name of the new zone, the “clone” command, and the name of the zone you would like use as the source of the clone operation (in the following example, the centostemplate zone is the source for the clone operation):

$ timex zoneadm -z centos_37_1 clone centostemplate

Cloning snapshot app/zones/centostemplate@SUNWzone1
Instead of copying, a ZFS clone has been created for this zone.

real          0.63
user          0.07
sys           0.10

As you can see from the output, it look just over .5 seconds to create a brand spanking new zone! Nice! If I run the zoneadm and zfs utilities again, you can see that a writeable snapshot was created to act as the backing store for the zone named centos_37_1:

$ /usr/sbin/zoneadm list -vc

  ID NAME             STATUS         PATH                           BRAND     
   0 global           running        /                              native    
   - centostemplate   installed      /app/zones/centostemplate      lx        
   - centos_37_1      installed      /app/zones/centos_37_1         lx        

$ zfs list

NAME                                 USED  AVAIL  REFER  MOUNTPOINT
app                                 3.05G  8.67G  27.5K  /app
app/zones                           1.05G  8.67G  27.5K  /app/zones
app/zones/centos_37_1                   0  8.67G  1.05G  /app/zones/centos_37_1
app/zones/centostemplate            1.05G  8.67G  1.05G  /app/zones/centostemplate
app/zones/centostemplate@SUNWzone1  87.5K      -  1.05G  -

This is some amazingly cool stuff, and Sun keeps adding more and more nifty stuff to Solaris!

Securing PHP installations

I have been using PHP for quite some time, and have developed a love-hate relationship with it. The ability to rapidly create dynamic web applications is extremely powerful, but PHP’s absymal security track record often leads me to wonder if I should be using it (especially since a fair number of opensource PHP applications are poorly coded, and using them can put your hosting platform at risk).

Over the past few months, I have spent a fair amount of time researching ways to protect my PHP enabled web servers from insecure code, and unknown PHP vulnerabilities. There are a number of ways you can tackle PHP security, the most important being designing and writing secure PHP code. The second way to protect your PHP applications is by applying the suhosin (formerly known as the hardended PHP patch) patch, which integrates a number of advanced security protections directly into the PHP runtime environment. The third method is to use an operating system that protects against stack and heap overflows, and the the last method is to tighten up the runtime environment by enabling one or more variables in the php.ini file.

Installing suhosin is extremely easy, and the benefits it provides are described on the suhosin website. To integrate suhosin into the PHP source code, you will first need to download and extract the PHP source code (you should also verify the public key signatures / MD5 checksums distributed with the source code):

$ wget http://us3.php.net/get/php-5.1.6.tar.gz/from/www.php.net/mirror

$ gtar xfvz php-5.1.6.tar

Once the PHP source code has been extracted, you will need to retrieve and apply the suhosin patch to the PHP source code (you can also compile suhosin as an extension, but I prefer to use the patch):

$ wget http://www.hardened-php.net/suhosin/_media/suhosin-patch-5.1.6-0.9.5.patch.gz

$ gunzip suhosin-patch-5.1.6-0.9.5.patch.gz

$ gpatch -p0 < suhosin-patch-5.1.6-0.9.5.patch
patching file php-5.1.6/configure
patching file php-5.1.6/configure.in
patching file php-5.1.6/ext/standard/basic_functions.c
patching file php-5.1.6/ext/standard/dl.c
patching file php-5.1.6/ext/standard/file.c
patching file php-5.1.6/ext/standard/file.h
patching file php-5.1.6/ext/standard/info.c
patching file php-5.1.6/ext/standard/syslog.c
patching file php-5.1.6/main/fopen_wrappers.c
patching file php-5.1.6/main/main.c
patching file php-5.1.6/main/php_config.h.in
patching file php-5.1.6/main/php.h
patching file php-5.1.6/main/php_logos.c
patching file php-5.1.6/main/snprintf.c
patching file php-5.1.6/main/spprintf.c
patching file php-5.1.6/main/suhosin_globals.h
patching file php-5.1.6/main/suhosin_logo.h
patching file php-5.1.6/main/suhosin_patch.c
patching file php-5.1.6/main/suhosin_patch.h
patching file php-5.1.6/main/suhosin_patch.m4
patching file php-5.1.6/sapi/apache/mod_php5.c
patching file php-5.1.6/sapi/apache2filter/sapi_apache2.c
patching file php-5.1.6/sapi/apache2handler/sapi_apache2.c
patching file php-5.1.6/sapi/cgi/cgi_main.c
patching file php-5.1.6/sapi/cli/php_cli.c
patching file php-5.1.6/TSRM/TSRM.h
patching file php-5.1.6/TSRM/tsrm_virtual_cwd.c
patching file php-5.1.6/TSRM/tsrm_virtual_cwd.h
patching file php-5.1.6/win32/build/config.w32
patching file php-5.1.6/Zend/Makefile.am
patching file php-5.1.6/Zend/zend_alloc.c
patching file php-5.1.6/Zend/zend_alloc.h
patching file php-5.1.6/Zend/zend.c
patching file php-5.1.6/Zend/zend_canary.c
patching file php-5.1.6/Zend/zend_compile.c
patching file php-5.1.6/Zend/zend_compile.h
patching file php-5.1.6/Zend/zend_constants.c
patching file php-5.1.6/Zend/Zend.dsp
patching file php-5.1.6/Zend/zend_errors.h
patching file php-5.1.6/Zend/zend.h
patching file php-5.1.6/Zend/zend_hash.c
patching file php-5.1.6/Zend/zend_llist.c
patching file php-5.1.6/Zend/ZendTS.dsp

After the suhosin patch is applied, you can configure and build PHP according to the PHP installation guide. The following example shows how to compile PHP with the most basic options:

$ ./configure –prefix=/opt/apache/php-5.1.6 –with-apxs2=/opt/apache/bin/apxs

$ make

$ make install

$ ln -s /opt/apache/php-5.1.6 /opt/apache/php

To verify that your module is built with the suhosin patch, you can run the php utility with the the “-i” (print PHP information) option:

$ /opt/apache/php/bin/php -i |more
phpinfo()
PHP Version => 5.1.6

System => SunOS web 5.10 Generic_118855-19 i86pc
Build Date => Oct 10 2006 17:10:49

[ ….. ]

This server is protected with the Suhosin Patch 0.9.5
Copyright (c) 2006 Hardened-PHP Project

If the string “This server is protected with the Suhosin Patch” is not listed in the php output, something most likely went wrong, and you will need to check the steps you used to integrate suhosin into the PHP source code. If the string is present, you can turn your attention to enabling stack and heap protections in your operating system (each OS handles this differently, so check google if you want more information), and modifying the php.ini file with more secure defaults.

The php.ini file is used to configure the PHP runtime environment, and there are several variables that can be set in the initiatialization file to provide a more secure execution environment. On the servers I support, I typically set the following variables (the items prefixed with a “;” are comments):

; Disable global variable registration
register_globals = Off

; Don’t display PHP errors
display_errors = Off

; Log all PHP errors to the Apache error log
log_errors = On

; Tell suhosin to log all security errors to syslog
suhosin.log.syslog = 511
suhosin.log.syslog.facility = 11
suhosin.log.syslog.priority = 4

; Verify that the uid the PHP script runs as is the
; same as the file they are trying to access, and
; escape backticks and disable shell exec()s
safe_mode = On

; Disable any and all execution of programs by
; setting safe_mode_exec_dir to an empty directory
; (and one that cannot have programs uploaded to it).
safe_mode_exec_dir = /opt/web/empty

; List the set of variables a user cannot change with putenv()
safe_mode_protected_env_vars = LD_PRELOAD, LD_LIBRARY_PATH, LD_RUN_PATH, PATH

; Set the directory where PHP programs can be run from
open_basedir = /opt/web/php_content

; Tell the PHP engine not to append anything to the
; web server “Server” header (NOTE: set ServerTokens
; to Prod if you are using Apache)
expose_php = Off

; Escape single quotes, double quotes, backslashes and
; NUL characters with a backslash
magic_quotes_gpc = On

; Disable dynamic loading of PHP extensions (this should be
; covered by safe_mode, but it’s ok to be extra safe)
enable_dl = Off

; Disable HTTP uploads to the server
file_uploads = Off

; Disable remote file opening
allow_url_fopen = off

Several of these variables WILL break existing applications, so you should evaluate their use prior to enabling them in a production environment. The combination of secure coding practices, operating system stack and heap overflow protections, suhosin, and a secure php.ini file should be used on every server that hosts a PHP runtime environment. If you take additional steps to secure PHP on your servers, please leave me a comment. I am always looking for new and interesting ways to protect the services I support!

Using ZFS compression

One of the nice things about ZFS is it’s ability to transparently compress file system data. Compression can be enabled or disabled dynamically, and compression policies can be applied on a per-file system basis. This can be extremely useful if you need to save space on a file system that is used primarily to archive data, since you can create a file system decicated to data archival, and then enable compression on that file system to save space. If on the other hand performance is your primary concern (e.g., actively used Oracle database files), you can create a file system without compression. Both of these file systems can reside in the same ZFS pool, and enabling and disabling compression is a simple and straight forward operation.

To enable compression on the ZFS file system named concat/archive, the zfs utilities set option can be used to set the file systems “compression” attribute :

$ zfs set compression=on concat/archive

To see if compression is enabled for the file system concat/archive, the zfs utilities get options can be used to retrieve the value for the file systems “compression” attribute:

$ zfs get compression concat/archive

NAME    PROPERTY     VALUE     SOURCE
concat  compression  on        local

Once compression is enabled on a file system, ZFS will track the compression ratio for the data on the file system. To view the compression ratio, the zfs utilities get option can be run to retrieve the “compressratio” attribute:

$ zfs get compressratio

NAME            PROPERTY       VALUE           SOURCE
concat          compressratio  1.82x           -
concat/archive  compressratio  1.00x           -
concat/zone2    compressratio  2.83x           -

ZFS compression is some sweet livin’!