Getting Apache backtraces with mod-backtrace

While perusing the web last week I came across Jeff Trawick’s mod-backtrace module. This module can be used to print a stack backtrace each time Apache receives a critical signal (e.g., SIGSEGV, SIGBUS, etc.), and can be an invaluable resource for locating modules that are misbehaving. Since the module only worked with Linux and FreeBSD, I created a patch to allow it to work on Solaris systems. To install and configure mod-backtrace on a Solaris system, you will need to build Apache with the “–enable-exception-hook” option, download mod-backtrace.c from Jeff’s website, apply the patch, and compile mod-backtrace with the apxs utility:

$ wget http://people.apache.org/~trawick/mod_backtrace.c

$ /usr/bin/patch -p0 < mod-backtrace.diff

  Looks like a unified context diff.
done

$ apxs -ci mod-backtrace.c

/var/tmp/apache2/build/libtool --silent --mode=compile gcc -prefer-pic  -DAP_HAVE_DESIGNATED_INITIALIZER -DSOLARIS2=10 -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -g -O2 -I/var/tmp/apache2/include  -I/var/tmp/apache2/include   -I/var/tmp/apache2/include   -c -o mod-backtrace.lo mod-backtrace.c && touch mod-backtrace.slo
/var/tmp/apache2/build/libtool --silent --mode=link gcc -o mod-backtrace.la  -rpath /var/tmp/apache2/modules -module -avoid-version    mod-backtrace.lo
/var/tmp/apache2/build/instdso.sh SH_LIBTOOL='/var/tmp/apache2/build/libtool' mod-backtrace.la /var/tmp/apache2/modules
/var/tmp/apache2/build/libtool --mode=install cp mod-backtrace.la /var/tmp/apache2/modules/
cp .libs/mod-backtrace.so /var/tmp/apache2/modules/mod-backtrace.so
chmod +x /var/tmp/apache2/modules/mod-backtrace.so
cp .libs/mod-backtrace.lai /var/tmp/apache2/modules/mod-backtrace.la
cp .libs/mod-backtrace.a /var/tmp/apache2/modules/mod-backtrace.a
ranlib /var/tmp/apache2/modules/mod-backtrace.a
chmod 644 /var/tmp/apache2/modules/mod-backtrace.a
----------------------------------------------------------------------
Libraries have been installed in:
   /var/tmp/apache2/modules

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - use the `-RLIBDIR' linker flag

See any operating system documentation about shared libraries for
more information, such as the ld(1) and ld.so(8) manual pages.
----------------------------------------------------------------------
chmod 755 /var/tmp/apache2/modules/mod-backtrace.so

Once apxs finishes it’s business, you can load the module and enable the exception handler with the following directives:

$ grep ‘(LoadModule|EnableExceptionHook)’ httpd.conf

LoadModule backtrace_module modules/mod_backtrace.so
EnableExceptionHook On

Once the module is enabled, each time Apache dies from a critical signal, a stack backtrace will be written to the error-log (you can also use the BacktraceLog directive to send backtraces to a separate logfile):

$ kill -SIGSEGV 8413

$ cat backtrace.log

[Thu Dec 29 11:24:36 2005] pid 8413 mod_backtrace backtrace for sig 6 (thread “pid” 8413)
[Thu Dec 29 11:24:36 2005] pid 8413 mod_backtrace main() is at 325b8
/var/tmp/apache2/modules/mod-backtrace.so:bt_exception_hook+0x108
/var/tmp/apache2/bin/httpd:ap_run_fatal_exception+0x34
/var/tmp/apache2/bin/httpd:0x29f58
/lib/libc.so.1:0xbfec8
/lib/libc.so.1:0xb4ff4
/lib/libc.so.1:_so_accept+0x8 [ Signal 6 (ABRT)]
/var/tmp/apache2/bin/httpd:unixd_accept+0x10
/var/tmp/apache2/bin/httpd:0x1c1f0
/var/tmp/apache2/bin/httpd:0x1c4b8
/var/tmp/apache2/bin/httpd:0x1c588
/var/tmp/apache2/bin/httpd:ap_mpm_run+0x76c
/var/tmp/apache2/bin/httpd:main+0x63c
/var/tmp/apache2/bin/httpd:_start+0x5c
[Thu Dec 29 11:24:36 2005] pid 8413 mod_backtrace end of backtrace

This is some c-weeeeeet livin’!