Getting Apache backtraces with mod-backtrace

While perusing the web last week I came across Jeff Trawick’s mod-backtrace module. This module can be used to print a stack backtrace each time Apache receives a critical signal (e.g., SIGSEGV, SIGBUS, etc.), and can be an invaluable resource for locating modules that are misbehaving. Since the module only worked with Linux and FreeBSD, I created a patch to allow it to work on Solaris systems. To install and configure mod-backtrace on a Solaris system, you will need to build Apache with the “–enable-exception-hook” option, download mod-backtrace.c from Jeff’s website, apply the patch, and compile mod-backtrace with the apxs utility:

$ wget

$ /usr/bin/patch -p0 < mod-backtrace.diff

  Looks like a unified context diff.

$ apxs -ci mod-backtrace.c

/var/tmp/apache2/build/libtool --silent --mode=compile gcc -prefer-pic  -DAP_HAVE_DESIGNATED_INITIALIZER -DSOLARIS2=10 -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -g -O2 -I/var/tmp/apache2/include  -I/var/tmp/apache2/include   -I/var/tmp/apache2/include   -c -o mod-backtrace.lo mod-backtrace.c && touch mod-backtrace.slo
/var/tmp/apache2/build/libtool --silent --mode=link gcc -o  -rpath /var/tmp/apache2/modules -module -avoid-version    mod-backtrace.lo
/var/tmp/apache2/build/ SH_LIBTOOL='/var/tmp/apache2/build/libtool' /var/tmp/apache2/modules
/var/tmp/apache2/build/libtool --mode=install cp /var/tmp/apache2/modules/
cp .libs/ /var/tmp/apache2/modules/
chmod +x /var/tmp/apache2/modules/
cp .libs/mod-backtrace.lai /var/tmp/apache2/modules/
cp .libs/mod-backtrace.a /var/tmp/apache2/modules/mod-backtrace.a
ranlib /var/tmp/apache2/modules/mod-backtrace.a
chmod 644 /var/tmp/apache2/modules/mod-backtrace.a
Libraries have been installed in:

If you ever happen to want to link against installed libraries
in a given directory, LIBDIR, you must either use libtool, and
specify the full pathname of the library, or use the `-LLIBDIR'
flag during linking and do at least one of the following:
   - add LIBDIR to the `LD_LIBRARY_PATH' environment variable
     during execution
   - use the `-RLIBDIR' linker flag

See any operating system documentation about shared libraries for
more information, such as the ld(1) and manual pages.
chmod 755 /var/tmp/apache2/modules/

Once apxs finishes it’s business, you can load the module and enable the exception handler with the following directives:

$ grep ‘(LoadModule|EnableExceptionHook)’ httpd.conf

LoadModule backtrace_module modules/
EnableExceptionHook On

Once the module is enabled, each time Apache dies from a critical signal, a stack backtrace will be written to the error-log (you can also use the BacktraceLog directive to send backtraces to a separate logfile):

$ kill -SIGSEGV 8413

$ cat backtrace.log

[Thu Dec 29 11:24:36 2005] pid 8413 mod_backtrace backtrace for sig 6 (thread “pid” 8413)
[Thu Dec 29 11:24:36 2005] pid 8413 mod_backtrace main() is at 325b8
/lib/ [ Signal 6 (ABRT)]
[Thu Dec 29 11:24:36 2005] pid 8413 mod_backtrace end of backtrace

This is some c-weeeeeet livin’!