While perusing the web last week I came across Jeff Trawick’s mod-backtrace module. This module can be used to print a stack backtrace each time Apache receives a critical signal (e.g., SIGSEGV, SIGBUS, etc.), and can be an invaluable resource for locating modules that are misbehaving. Since the module only worked with Linux and FreeBSD, I created a patch to allow it to work on Solaris systems. To install and configure mod-backtrace on a Solaris system, you will need to build Apache with the “–enable-exception-hook” option, download mod-backtrace.c from Jeff’s website, apply the patch, and compile mod-backtrace with the apxs utility:
$ wget http://people.apache.org/~trawick/mod_backtrace.c
$ /usr/bin/patch -p0 < mod-backtrace.diff
Looks like a unified context diff. done
$ apxs -ci mod-backtrace.c
/var/tmp/apache2/build/libtool --silent --mode=compile gcc -prefer-pic -DAP_HAVE_DESIGNATED_INITIALIZER -DSOLARIS2=10 -D_POSIX_PTHREAD_SEMANTICS -D_REENTRANT -g -O2 -I/var/tmp/apache2/include -I/var/tmp/apache2/include -I/var/tmp/apache2/include -c -o mod-backtrace.lo mod-backtrace.c && touch mod-backtrace.slo
/var/tmp/apache2/build/libtool --silent --mode=link gcc -o mod-backtrace.la -rpath /var/tmp/apache2/modules -module -avoid-version mod-backtrace.lo
/var/tmp/apache2/build/instdso.sh SH_LIBTOOL='/var/tmp/apache2/build/libtool' mod-backtrace.la /var/tmp/apache2/modules
/var/tmp/apache2/build/libtool --mode=install cp mod-backtrace.la /var/tmp/apache2/modules/
cp .libs/mod-backtrace.so /var/tmp/apache2/modules/mod-backtrace.so
chmod +x /var/tmp/apache2/modules/mod-backtrace.so
cp .libs/mod-backtrace.lai /var/tmp/apache2/modules/mod-backtrace.la
cp .libs/mod-backtrace.a /var/tmp/apache2/modules/mod-backtrace.a
ranlib /var/tmp/apache2/modules/mod-backtrace.a
chmod 644 /var/tmp/apache2/modules/mod-backtrace.a
----------------------------------------------------------------------
Libraries have been installed in:
/var/tmp/apache2/modules
If you ever happen to want to link against installed libraries in a given directory, LIBDIR, you must either use libtool, and specify the full pathname of the library, or use the `-LLIBDIR’ flag during linking and do at least one of the following:
chmod 755 /var/tmp/apache2/modules/mod-backtrace.so
Once apxs finishes it’s business, you can load the module and enable the exception handler with the following directives:
$ grep '(LoadModule|EnableExceptionHook)' httpd.conf
LoadModule backtrace_module modules/mod_backtrace.so
EnableExceptionHook On
Once the module is enabled, each time Apache dies from a critical signal, a stack backtrace will be written to the error-log (you can also use the BacktraceLog directive to send backtraces to a separate logfile):
$ kill -SIGSEGV 8413
$ cat backtrace.log
[Thu Dec 29 11:24:36 2005] pid 8413 mod_backtrace backtrace for sig 6
(thread "pid" 8413)
[Thu Dec 29 11:24:36 2005] pid 8413 mod_backtrace main() is at 325b8
/var/tmp/apache2/modules/mod-backtrace.so:bt_exception_hook+0x108
/var/tmp/apache2/bin/httpd:ap_run_fatal_exception+0x34
/var/tmp/apache2/bin/httpd:0x29f58
/lib/libc.so.1:0xbfec8
/lib/libc.so.1:0xb4ff4
/lib/libc.so.1:_so_accept+0x8 [ Signal 6 (ABRT)]
/var/tmp/apache2/bin/httpd:unixd_accept+0x10
/var/tmp/apache2/bin/httpd:0x1c1f0
/var/tmp/apache2/bin/httpd:0x1c4b8
/var/tmp/apache2/bin/httpd:0x1c588
/var/tmp/apache2/bin/httpd:ap_mpm_run+0x76c
/var/tmp/apache2/bin/httpd:main+0x63c
/var/tmp/apache2/bin/httpd:_start+0x5c
[Thu Dec 29 11:24:36 2005] pid 8413 mod_backtrace end of backtrace
This is some c-weeeeeet livin’!
I recently got sick of the Safari spinning ball, and decided to conduct some research to speed up my favorite web browser. After reading numerous posts on the Apple discussion board, I made the following changes to significantly boost page rendering time:
1. Add Mike’s Ad Blocking Hosts file to /etc/hosts and restart lookupd
2. Purge the Safari cache from Safari -> Clear Cache
3. Remove old RSS articles by clicking Preferences -> RSS -> Remove Now
Once these three actions were complete, Safari was back to post install speeds! Nice!
I came across my new favorite error message last night while debugging some issues with the Solaris printstack() function:
$ gdb -q /var/tmp/apache2/bin/httpd
(gdb) **shell ps -ef | grep gdb**
matty 9216 8963 1 20:30:34 pts/1 0:00 gdb -q
/var/tmp/apache2/bin/httpd
matty 9217 9216 0 20:30:41 pts/1 0:00 bash -c ps -ef | grep gdb
(gdb) **attach 9216**
Attaching GDB to itself is not a good idea...
You gotta love some geek humor. :)
If you have ever done a long listing of /usr/lib on a Linux system, you probably choked and asked yourself what the f$%^ is this mess? After reading through Peter Seebach’s article Dissecting Shared Libraries, things don’t seem so bad, and the large number of files actually starts to make sense. Step one in sorting out the library madness requires making sense of the digits (the major and minor revision numbers) that appear in the shared library file name. Peter’s article clarifies this with the following description:
“One of the potential advantages of dynamic linking, however, is in fixing bugs. It’d be nice if you could fix a bug in the library and not have to recompile a thousand programs to take advantage of that fix. So sometimes, you want to link to a newer version. Unfortunately, that creates some cases where you want to link to the newer version and some cases where you’d rather stick with an older version. There is a solution, though – two kinds of version numbers:
- A major number indicates a potential incompatibility between library versions.
- A minor number indicates only bug fixes.
So under most circumstances, it is safe to load a library with the same major number and a higher minor number; consider it an unsafe practice to load a library with a higher major number.”
This makes sense, and seems like a good thing for systems that don’t want to break applications. Now what about all those links in /usr/lib? Peter provides the following description:
To prevent users (and programmers) from needing to track library numbers and updates, the system comes with a large number of symbolic links. In general, the pattern is that
libexample.so
will be a link to
libexample.so.N
in which N is the highest major version number found on the system. For every major version number supported,
libexample.so.N
will be a link in turn to
libexample.so.N.M
in which M is the largest minor version number.
If you haven’t checked out IBM’s redbook or Linux developerworks sites, I highly recommend doing so. There are numerous awesome pieces of documentation spanning numerous technologies.
This past week I wanted to use the Solaris cfgadm utility to unconfigure a few LUNs. When I ran ‘cfgadm -al’, I noticed that the FC adaptors were not visible in the cfgadm output:
$ cfgadm -al
Ap_Id Type Receptacle Occupant Condition
c2 scsi-bus connected configured unknown
c2::dsk/c2t0d0 CD-ROM connected configured unknown
usb0/1 unknown empty unconfigured ok
usb0/2 unknown empty unconfigured ok
This seemed odd, since I could see the controllers in the vxdmpadm output:
$ vxdmpadm listctlr all
CTLR-NAME ENCLR-TYPE STATE ENCLR-NAME
=====================================================
c1 Disk ENABLED Disk
c0 Disk ENABLED Disk
c4 EMC ENABLED EMC0
c3 EMC ENABLED EMC0
c4 EMC_CLARiiON DISABLED EMC_CLARiiON0
c3 EMC_CLARiiON DISABLED EMC_CLARiiON0
Since the controllers in question were Emulex adaptors, I read through the Emulex admin guide and found that the platform and FC adaptor had to be DR aware to support configure/unconfigure operations. Since I couldn’t locate a “DR aware” label in our vendors documentation, I decided to open a ticket to see if the servers supported cfgadm. After a week or two of chatting with support, our vendor indicated that we would need to use the Sun Leadville drivers to configure and unconfigure LUNs with Emulex adaptors in Solaris systems. This was awesome news, and I am super happy that the lpfc driver will now be installed in /kernel/drv by default! Niiiiiiice.