Blog O’ Matty

While checking my e-mail recently, I noticed that my inbox contained several new messages that were roughly 3MB in size. The quantity of messages was choking my poor text-based e-mail client, so I needed to figure out what has happening. After spending a few minutes reviewing my maillog, I noticed that my mail retrieval program was erroring out while popping the 3MB message (it transferred the message, but wasn’t issuing a dele when it finished). Since my mail retrieval program was having issues retrieivng the 3MB file, I decided to fire up openssl and delete the message myself:

$ openssl s_client -quiet -connect mail.prefetch.net:995

depth=0 /C=US/O=mail.prefetch.net/OU=https://services.choicepoint.net/get.jsp?1605445126/OU=See www.rapidssl.com/cps (c)04/OU=Domain Control Validated - StarterSSL(TM)/CN=mail.prefetch.net
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /C=US/O=mail.prefetch.net/OU=https://services.choicepoint.net/get.jsp?1605445126/OU=See www.rapidssl.com/cps (c)04/OU=Domain Control Validated - StarterSSL(TM)/CN=mail.prefetch.net
verify error:num=27:certificate not trusted
verify return:1
depth=0 /C=US/O=mail.prefetch.net/OU=https://services.choicepoint.net/get.jsp?1605445126/OU=See www.rapidssl.com/cps (c)04/OU=Domain Control Validated - StarterSSL(TM)/CN=mail.prefetch.net
verify error:num=21:unable to verify the first certificate
verify return:1

+OK mail.prefetch.net Cyrus POP3 v2.2.12 server ready <1499164100.1140385444@mail.prefetch.net>

user myaccount
+OK Name is a valid mailbox

pass MYSPECIALPASSWORD
+OK Mailbox locked and ready

list
+OK scan listing follows
1 1893014
2 599397
3 4219
.

dele 1
+OK message deleted

quit
+OK
read:errno=0

I really dig openssl, and find myself using it daily. Giddie up!

While reading through pollstar.com, I read that Jerry Cantrell is reforming Alice In Chains with a new lead singer. This is one of the happiest moments in my life, and I am going to brave my fear of planes to wander to several AIC shows!! Rock on!

While setting up some machines to use one of Penn State’s NTP sources, I was curious where their master time source was located. After reading a bunch of NTP documentation, I came across the ntptrace utility:

$ ntptrace -dv clock.psu.edu

DoTransmit(128.118.25.3)
DoTransmit to 128.118.25.3
ReceiveBuf(128.118.25.3, 128.118.25.3)
server 128.118.25.3, port 123
stratum 16, precision -16, leap 00
refid otc1.psu.edu delay 0.05704, dispersion 0.00000 offset 233.148443
rootdelay 0.00020, rootdispersion 0.44749, synch dist 0.44759
reference time:      c7acb55e.088e4755  Sun, Feb 26 2006 18:16:46.033
originate timestamp: c7acb95f.f156e264  Sun, Feb 26 2006 18:33:51.942
transmit timestamp:  c7acb876.c4037000  Sun, Feb 26 2006 18:29:58.765

DoTransmit(128.118.25.5)
DoTransmit to 128.118.25.5
ReceiveBuf(128.118.25.5, 128.118.25.5)
server 128.118.25.5, port 123
stratum 1, precision -29, leap 00
refid 'WWV' delay 0.03889, dispersion 0.00000 offset 233.022950
rootdelay 0.00000, rootdispersion 0.22423, synch dist 0.22423
reference time:      c7acb95d.72909aed  Sun, Feb 26 2006 18:33:49.447
originate timestamp: c7acb95f.de6d04e6  Sun, Feb 26 2006 18:33:51.868
transmit timestamp:  c7acb876.d3904000  Sun, Feb 26 2006 18:29:58.826

This nifty little utility will print the stratum for each source between the client and the master, and will also decode and print the contents of each NTP message (the NTP RFC describes how each timestamp is used to compute the offset ) it receives. This is some sweeeeet livin’!

If you happen to read my blog, you may have noticed that I dig alternative music. One day while searching for cool new alternative music, I stumbled across pandora.com. This nifty site provides a medium to connect people to music based on a variety of musical heuristics, and it works really really well. If you get some time, check them out! You won’t be sorry!

While chatting with a friend last week, he mentioned that he uses the apachectl restart option to get Apache to reread configuration files. This works, but it forcefully terminates all children (even ones processing requests), and can lead to user sessions getting blotched. As of Apache 1.3, you can now use the apachectl graceful option to gracefully restart a web server:

$ apachectl graceful

This utilizes the generation values in the Apache scoreboard, and allows each child to die once they complete the request processing loop. If you are interested in seeing how this is done, check out prefork.c in the Apache source code.

While reading up on website performance monitoring applications last week, I came across the cricket HTTP-performance module. HTTP-performance allows you to graph the time it takes to connect to a website and to render a page. This sounded interesting, so I decided to download and install cricket. After reading through the beginners guide, I installed all of the required Perl modules, and proceeded to run the cricket compile script:

$ ./compile

Can't load '/usr/perl5/5.8.4/lib/i86pc-solaris-64int/auto/DB_File/DB_File.so' for module DB_File: ld.so.1: /usr/perl5/5.8.4/bin/perl: fatal: libdb-4.2.so: open failed: No such file or
directory at /usr/perl5/5.8.4/lib/i86pc-solaris-64int/XSLoader.pm line 68.
 at /usr/perl5/5.8.4/lib/i86pc-solaris-64int/DB_File.pm line 251
Compilation failed in require at /home/rmatteso/cricket-1.0.5/lib/ConfigTree/Node.pm line 25.
BEGIN failed--compilation aborted at /home/rmatteso/cricket-1.0.5/lib/ConfigTree/Node.pm line 25.
Compilation failed in require at ./compile line 39.
BEGIN failed--compilation aborted at ./compile line 39.I 

Ack!!!! — it looks like DB_File.so can’t find the BerkelyDB shared library:

$ ldd /usr/perl5/5.8.4/lib/i86pc-solaris-64int/auto/DB_File/DB_File.so

        libdb-4.2.so =>  (file not found)
        libc.so.1 =>     /lib/libc.so.1
        libm.so.2 =>     /lib/libm.so.2

Since I hate using LD_LIBRARY_PATH, I decided to adjust the link line to include a hard coded runtime path. To determine which variable to edit, I searched the Makefile until I came to the following line:

$ less Makefile

$(LD) $(LDDLFLAGS) $(LDFROM) $(OTHERLDFLAGS) -o $@ $(MYEXTLIB) $(PERL_ARCHIVE) \ $(LDLOADLIBS) $(PERL_ARCHIVE_AFTER) $(EXPORT_LIST) $(INST_DYNAMIC_FIX)

After locating this line, I added “-R/usr/sfw/lib:/usr/local/BerkeleyDB/lib” to the LDDLFLAGS variable and rebuilt the Perl module:

$ make

cc -c  -I/usr/local/BerkeleyDB/include  -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_TS_ERR
NO     -DVERSION="1.814"  -DXS_VERSION="1.814"  "-I/usr/perl5/5.8.4/lib/i86pc-solaris-64
int/CORE"  -D_NOT_CORE  -DmDB_Prefix_t=size_t -DmDB_Hash_t=u_int32_t version.c
cc -c  -I/usr/local/BerkeleyDB/include  -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_TS_ERR
NO     -DVERSION="1.814"  -DXS_VERSION="1.814"  "-I/usr/perl5/5.8.4/lib/i86pc-solaris-64
int/CORE"  -D_NOT_CORE  -DmDB_Prefix_t=size_t -DmDB_Hash_t=u_int32_t DB_File.c
Running Mkbootstrap for DB_File ()
chmod 644 DB_File.bs
rm -f blib/arch/auto/DB_File/DB_File.so
LD_RUN_PATH="/usr/local/BerkeleyDB/lib"
cc  -G -R/usr/sfw/lib:/usr/local/BerkeleyDB/lib version.o DB_File.o  -o blib/arch/auto/DB_File/DB_File.so   -L/usr/local/BerkeleyDB/lib -ldb
chmod 755 blib/arch/auto/DB_File/DB_File.so
cp DB_File.bs blib/arch/auto/DB_File/DB_File.bs
chmod 644 blib/arch/auto/DB_File/DB_File.bs
Manifying blib/man3/DB_File.3

$ make install

Installing /usr/perl5/5.8.4/lib/i86pc-solaris-64int/auto/DB_File/DB_File.so
Files found in blib/arch: installing files in blib/lib into architecture dependent library tree
Installing /usr/perl5/5.8.4/man/man3/DB_File.3
Writing /usr/perl5/5.8.4/lib/i86pc-solaris-64int/auto/DB_File/.packlist
Appending installation info to /usr/perl5/5.8.4/lib/i86pc-solaris-64int/perllocal.pod

Everything seemed to work, but did the runpath get set correctly? A quick check revealed that it did:

$ dump -Lv /usr/perl5/5.8.4/lib/i86pc-solaris-64int/auto/DB_File/DB_File.so

/usr/perl5/5.8.4/lib/i86pc-solaris-64int/auto/DB_File/DB_File.so:

  **** DYNAMIC SECTION INFORMATION ****
.dynamic:
[INDEX] Tag         Value
[1]     NEEDED          libdb-4.2.so
[2]     NEEDED          libc.so.1
[3]     INIT            0xe030
[4]     FINI            0xe050
[5]     RUNPATH         /usr/sfw/lib:/usr/local/BerkeleyDB/lib
[6]     RPATH           /usr/sfw/lib:/usr/local/BerkeleyDB/lib

This was a fun experience, since it finally forced me to sit down and read through the ELF standard (great read if you are into that kind of stuff).

While perusing digg.com this week, I came across a link to numerous awesome lectures at Berkeley University. The lectures are given by professors at Berkeley, and include courses on Biology, Chemistry, Computer Science (the OS course is awesome), Economics, History, Physics and Electrical Engineering. Good stuff!

While perusing the opensolaris-dtrace discussion list this week, I came across Jonathon Adams recommendation to LD_PRELOAD the Solaris libumem library to find the source of a memory leak. Since I have never taken the opportunity to play with libumem, I decided to test it out this weekend. To get started, I created a C program that leaked memory. Once I verified the program was working as expected, I preloaded the libumem shared library, and executed the program:

$ LD_PRELOAD=libumem.so UMEM_DEBUG=default ./leaky
Leaking 10 bytes of memory
Leaking 10 bytes of memory
Leaking 10 bytes of memory
Leaking 10 bytes of memory
Leaking 10 bytes of memory
Leaking 10 bytes of memory

After the program leaked a few bytes of memory, I used the Solaris gcore utility to generate a core file with the programs memory state:

$ gcore `pgrep leaky`
gcore: core.10973 dumped

This will produce a core file with memory debugging data, which can be analyze by the Solaris modular debugger “::findleaks” dcmd:

$ mdb core.10973

Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]

> ::findleaks -dv
findleaks:                maximum buffers => 22
findleaks:                 actual buffers => 9
findleaks:
findleaks:             potential pointers => 23567
findleaks:                     dismissals => 17080         (72.4%)
findleaks:                         misses => 6209          (26.3%)
findleaks:                           dups => 274           ( 1.1%)
findleaks:                        follows => 4             ( 0.0%)
findleaks:
findleaks:              elapsed wall time => 0 seconds
findleaks:
CACHE     LEAKED   BUFCTL CALLER
00029c08       5 0003e000 leakingmemory+0xc
———————————————————————-
   Total       5 buffers, 120 bytes

umem_alloc_24 leak: 5 buffers, 24 bytes each, 120 bytes total
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
           3e000            37fc0    33ec48f6751b4                1
                            29c08                0                0
                 libumem.so.1`umem_cache_alloc+0×13c
                 libumem.so.1`umem_alloc+0×60
                 libumem.so.1`malloc+0×28
                 leakingmemory+0xc
                 main+0×18
                 _start+0×5c

In this example you can see that we leaked 5-buffers at address leakingmemory+0xc, which happens to be the location where my program is leaking memory (leakingmemory is called from a loop in main):

void leakingmemory(int bytes)
{
   char *oof = (char *)malloc(bytes);
   int i;

   printf("Leaking %d bytes of memory\n",bytes);
   for (i = 0; i < bytes; i++) {
        oof[i] = 1;
   }
}

This is some amazingly useful stuff, and from the documentation I read it can be used to detect problems on production systems. Libumem can also provide some substantial performance benefits, which are covered in the USENIX paper Magazines and Vmem: Extending the Slab Allocator to Many CPUs and Arbitrary Resources.