Finding memory leaks on Solaris systems

While perusing the opensolaris-dtrace discussion list this week, I came across Jonathon Adams recommendation to LD_PRELOAD the Solaris libumem library to find the source of a memory leak. Since I have never taken the opportunity to play with libumem, I decided to test it out this weekend. To get started, I created a C program that leaked memory. Once I verified the program was working as expected, I preloaded the libumem shared library, and executed the program:

$ LD_PRELOAD=libumem.so UMEM_DEBUG=default ./leaky
Leaking 10 bytes of memory
Leaking 10 bytes of memory
Leaking 10 bytes of memory
Leaking 10 bytes of memory
Leaking 10 bytes of memory
Leaking 10 bytes of memory

After the program leaked a few bytes of memory, I used the Solaris gcore utility to generate a core file with the programs memory state:

$ gcore `pgrep leaky`
gcore: core.10973 dumped

This will produce a core file with memory debugging data, which can be analyze by the Solaris modular debugger “::findleaks” dcmd:

$ mdb core.10973

Loading modules: [ libumem.so.1 libc.so.1 ld.so.1 ]

> ::findleaks -dv
findleaks:                maximum buffers => 22
findleaks:                 actual buffers => 9
findleaks:
findleaks:             potential pointers => 23567
findleaks:                     dismissals => 17080         (72.4%)
findleaks:                         misses => 6209          (26.3%)
findleaks:                           dups => 274           ( 1.1%)
findleaks:                        follows => 4             ( 0.0%)
findleaks:
findleaks:              elapsed wall time => 0 seconds
findleaks:
CACHE     LEAKED   BUFCTL CALLER
00029c08       5 0003e000 leakingmemory+0xc
----------------------------------------------------------------------
   Total       5 buffers, 120 bytes

umem_alloc_24 leak: 5 buffers, 24 bytes each, 120 bytes total
            ADDR          BUFADDR        TIMESTAMP           THREAD
                            CACHE          LASTLOG         CONTENTS
           3e000            37fc0    33ec48f6751b4                1
                            29c08                0                0
                 libumem.so.1`umem_cache_alloc+0x13c
                 libumem.so.1`umem_alloc+0x60
                 libumem.so.1`malloc+0x28
                 leakingmemory+0xc
                 main+0x18
                 _start+0x5c

In this example you can see that we leaked 5-buffers at address leakingmemory+0xc, which happens to be the location where my program is leaking memory (leakingmemory is called from a loop in main):

void leakingmemory(int bytes)
{
   char *oof = (char *)malloc(bytes);
   int i;

   printf("Leaking %d bytes of memory\n",bytes);
   for (i = 0; i < bytes; i++) {
        oof[i] = 1;
   }
}

This is some amazingly useful stuff, and from the documentation I read it can be used to detect problems on production systems. Libumem can also provide some substantial performance benefits, which are covered in the USENIX paper Magazines and Vmem: Extending the Slab Allocator to Many CPUs and Arbitrary Resources.

4 Comments

sudhakar  on July 20th, 2007

good explanation

seema  on December 24th, 2007

very well explained even novice could understand.

Amit Sutar  on August 21st, 2009

Hey Matty,
Very well written….. Cleared every doubt in my mind

Rachmat Febrianto  on December 30th, 2011

Very good explanation, thanks

Leave a Comment