When an application is running, it is sometimes useful to see which stack frame the application is currently in. This is especially useful when debugging hung processes, since you can see which function the program is currently executing, and the code path taken to get to that point. Both Linux and Solaris provide utilities to capture live stack traces from processes. The Solaris tool to grab this information is called pstack. Pstack takes a process id as an argument, and will print stack traces for each thread in the process id passed as an argument:
$ pstack 539
539: slapd -f /opt/openldap-common/etc/slapd.conf -u openldap -g openldap -
----------------- lwp# 1 / thread# 1 --------------------
fee412d0 lwp_wait (2, ffbffaac)
fee3ccd8 _thrp_join (2, 0, 0, 1, ffbffaac, fee6cbc0) + 34
00032f30 slapd_daemon (0, 130118, 0, 0, 0, 1) + cc
00023bb8 main (9, ffbffc2c, a0, 199920, 0, 199338) + ed4
000228b0 _start (0, 0, 0, 0, 0, 0) + 5c
----------------- lwp# 2 / thread# 2 --------------------
fee40e7c pollsys (f83fd000, 3, 0, 0)
fede25b0 pselect_large_fdset (f83fd018, 3, f83fd208, f83fd200, 20, f83fd20c) + 2ec
fede29f8 select_large_fdset (13, f83ffd88, 0, 0, 0, 199500) + a0
00032300 ???????? (134c00, f83ffb88, 0, 173c00, 170400, 176d80)
fee400b0 _lwp_start (0, 0, 0, 0, 0, 0)
----------------- lwp# 3 / thread# 3 --------------------
fee40150 lwp_park (0, 0, 0)
fee3a1e4 cond_wait_queue (1acbc8, 1acbb0, 0, 0, 0, 0) + 28
fee3a764 cond_wait (1acbc8, 1acbb0, 0, fe9f0000, 2280b8, 0) + 10
fee3a7a0 pthread_cond_wait (1acbc8, 1acbb0, 0, 0, 0, 3) + 8
000df5b4 ???????? (1acba8, f7c00000, 0, 0, df42c, 0)
fee400b0 _lwp_start (0, 0, 0, 0, 0, 0)
----------------- lwp# 4 / thread# 4 --------------------
fee40150 lwp_park (0, 0, 0)
fee3a1e4 cond_wait_queue (1acbc8, 1acbb0, 0, 0, 0, 0) + 28
fee3a764 cond_wait (1acbc8, 1acbb0, 0, fe9f0400, 2280b8, 0) + 10
fee3a7a0 pthread_cond_wait (1acbc8, 1acbb0, 0, 0, 0, 3) + 8
000df5b4 ???????? (1acba8, f7400000, 0, 0, df42c, 0)
fee400b0 _lwp_start (0, 0, 0, 0, 0, 0)
The Linux utility to grab comparable information is called gstack, and it works nearly identically to pstack:
$ gstack 2195
#0 0x00194402 in __kernel_vsyscall ()
#1 0x00469e7d in ___newselect_nocancel () from /lib/libc.so.6
#2 0x00cc1c5b in main () from /proc/2195/exe
These utilities have helped me time and time again, and are two extremely useful pieces of software.