While catching up on the DTrace discussion list tonight, I came across a cool DTrace debugging recommendation from Adam Leventhal. Adam mentioned using the DTRACE_DEBUG flag, which will cause libdtrace to spew tons of debugging data each time dtrace is invoked:
$ export DTRACE_DEBUG=1
$ dtrace -n 'syscall:::entry { @calls[execname] = count(); }'
libdtrace DEBUG: opened provider fasttrap
libdtrace DEBUG: opened provider fbt
libdtrace DEBUG: opened provider lockstat
libdtrace DEBUG: opened provider profile
libdtrace DEBUG: opened provider sdt
libdtrace DEBUG: opened provider systrace
libdtrace DEBUG: opened 64-bit module unix (/platform/sun4u/kernel/sparcv9/unix) [0]
libdtrace DEBUG: opened 64-bit module krtld (misc/sparcv9/krtld) [1]
libdtrace DEBUG: opened 64-bit module genunix (/platform/sun4u/kernel/sparcv9/genunix) [2]
libdtrace DEBUG: opened 64-bit module platmod (/platform/SUNW,Ultra-5_10/kernel/misc/sparcv9/platmod) [3]
libdtrace DEBUG: opened 64-bit module SUNW,UltraSPARC-IIi (/platform/sun4u/kernel/cpu/sparcv9/SUNW,UltraSPARC-IIi) [4]
libdtrace DEBUG: opened 64-bit module specfs (/kernel/fs/sparcv9/specfs) [6]
libdtrace DEBUG: opened 64-bit module dtrace (/kernel/drv/sparcv9/dtrace) [7]
[ ...... ]
dtrace: description 'syscall:::entry ' matched 225 probes
^C
utmpd 2
fmd 4
inetd 4
svc.configd 5
svc.startd 5
sshd 16
nscd 77
ipmon 94
mysqld 149
dtrace 1784
libdtrace DEBUG: dt_buf_destroy(section headers): size=1024 resizes=1
libdtrace DEBUG: dt_buf_destroy(string table): size=512 resizes=0
libdtrace DEBUG: dt_buf_destroy(loadable data): size=512 resizes=0
libdtrace DEBUG: dt_buf_destroy(unloadable data): size=512 resizes=0
libdtrace DEBUG: dt_buf_destroy(probe data): size=512 resizes=0
libdtrace DEBUG: dt_buf_destroy(probe args): size=512 resizes=0
libdtrace DEBUG: dt_buf_destroy(probe offs): size=512 resizes=0
libdtrace DEBUG: dt_buf_destroy(probe rels): size=512 resizes=0
This is super cool, and useful for tracking down problems, or learning more about what DTrace does behind the scenes.
Solaris 10 ships with the IP filter firewall package, which is a fast, flexible and verstile firewall engine. IP filter by default will read /etc/ipf/ipf.conf to get the list of rules (e.g., pass in all) to apply to traffic as it traverses the interfaces in a system, and these rules will be applied in a top down fashion. If rules are not optimally ordered and grouped, CPU cycles will be wasted, since IP filter will have to check the packet against a set of rules that don’t apply to the type of traffic that is passing through the interface. The following rule set is a good example of a poorly written IP filter configuration file:
# Allow all packets to flow on the loopback interface (wish we had skip
on here)
pass in quick on lo0 all
pass out quick on lo0 all
# Set up default block all policy for eri0
block in log on eri0 all
# Set up default block all policy for eri1
block in log on eri1 all
# Apply rules
pass in quick on eri0 proto tcp from any to 192.168.1.3/32 port = 22
keep state
pass in quick on eri0 proto udp from any to 192.168.1.3/32 port = 53
keep state
pass in quick on eri0 proto tcp from any to 192.168.1.3/32 port = 80
keep state
pass in quick on eri0 proto tcp from any to 192.168.1.3/32 port = 110
keep state
pass in quick on eri0 proto tcp from any to 192.168.1.3/32 port = 143
keep state
pass in quick on eri0 proto tcp from any to 192.168.1.3/32 port = 443
keep state
pass in quick on eri1 proto tcp from any to 192.168.1.3/32 port = 993
keep state
pass in quick on eri1 proto tcp from any to 192.168.1.3/32 port = 995
keep state
When a sytem using this rule set receives a new connection on TCP port 995 (secure POP), the IP Filter firewall engine will need to evaluate 8 rules prior to allowing the connection through. As the number of rules increases, the CPU overhead required to enforce policy can quickly exceed the resources in a system. IP filter solves this problem by allowing rules to be grouped and assigned to an object (e.g, network interface) with the “group” statement. The following example shows how to arrange the above rules with better performing group statements:
# Allow all packets to flow on the loopback interface (wish we had skip
on here)
pass in quick on lo0 all
pass out quick on lo0 all
# Set up default block all policy for eri0, and associate rule group
100 to interface
block in log on eri0 all head 100
# Set up default block all policy for eri1, and associate rule group
200 to interface
block in log on eri1 all head 200
# Associate the following rules with group 100, and order by usage
pass in quick proto udp from any to 192.168.1.3/32 port = 53 keep state
group 100
pass in quick proto tcp from any to 192.168.1.3/32 port = 80 keep state
group 100
pass in quick proto tcp from any to 192.168.1.3/32 port = 443 keep
state group 100
pass in quick proto tcp from any to 192.168.1.3/32 port = 110 keep
state group 100
pass in quick proto tcp from any to 192.168.1.3/32 port = 143 keep
state group 100
pass in quick proto tcp from any to 192.168.1.3/32 port = 22 keep state
group 100
Associate the following rules with group 200, and order by usage
pass in quick proto tcp from any to 192.168.1.3/32 port = 993 keep
state group 200
pass in quick proto tcp from any to 192.168.1.3/32 port = 995 keep
state group 200
When this rule set is used, a connection to TCP port 995 will be matched by the second rule in group 200. Since we avoided six rule checks in this example, IP filter had to perform less work to get the same results (an allowed connection). This might not seem like a big deal on a workstation or lightly loaded server, but firewalls that handle thousands of connections each minute can definitely benefit from group statements and rule placement optimizations.
If you live in the Atlanta, GA area, you might be interested to know that Jarod Jenson will be talking about DTrace April 19th at the Marriott Perimeter Center. The event is being put on by Sun FREE of charge, and will be way better than any training course you could take on the subject ( there is one exception; a class by Brendan Gregg would be just as awesome )! Sun is requiring pre-registration to ensure that enough seats are available, and the meeting will be held at a swank location! I plan to wander to the event with Clay, and I am hoping to learn more about DTrace, the DVM provider, debugging and DTrace implementation details. Jarod is one of the smartest people at Sun, and the Atlanta SysAdmin community is truly fortunate to get him for a day. If you can make time, I HIGHLY recommend wandering to the Marriott to hear Jarod talk! You WILL NOT be disappointed!
I posted a blog entry a while back that showed how to setup smartd on a Solaris system. Two individuals left comments indicating that smartmontools wouldn’t build, and one individual (Peter) received the following errors during the build process:
gcc -g -O2 -Wall -W -c `test -f 'os_solaris_ata.s' || echo './'`os_solaris_ata.s
gcc -g -O2 -Wall -W -o smartd smartd.o atacmdnames.o atacmds.o ataprint.o knowndrives.o scsicmds.o scsiprint.o utility.o o
s_solaris.o os_solaris_ata.o -lnsl
ld: fatal: relocation error: R_SPARC_32: file os_solaris_ata.o: symbol : offset 0xfea5feef is non-aligned
ld: fatal: relocation error: R_SPARC_32: file os_solaris_ata.o: symbol : offset 0xfea5fef5 is non-aligned
ld: fatal: relocation error: R_SPARC_32: file os_solaris_ata.o: symbol : offset 0xfea5fef9 is non-aligned
ld: fatal: relocation error: R_SPARC_32: file os_solaris_ata.o: symbol : offset 0xfea5fefd is non-aligned
ld: fatal: relocation error: R_SPARC_32: file os_solaris_ata.o: symbol : offset 0xfea72806 is non-aligned
ld: fatal: relocation error: R_SPARC_32: file os_solaris_ata.o: symbol : offset 0xfea77436 is non-aligned
collect2: ld returned 1 exit status
I was able to recreate the problem on my Sun Ultra 10 running Solaris 10. To get smartmontools to compile, I ran “configure,” “make,” and then waited for the build process to fail. Once the build errored out, I removed the “-g” option from the gcc options and executed the following statements by hand:
$ gcc -O2 -Wall -W -ctest -f ‘os_solaris_ata.s’ || echo './'os_solaris_ata.s**
$ make
Since os_solaris_ata.s is a hand coded SPARC assembly file, I am not 100% certain why the smartmontools authors are trying to add debugging symbols to the object file. I will need to do some additional digging to find the answer.
Large dynamically linked executables can have a LOT of dependencies, which are resolved by the runtime linker when a program is executed. To see which libraries an executable and the executable’s shared dependencies depend on, the ldd utility can be used:
$ ldd /usr/sbin/metastat
libmeta.so.1 => /lib/libmeta.so.1
libc.so.1 => /lib/libc.so.1
libnsl.so.1 => /lib/libnsl.so.1
libadm.so.1 => /lib/libadm.so.1
libdevid.so.1 => /lib/libdevid.so.1
libgen.so.1 => /lib/libgen.so.1
libefi.so.1 => /lib/libefi.so.1
libdevinfo.so.1 => /lib/libdevinfo.so.1
libscf.so.1 => /lib/libscf.so.1
libmp.so.2 => /lib/libmp.so.2
libmd5.so.1 => /lib/libmd5.so.1
libuuid.so.1 => /lib/libuuid.so.1
libnvpair.so.1 => /lib/libnvpair.so.1
libdoor.so.1 => /lib/libdoor.so.1
libuutil.so.1 => /lib/libuutil.so.1
libsocket.so.1 => /lib/libsocket.so.1
libm.so.2 => /lib/libm.so.2
/platform/SUNW,Ultra-5_10/lib/libc_psr.so.1
/platform/SUNW,Ultra-5_10/lib/libmd5_psr.so.1
To view just the shared libraries that the executable depends on, the elfdump “-d” (dump the contents of the dynamic symbol table) option can be used:
$ /usr/ccs/bin/elfdump -d /usr/sbin/metastat | grep NEEDED
[0] NEEDED 0x867 libmeta.so.1
[1] NEEDED 0x884 libc.so.1
In the examples above, metastat only requires two libraries, but those two libraries have numerous dependencies.