Debugging net-snmp problems


I spent a fair amount of time debugging a bizarre net-snmp issue yesterday, and learned a TON about how net-snmp is implemented. While reading through the net-snmp code, I came across a number of macros similar to the following:

DEBUGMSGTL(("mibII/udpScalar", "Initialising UDP scalar group "));

The first argument to the DEBUGMSGCTL() macros contains a token name, which can be passed to the snmpd daemon’s “-D” option to get verbose debugging data:

$ snmpd -c /etc/snmpd.conf -p /var/run/snmpd.pid -Lo -q -V -f -D

kernel_sunos5,mibII/udpScalar

Received SNMP packet(s) from UDP: [0.0.0.0]->[1.2.3.4.102]:-14599
GETNEXT message
-- UDP-MIB::udpInDatagrams
kernel_sunos5: getMibstat (10, *, 40, 0, *, *)
kernel_sunos5: ... cache_valid 0 time 0 ttl 30 now 1239840358
kernel_sunos5: ...... getmib (263, 0, ...)
kernel_sunos5:dlpi: calling getentry aac: req_type: 0, buf: 24, entrysize: 40
kernel_sunos5: bad cache length 24 - not multiple of entry size 40
kernel_sunos5: ...... getmib returns 2
kernel_sunos5: ... result 1 rc 2
kernel_sunos5: ... getMibstat returns 1
mibII/udpScalar: getMibstat call from udp_load : MIB_UDP 10, sizeof(mib2_udp_t): 40
mibII/udpScalar: Loaded UDP scalar Group (solaris)
mibII/udpScalar: Handler - mode GETmibII/udpScalar: oid: UDP-MIB::udpInDatagrams.0Received SNMP packet(s) from UDP: [0.0.0.0]->[1.2.3.4]:-14599

To find the tokens that are available, you can bust out the trusty find utility:

$ cd net-snmp-5.4.2.1

$ find . -type f | xargs grep DEBUGMSG |more

./agent/agent_handler.c: DEBUGMSGTL(("handler::register", "Registering %s (", reginfo->handlerName));
./agent/agent_handler.c: DEBUGMSG(("handler::register", "::%s", handler->handler_name));
./agent/agent_handler.c: DEBUGMSG(("handler::register", ") at "));
......

While the debugging output is a bit primitive, it is extremely useful when you can compare it side-by-side with the net-snmp source code. This helped me locate and fix an annoying bug (data is incorrect on Solaris 10 u4+ hosts), which allowed me to roll out the new version of the code to various hosts (the new version fixes a couple of bugs that lead to the daemon hanging after a period of time). Debugging is a bunch of fun, and there is nothing better than finding a solution to an issue!

This article was posted by Matty on 2009-04-16 22:34:00 -0400 -0400