Posts in Monitoring

Getting syslog-ng to filter messages by source IP address

monitoringMar 2, 2010 2 min read

I received a call this week from one of our network guys because messages from several network devices weren't being logged by our centralized log server. When I started debugging the issue, I noticed that traffic from the hosts (host1 in this example) was making it to our syslog-ng server: While the traffic was making it to the server, the syslog messages were not being matched against a rule we had defined. Being the curious guy I am, I decided to read the syslog RFC to get a better understanding of the syslog message format. If you aren't familiar with syslog messages, they take the following form: The PRI section contains the facility and priority, the HEADER section contains a timestamp and the hostname or IP address of the device, and the MSG section contains the message the host is trying to send to the server…

$ read more →

earlier

Using Net-SNMP to monitor processes and execute Perl scripts by hitting a MIB

monitoringNov 14, 2009 2 min

One really cool feature about net-snmp is that it can monitor processes on a system. If snmpd notices that a process is no longer running, you can specify a script to be executed which can correct the problem. Lets take a look at some examples found here At least one web server process must be running at all times: proc httpd procfix httpd /etc/rc.d/init.d/httpd restart There should never be more than 10 mail processes running: proc sendmail 10 procfix sendmail /etc/rc.d/init.d/sendmail stop There should be a single network management agent running ("There can be only one"): proc snmpd 1 1 This can be an interesting use of Puppet, Chef, or CFengine, which are configuration management engines. Depending upon the type of host (webserver, mail server, etc.) you could set up and establish different types of snmpd.conf files for that environment…

$ read more →

Net-snmp should now work in an opensolaris non-global zone

monitoringMay 10, 2009 1 min

While debugging a net-snmp issue a while back, I came across the following error: error on subcontainer 'interface container' insert (-1) These errors are caused by opensolaris bug 6640675 which causes all interfaces to be assigned an index value of 0 (this leads net-snmp to think there are duplicate interfaces). The fix was just integrated into Nevada, so hopefully the code will be back ported to Solaris 10.

$ read more →

Net-snmp returns zeros for various UDP and TCP mibII data

monitoringApr 16, 2009 1 min

While testing out the latest net-snmp bits, I noticed that various TCP and UDP mibII OIDs would continuosly display zeros on my Solaris 10 update 6 host: Generate lots of UDP traffic. This turns out to be caused by a net-snmp bug, which is fixed by applying the patch attached to the bug report. I am posting this here to help others who may bump into this issue.

$ read more →

Debugging net-snmp problems

monitoringApr 16, 2009 2 min

I spent a fair amount of time debugging a bizarre net-snmp issue yesterday, and learned a TON about how net-snmp is implemented. While reading through the net-snmp code, I came across a number of macros similar to the following: The first argument to the DEBUGMSGCTL() macros contains a token name, which can be passed to the snmpd daemon's "-D" option to get verbose debugging data: To find the tokens that are available, you can bust out the trusty find utility: While the debugging output is a bit primitive, it is extremely useful when you can compare it side-by-side with the net-snmp source code. This helped me locate and fix an annoying bug (data is incorrect on Solaris 10 u4+ hosts), which allowed me to roll out the new version of the code to various hosts (the new version fixes a couple of bugs that lead to the daemon hanging after a period of time). Debugging is a bunch of fun, and there is nothing better than finding a solution to an issue!

$ read more →