Earlier this month I described how Orca could be used to graph JVM utilization. In order to get this solution working, you need to enable the Java SNMP agent. This can be a problem if you are running multiple SNMP agents on the server, and want both agents to use the standard SNMP port (UDP port 161). Fortunately the Solaris SNMP daemon (and anything derived from the net-snmp SNMP implementation) can be configured to proxy requests to one or more agents on the local machine (it can also proxy requests to remote systems).
To configure the Solaris SNMP daemon to proxy requests to a secondary agent, you can add one or more “proxy” statement to your snmpd.conf configuration file. The following example shows a sample snmpd.conf entry that can be used to proxy incoming requests for the JVM management MIB namespace to an agent that is bound to localhost on UDP port 8161, using version two of the SNMP protocol to communicate with this agent:
proxy -v 2c -c public localhost:8161 .1.3.6.1.4.1.42.2.145.3.163.1
As you can imagine, this opens up a number of interesting possibilities. This allows you to enforce access policies from a single location (the snmpd.conf configuration file), and limit the number of services that need to use the wildcard address when binding to a socket. Niiiiice!
It seems like just yesterday The Smashing Pumpkins announced that the band was breaking up, and the band members would be venturing out to do their own thing. James Iha joined The Perfect Circle, Billy Corgan and Jimmy Chamberlain formed Zwan, and I have no idea what Darcy ended up doing with herself (though – she quit the band prior to the band formally calling it quits). While these side projects were good, they definitely didn’t fill the void that was left when the pumpkins broke up. So when Billy Corgan took out a full page ad to announce that the pumpkins would be reuniting late last year, I was stoked, and planned to attend one or more of the reunion shows.
My first chance came this week when the pumpkins played a two night stint at the lovely Fox theatre. I was fortunate to get tickets to both shows, and am extremely glad I did! Billy opened night one with four acoustic songs, including my personal favorite “Disarm.” After the acoustic set, the other band members (Jimmy Chamberlain, Ginger Reyes and Jeff Schroeder) joined Billy and his electric guitar to play a number of new songs, as well as some of the classics. In addition to the four acoustic songs Billy played to open the show, the band played “Bullet with butterfly wings,” “Hummer,” “Drown,” “Glass and the ghost children,” “1979,” “Cherub Rock,” “Tonight Tonight,” “Today,” “Tarantula,” “Set the Ray to Jerry,” “Heavy metal machine,” “Perfect” and a few songs I didn’t recognize. The opening night was awesome, and I was hopeful night two would be just as good!
On night two, the band came out dressed in white and immediately went to town on what turned out to be one of the best sets I have ever heard them play. The band opened with their mega-hit “Today,” and followed that with “Stand inside your love,” “Ava Adore,” “To Sheila, “Bullet with butterfly wings,” “Tonight Tonight,” “1979,” “Cherub Rock,” “Tarantula,” “Bring the light,” “Rocket,” “Drown,” “Zero,” “Thirty-Three,” “Starla,” and a drawn out version of “Heavy metal machine” (the songs are not listed in the actual order they played them). Night two was definitely the better of the two nights, but I was a bit disappointed that they didn’t play “Disarm.” I can empathetically say the band sounded incredible live, and it appears they didn’t drop a step from when I saw them on their Machine fair-well tour several years ago. While I have no idea what the future holds for the pumpkins, hopefully Billy will decide to keep making and playing music under the pumpkins name. Viva la rock and roll!
I gave a presentation on the Solaris Fault Management Architecutre (FMA) at the Atlanta opensolaris users group meeting this evening. If your curious what was discussed, you can look through the slides that were used for the presentation. I would like to thank everyone for coming out, and for being such a great audience. I would also like to thank Scott Dickson for putting the meeting together, and for getting the attendees some tasty pizza!
During much of my IT career, I have needed to support SNMP in one form or another. Typically the companies I have worked for deploy an SNMP agent to each server, and a network management station periodically polls this agent to retrieve health information. Most of the SNMP daemons I have worked with run as the user root by default, which opens a big gaping whole in system security. The Solaris SNMP daemon is no different in this respect, though you can configure it to run as a non-privileged user. To do this, you can add the “agentuser” directive and the name of an unprivileged user to the snmpd.conf configuration:
$ grep agentuser /etc/sma/snmp/snmpd.conf
agentuser snmp
This directive will cause the daemon change it’s effective user id to the user snmp once it binds to UDP port 161. If you want to take this one step further, you can follow the directions in the Limiting Service Privileges in the SolarisTM 10 Operating System to alleviate the need to use root altogether.
The Sun JDK has included anSNMP agent since version 5, and the agent has a number of objects in its MIB that provide runtime utilization data. The current set of objects include thread utilization counters, garbage collection events, time spent performing garbage collection, heap usage, class loader activity, compiler metrics, etc. Graphing these values can help with performance analysis, since you can view activity over time. This data is also useful for capacity planningl, since the graphs can be used to size hardware for an application, and for forecasting future hardware upgrades.
Numerous tools are available to graph utilization data, but my personal favorite is Orca. Orca can graph arbitrary data, and is a perfect fit for generating JVM performance graphs. Configuring Orca to generate JVM performance graphs is a snap. To begin, you will first need to enable the Java SNMP agent. To enable the agent, you can add the “-Dcom.sun.management.snmp.interface” (interface to bind to), “-Dcom.sun.management.snmp.port” (use a different SNMP port other that the default, port 161), “-Dcom.sun.management.snmp.acl” (use ACLs to limit who can read and write to the server) and " -Dcom.sun.management.snmp.acl.file” (ACL file to use) options to the java command line:
$ java -Dcom.sun.management.snmp.interface=192.168.1.10 \ -Dcom.sun.management.snmp.port=8161 \ -Dcom.sun.management.snmp.acl=true \ -Dcom.sun.management.snmp.acl.file=/usr/java/jre/lib/management/snmp.acl
The $JAVA_HOME/jre/lib/management/ directory contains a sample SNMP ACL file (aptly named snmp.acl.template) that can be tailored to fit your site’s security policy (if you use the default locations, you do not need to specify the last two parameters). After the SNMP agent is enabled, you can run jvmstats.pl from cron to collect statistics at specific intervals. Each time Jvmstats.pl runs, it will write the current values of the Java agent’s performance counters (e.g., number of active threads, time spent performing garbage collection, etc.) to the file pased to the “-f” option, and will add a header to the file if it doesn’t exist.
The example below show how jvmstats.pl can be used to connect to server foo.prefetch.net on UDP port 8161, collect the current set of JVM statistics, and write these statistics to a file (the file name should match up with the regex used in the Orca find_files configuration stanza):
$ jvmstats.pl -s foo.prefetch.net -p 8161 -c public -f /home/matty/java/jvmstats/server1_myappjvm/javallator-2007-11-05
$ cat /home/matty/java/jvmstats/server1_myappjvm/javallator-2007-11-05
TIMESTAMP JVMUPTIME JITTIME THREADSTOTAL THREADSACTIVE CLASSLOADS CLASSUNLOADS
NGGCEVENTS OLDGCEVENTS NGGCTIME OLDGCTIME OBJECTFINALIZATION HEAPCOMMITTED
HEAPUSED NONHEAPCOMMITTED NONHEAPUSED
1194308951 218176652 735 308464 10 1348 0 380934 0 5309065 0 0
266797056 165481912 19136512 7156344
After a few samples are taken, the orca utility can be run with the javallator.cfg configuration file to graph the data collected above (if you haven’t used Orca before, you can peruse my monitoring LDAP article to see how to configure it). This will result in a set of graphs that look similar to the samples posted to my website. I really dig Java, and having historical performance graphs handy sure makes performance analysis a whole lot easier!