Graphing JVM utilization with orca

The Sun JDK has included an SNMP agent since version 5, and the agent has a number of objects in its MIB that provide runtime utilization data. The current set of objects include thread utilization counters, garbage collection events, time spent performing garbage collection, heap usage, class loader activity, compiler metrics, etc. Graphing these values can help with performance analysis, since you can view activity over time. This data is also useful for capacity planningl, since the graphs can be used to size hardware for an application, and for forecasting future hardware upgrades.

Numerous tools are available to graph utilization data, but my personal favorite is Orca. Orca can graph arbitrary data, and is a perfect fit for generating JVM performance graphs. Configuring Orca to generate JVM performance graphs is a snap. To begin, you will first need to enable the Java SNMP agent. To enable the agent, you can add the “-Dcom.sun.management.snmp.interface” (interface to bind to), “-Dcom.sun.management.snmp.port” (use a different SNMP port other that the default, port 161), “-Dcom.sun.management.snmp.acl” (use ACLs to limit who can read and write to the server) and ” -Dcom.sun.management.snmp.acl.file” (ACL file to use) options to the java command line:

$ java -Dcom.sun.management.snmp.interface=192.168.1.10 \
        -Dcom.sun.management.snmp.port=8161 \
        -Dcom.sun.management.snmp.acl=true \
        -Dcom.sun.management.snmp.acl.file=/usr/java/jre/lib/management/snmp.acl …

The $JAVA_HOME/jre/lib/management/ directory contains a sample SNMP ACL file (aptly named snmp.acl.template) that can be tailored to fit your site’s security policy (if you use the default locations, you do not need to specify the last two parameters). After the SNMP agent is enabled, you can run jvmstats.pl from cron to collect statistics at specific intervals. Each time Jvmstats.pl runs, it will write the current values of the Java agent’s performance counters (e.g., number of active threads, time spent performing garbage collection, etc.) to the file pased to the “-f” option, and will add a header to the file if it doesn’t exist.

The example below show how jvmstats.pl can be used to connect to server foo.prefetch.net on UDP port 8161, collect the current set of JVM statistics, and write these statistics to a file (the file name should match up with the regex used in the Orca find_files configuration stanza):

$ jvmstats.pl -s foo.prefetch.net -p 8161 -c public
        -f /home/matty/java/jvmstats/server1_myappjvm/javallator-2007-11-05

$ cat /home/matty/java/jvmstats/server1_myappjvm/javallator-2007-11-05

TIMESTAMP JVMUPTIME JITTIME THREADSTOTAL THREADSACTIVE CLASSLOADS CLASSUNLOADS \
NGGCEVENTS OLDGCEVENTS NGGCTIME OLDGCTIME OBJECTFINALIZATION HEAPCOMMITTED \
HEAPUSED NONHEAPCOMMITTED NONHEAPUSED
1194308951 218176652 735 308464 10 1348 0 380934 0 5309065 0 0 \
266797056 165481912 19136512 7156344

After a few samples are taken, the orca utility can be run with the javallator.cfg configuration file to graph the data collected above (if you haven’t used Orca before, you can peruse my monitoring LDAP article to see how to configure it). This will result in a set of graphs that look similar to the samples posted to my website. I really dig Java, and having historical performance graphs handy sure makes performance analysis a whole lot easier!

Printing status displays in Perl scripts

While developing a Perl script this weekend to summarize Solaris zone usage, I wanted to display some type of status while my script did it’s thing. My friend Clay came up with a cool way to do this, and I thought I would share it here in case others needed to do something similar. Here is the code:

#!/usr/bin/perl -w

# Initialize the index.
my $index=0;

# The characters to use as spinners.
my @spinners = ("-", "\\", "|", "/");
my $totalspinners =  $#spinners + 1;

# Turn on autoflush.
$| = 1;

print "Factoring primes:  ";

while (1) {
    print "\\b$spinners[$index++ % $totalspinners]";
    sleep 1;
}

# If you were to exit the loop, turn off autoflush.
$| = 0;

There may be a better way to do this (adding an additional module isn’t an option), but I have yet to find it. Niiiiiice!