Getting alerts when Java processes crash

When bugs occur in the Java runtime environment, most administrators want to get notified so they can take corrective action. These actions can range from restarting a Java process, collecting postmortem data or calling in application support personnel to debug the situation further. The Java runtime has a number of useful options that can be used for this purpose. The first option is “-XX:OnOutOfMemoryError”, which allows a command to be run when the runtime environment incurs an out of memory condition. When this option is combined with the logger command line utility:

$ java -XX:OnOutOfMemoryError=”logger Java process %p encountered an OOM condition” …

Syslog entries similar to the following will be generated each time an OOM event occurs:

Jan 21 19:59:17 nevadadev root: [ID 702911 daemon.notice] Java process 19001 encountered an OOM condition

Another super useful option is “-XX:OnError”, which allows a command to be run when the runtime environment incurs a fatal error (i.e., a hard crash). When this option is combined with the logger utility:

$ java -XX:OnError=”logger -p Java process %p encountered a fatal condition” …

Syslog entries similar to the following will be generated when a fatal event occurs:

Jan 21 19:52:17 nevadadev root: [ID 702911 daemon.notice] Java process 19004 encountered a fatal condition

The options above allow you to run one or more commands when these errors are encountered, so you could chain together a postmortem debugging tool, a utility (logger or mail) to generate alerts, and a restarter script to start a new Java process (this assumes you aren’t using SMF). Nice!

matty on January 29, 2008 | Filed Under Java

2 Comments

Jani Mikkonen  on February 6th, 2008

Stumbled here from rootprompt – and nice to get some tidbits that i didnt know before and i’ll leave my own here for others.

There’s a tool called “Monit” which was first written just to monitor rogue java application and crashes. It has grown since to be a quite a nice general monitoring solution but still does a good job to keep java servers in shape.

http://www.tildeslash.com/monit/

Andrew Scherpbier  on February 24th, 2008

I have found that the Java Service Wrapper is a much better solution to these issues. With the wrapper, you can setup pattern filters on the stdout and stderr of the java process. One of the things I do is setup a pattern for OutOfMemoryError and make the action restart the java process up to 5 times. After that it will halt the process. If you setup the logger to use syslog, then all of this activity can easily be caught using a log monitoring tool like logcheck.
Another nice thing about the Java Service Wrapper is that it can product a full thread dump of the running process at any time. Very handy and much easier than using jconsole.
Java Service Wrapper is cross platform, although the thread dump only works on Solaris and Linux AFAIK.

Anyway, check out the Java Service Wrapper:

http://wrapper.tanukisoftware.org/doc/english/introduction.html

Leave a Comment