Using systemd to restart processes that crash

The gmetad process on my Ganglia server has been a bit finicky lately. Periodically it segfaults which prevents new metrics from making their way into the RRD databases it manages:

[14745149.528104] gmetad[24286]: segfault at 0 ip 00007fb498c413c1 sp 00007fb48db40358 error 4 in[7fb498ade000+1b7000]

Luckily The gmetad service runs under systemd which provides a Restart directive to revive failed processes. You can take advantage of this nifty feature by adding “Restart=always” to your unit files:

$ cat /usr/lib/systemd/system/gmetad.service
Description=Ganglia Meta Daemon

ExecStart=/usr/sbin/gmetad -d 1


Now each time gmetad pukes systemd will automatically restart it. Hopefully I will get some time in the next few weeks to go through the core file to see why it keeps puking. Until then, this band aid should work rather nicely.

Riemann and the case of the unhappy system call

This past weekend I spent a good deal of time playing with riemann. Riemann is a powerful stream processor and I’m hoping to use it to correlated and analyze metrics from disparate data sources. After downloading and installing it I received the following error:

$ ./riemann
ERROR [2016-10-10 12:21:36,614] main – riemann.bin – Couldn’t start
java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: /tmp/ /tmp/ failed to map segment from shared object

Not exactly what I was exepecting on our first date but I guess riemann plays hard to get. :) To make a little more sense out of this error I fired up strace to retrieve the ERRNO value and to see which system call was failing:

$ strace -f ./riemann
[pid 9826] open(“/tmp/”, O_RDONLY|O_CLOEXEC) = 45
[pid 9826] read(45, “\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240>\0\0\0\0\0\0″…, 832) = 832
[pid 9826] fstat(45, {st_mode=S_IFREG|0644, st_size=63749, …}) = 0
[pid 9826] mmap(NULL, 2146168, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 45, 0) = -1 EPERM (Operation not permitted)

Now that hits the spot! mmap() is getting an EPERM which the mmap(2) manual page describes as:

“EPERM The prot argument asks for PROT_EXEC but the mapped area belongs to a file on a filesystem that was mounted no-exec.”

One problem solved! The tmp file system is mounted with the noexec flag which prohibits the mmap() from running with PROT_EXEC set. Here’s a picture of tmp:

$ mount | grep “/tmp”
tmpfs on /tmp type tmpfs (rw,noexec)

I’m not one to disable security measures so I went looking for a workaround. After reading through the netty source code for 20 minutes I came across this nugget:
            f = toDirectory(SystemPropertyUtil.get(""));
            if (f != null) {
                logger.debug("-Dio.netty.tmpdir: " + f + " (");
                return f;

Netty uses the property to craft the temporary file location. Further digging showed that riemann passes properties to the Java runtime through the EXTRA_JAVA_OPTS variable:

exec java $EXTRA_JAVA_OPTS $OPTS -cp "$JAR" riemann.bin "$COMMAND" "$CONFIG"

Furthermore, you can pass properties to the Java runtime via the “-D” option. So to make riemann happy I set the EXTRA_JAVA_OPTS environment variable to “” and fired off riemann:

$ cd $RIEMANN_HOME && ./riemann

Bingo! Riemann started and I was able to start working with it. Clojure takes me back to my Lisp programming days in college. Glad vim has () matching built in! :)

Python generators you had me at first yield

This post is added as a reference for myself.

I’ve been reading a lot about writing clean, readable and performant Python code. One feature that I now adore are generators. In the words of Python guru David Beazley a generator is:

“a function that produces a sequence of results instead of a single value”

The presentation above is an incredible overview of this amazing feature and I’ve been taking advantage of it whenever I can. Here is one case where I was able to use this today:

#!/usr/bin/env python

import os
import re

SYSFS_TAPE_PATH = "/sys/class/scsi_tape/"

def find_drives():
       Return a list of tape drives
    for drive in os.listdir(SYSFS_TAPE_PATH):
        if re.match(r'^nst[0-9]+$', drive):
            yield drive

for drive in find_drives():
    print drive

Running the script produces all of the drives on the system:

$ ./generator 

In the past I would have created a list and returned it after processing all of the drives in the scsi_tape directory. With generators that is no longer necessary and I find the code is much more readable. A huge thanks to David for the great presentation! Hopefully one day I will get to take one of his awesome Python programming classes.

Ganglia never met a quote it didn’t like. Wait it did …

This week I encountered a weird issue while developing a new Ganglia plug-in. After moving my ganglia processes to a docker container I noticed that the grid overview images weren’t displaying. This was a new ganglia installation so I figured I typo’ed something in the gmetad.conf configuration file. I reviewed the file in my git repo and everything looked perfectly fine. Any Apache errors? Gak!:

$ tail -6 /var/log/httpd/error_log

sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file
sh: -c: line 0: unexpected EOF while looking for matching `''
sh: -c: line 1: syntax error: unexpected end of file

Well those messages are super helpful lol. Something is being run through /bin/sh and it’s missing a matching single quote. But what command or script? Luckily I follow the world famous performance expert Brendan Gregg’s blog so I was aware of the Linux eBPF and bcc project. If you aren’t familiar with this amazing project here is a short blurb from their website:

“BCC is a toolkit for creating efficient kernel tracing and manipulation programs, and includes several useful tools and examples. It makes use of extended BPF (Berkeley Packet Filters), formally known as eBPF, a new feature that was first added to Linux 3.15. Much of what BCC uses requires Linux 4.1 and above.”

I knew Brendan ported a large number of the DTraceToolkit scripts to bcc so I went searching for execsnoop to see what is being executed:

$ ./execsnoop

PCOMM            PID    PPID   RET ARGS
sh               18748  18350    0   /usr/bin/rrdtool
rrdtool          18748  18350    0 /usr/bin/rrdtool
sh               18749  18350    0   /usr/bin/rrdtool graph /dev/null  --start -3600s --end now DEF:avg='/ganglia/rrds// __SummaryInfo__/cpu_num.rrd':'sum':AVERAGE PR

Bingo! Now I have a lead, but it appears execsnoop limit’s the arguments displayed (NTB: need to figure out why). So lets go to the Ganglia source code to see whats up:

$ cd /usr/share/ganglia

$ grep rrdtool *.php
*** lots of output ***

In the output above I noticed several exec() statements in graph.php so I decided to begin there. Starting from the main code block and reading down I came across the following:

if ($debug) {
  error_log("Final rrdtool command:  $command");

Digging further I noticed that debug output (including the full string that is being passed to the shell) could be set through a query string variable:

$debug = isset($_GET['debug']) ? clean_number(sanitize($_GET["debug"])) : 0;

Sweet! Adding “&debug=1” to the end of the ganglia URL got me a useful log entry in my error_log:

[Fri Oct 07 13:18:01.429310 2016] [:error] [pid 19884] [client] Final rrdtool command:  /usr/bin/rrdtool graph -   --start '-3600s' --end now --width 650 --height 300 --title 'Matty\\'s Bodacious Grid Load last hour' --vertical-label 'Loads/Procs' --lower-limit 0 --slope-mode   DEF:'a0'='/ganglia/rrds//__SummaryInfo__/load_one.rrd':'sum':AVERAGE  DEF:'a1'='/ganglia/rrds//__SummaryInfo__/cpu_num.rrd':'num':AVERAGE  DEF:'a2'='/ganglia/rrds//__SummaryInfo__/cpu_num.rrd':'sum':AVERAGE  DEF:'a3'='/ganglia/rrds//__SummaryInfo__/proc_run.rrd':'sum':AVERAGE   AREA:'a0'#BBBBBB:'1-min' VDEF:a0_last=a0,LAST VDEF:a0_min=a0,MINIMUM VDEF:a0_avg=a0,AVERAGE VDEF:a0_max=a0,MAXIMUM GPRINT:'a0_last':'Now\\:%5.1lf%s' GPRINT:'a0_min':'Min\\:%5.1lf%s' GPRINT:'a0_avg':'Avg\\:%5.1lf%s' GPRINT:'a0_max':'Max\\:%5.1lf%s\\l' LINE2:'a1'#00FF00:'Nodes' VDEF:a1_last=a1,LAST VDEF:a1_min=a1,MINIMUM VDEF:a1_avg=a1,AVERAGE VDEF:a1_max=a1,MAXIMUM GPRINT:'a1_last':'Now\\:%5.1lf%s' GPRINT:'a1_min':'Min\\:%5.1lf%s' GPRINT:'a1_avg':'Avg\\:%5.1lf%s' GPRINT:'a1_max':'Max\\:%5.1lf%s\\l' LINE2:'a2'#FF0000:'CPUs ' VDEF:a2_last=a2,LAST VDEF:a2_min=a2,MINIMUM VDEF:a2_avg=a2,AVERAGE VDEF:a2_max=a2,MAXIMUM GPRINT:'a2_last':'Now\\:%5.1lf%s' GPRINT:'a2_min':'Min\\:%5.1lf%s' GPRINT:'a2_avg':'Avg\\:%5.1lf%s' GPRINT:'a2_max':'Max\\:%5.1lf%s\\l' LINE2:'a3'#2030F4:'Procs' VDEF:a3_last=a3,LAST VDEF:a3_min=a3,MINIMUM VDEF:a3_avg=a3,AVERAGE VDEF:a3_max=a3,MAXIMUM GPRINT:'a3_last':'Now\\:%5.1lf%s' GPRINT:'a3_min':'Min\\:%5.1lf%s' GPRINT:'a3_avg':'Avg\\:%5.1lf%s' GPRINT:'a3_max':'Max\\:%5.1lf%s\\l' 

Running that manually from a shell prompt generated the same error as the one I originally found! Excellent, now I know exactly what is causing the error. I opened the output above in vim and searched for a single quote. That highlighted all single quotes and my eyes immediately picked up the following:

–title ‘Matty\’s Bodacious Grid Load last hour’

Oh snap, I found a gem here. A LOOOONG time ago in a shell scripting book (possibly in UNIX shell programming?) far, far away I remembered reading that you can never have a single quote inside a pair of single quotes. Googling this turned up the following POSIX standard which states:

“A single-quote cannot occur within single-quotes.”

When I removed the single quote after Matty and re-ran rrdtool it ran w/o error and spit out a nice png. So could that be the issue? I removed the single quote from the gridname directive in gmetad.conf, bounced gmetad and low and behold all of my graphs started showing up. Now to finish up my plug-in.

Python console editing with VIM

I’m a long time console guy and haven’t found a graphical Python development tool that suits my needs as well as vim. It takes a couple of amazing vim plug-ins but the experience is great IMHO. I’m currently using the vim-jedi, pylint and vim-syntastic plug-ins which can be installed with yum on CentOS/Fedora machines:

$ sudo yum install pylint

$ sudo yum install vim-jedi

$ sudo yum install vim-syntastic-python

To enable syntax highlighting and auto indentation you can add the following to your $HOME/.vimrc:

syntax on
filetype indent plugin on

Once these are in place your code will be highlighted with a nice set of colors, hitting return will generate the correct spacing and pylint will be run automatically to note problems with your code. If you are using other plug-ins let me know via a comment.

Printing tape drive serial numbers

While automating a process this week I needed a way to get the serial number off a batch of tape drives. At first I thought I could retrieve this information through /sys. But after a bunch of poking around with cat, systool and udevadm I realized I couldn’t get what I want through /sys. One such failure:

$ udevadm info -a -n /dev/nst0 | grep -i serial

If I can’t get the serial # through /sys I can always poke the drive directly, right? Definitely! SCSI vital product data (VPD) is stored on the drive in a series of SCSI mode pages. These pages can be viewed with the sg_vpd utility:

$ sg_vpd –page=0x80 /dev/nst0

VPD INQUIRY: Unit serial number page
  Unit serial number: 123456789012

The command above retrieves SCSI mode page 0x80 (Unit Serial Number) and displays it in an easily parsed format. Seagates SCSI commands reference is an excellent reference for understanding the SCSI protocol.