Riemann and the case of the unhappy system call

This past weekend I spent a good deal of time playing with riemann. Riemann is a powerful stream processor and I’m hoping to use it to correlated and analyze metrics from disparate data sources. After downloading and installing it I received the following error:

$ ./riemann
ERROR [2016-10-10 12:21:36,614] main – riemann.bin – Couldn’t start
java.util.concurrent.ExecutionException: java.lang.UnsatisfiedLinkError: /tmp/libnetty-transport-native-epoll3719688563306389605.so: /tmp/libnetty-transport-native-epoll3719688563306389605.so: failed to map segment from shared object

Not exactly what I was exepecting on our first date but I guess riemann plays hard to get. :) To make a little more sense out of this error I fired up strace to retrieve the ERRNO value and to see which system call was failing:

$ strace -f ./riemann
…..
[pid 9826] open(“/tmp/libnetty-transport-native-epoll27354233456383270.so”, O_RDONLY|O_CLOEXEC) = 45
[pid 9826] read(45, “\177ELF\2\1\1\0\0\0\0\0\0\0\0\0\3\0>\0\1\0\0\0\240>\0\0\0\0\0\0″…, 832) = 832
[pid 9826] fstat(45, {st_mode=S_IFREG|0644, st_size=63749, …}) = 0
[pid 9826] mmap(NULL, 2146168, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_DENYWRITE, 45, 0) = -1 EPERM (Operation not permitted)
…..

Now that hits the spot! mmap() is getting an EPERM which the mmap(2) manual page describes as:

“EPERM The prot argument asks for PROT_EXEC but the mapped area belongs to a file on a filesystem that was mounted no-exec.”

One problem solved! The tmp file system is mounted with the noexec flag which prohibits the mmap() from running with PROT_EXEC set. Here’s a picture of tmp:

$ mount | grep “/tmp”
tmpfs on /tmp type tmpfs (rw,noexec)

I’m not one to disable security measures so I went looking for a workaround. After reading through the netty source code for 20 minutes I came across this nugget:

NativeLibraryLoader.java
            f = toDirectory(SystemPropertyUtil.get("java.io.tmpdir"));
            if (f != null) {
                logger.debug("-Dio.netty.tmpdir: " + f + " (java.io.tmpdir)");
                return f;
            }

Netty uses the java.io.tmpdir property to craft the temporary file location. Further digging showed that riemann passes properties to the Java runtime through the EXTRA_JAVA_OPTS variable:

exec java $EXTRA_JAVA_OPTS $OPTS -cp "$JAR" riemann.bin "$COMMAND" "$CONFIG"

Furthermore, you can pass properties to the Java runtime via the “-D” option. So to make riemann happy I set the EXTRA_JAVA_OPTS environment variable to “-Djava.io.tmpdir=/path/to/new/tmpdir” and fired off riemann:

$ EXTRA_JAVA_OPTS=-Djava.io.tmpdir=/sfw/riemann/latest/tmp cd $RIEMANN_HOME && ./riemann

Bingo! Riemann started and I was able to start working with it. Clojure takes me back to my Lisp programming days in college. Glad vim has () matching built in! :)