Using Linux auditing to observe program executions and file accesses

Linux systems have a rather nice auditing framework built into the kernel. This framework consists of a kernel component that inspects and filters events and a userland daemon (auditd) that takes the events and archives them to persistent storage. These audit log events can then be searched and summarized with the ausearch and aureport utilities. The following ausearch command will dump all audit records for UID 7005:

$ ausearch --uid-all 7005 | head -4

time->Thu Oct 20 15:01:49 2016
type=PROCTITLE msg=audit(1476990109.442:153031): proctitle="-bash"
type=SYSCALL msg=audit(1476990109.442:153031): arch=c000003e syscall=1 success=yes exit=1024 a0=1 a1=55b32ff2fe40 a2=400 a3=6553726567616e61 items=0 ppid=18901 pid=19071 auid=7005 uid=7005 gid=7005 euid=7005 suid=7005 fsuid=7005 egid=7005 sgid=7005 fsgid=7005 tty=pts9 ses=6031 comm="ps" exe="/usr/bin/ps" key=(null)

Each audit event contains the type of event (system call executed, file read, etc.) as well as a bunch of user and system context. To control the types of events that get audited you can create an audit.rules file in /etc/audit/rules.d. This file can contain control rules, file system rules and system call rules. Control rules are used to control the behavior of the audit subsystem (e.g., what to do when the disk fills up, how many audit buffers to allocate, etc.). File system rules allow you to audit when files or directories are accessed and system call rules allow you to audit system calls (full list in /usr/include/asm/unistd_64.h).

To illustrate how useful auditing is lets say you are tasked with figuring out what a black box application does. Furthermore, lets assume this application runs as the user fmep with the UID 7005. To log EVERY system call generated by processes with UID 7005 we can add the following audit rule (WARNING: DO NOT ENABLE THIS ON PRODUCTION SYSTEMS WITHOUT PROPER TESTING :END WARNING):

$ auditctl -a always,exit -S all -F auid=7005

The “-a” option tells auditd to append the rule to the chain, “always,exit” audits system call events at exit, “-S all” audits all system calls and the “-F auid=7005” filters events where the loginuid is set to 7005 (more info on AUID is available here). To verify the rule was added we can run auditctl with the “-l” option:

$ auditctl -l

-a always,exit -S all -F auid=7005

Once the rules are activated you will start seeing events similar to the following in /var/log/audit/audit.log:

*type=SYSCALL msg=audit(1476990167.749:160759): arch=c000003e syscall=14
success=yes exit=0 a0=2 a1=7ffe53ea8fc0 a2=0 a3=8 items=0 ppid=18884
pid=18900 auid=7005 uid=7005 gid=7005 euid=7005 suid=7005 fsuid=7005
egid=7005 sgid=7005 fsgid=7005 tty=(none) ses=6031 comm="sshd"
exe="/usr/sbin/sshd" key=(null)*

Each system call audit event contains the system call number as well as a good deal of context (pid, uid, ppid, etc.) to describe the event. To better understand the list of commands and files being accessed by a given UID I wrote auditloginfo. This script parses the audit log and produces a report listing the commands executed, files accessed and a summary of the system calls made:

$ sudo --uid 7005

Audit report for AUID 7005

System calls made:
read 207
stat 179
close 147
open 142
rt_sigprocmask 141
clock_gettime 120
write 88

Commands executed:
clear 2
/bin/basename bash 2
uname -m 1
ps -ef 1
ls --color=auto -la 1

Files opened:
/etc/ 17
/lib64/ 15
/etc/passwd 14
/usr/lib/locale/locale-archive 12
/lib64/ 8

I’ve found auditing rather useful and it’s amazing what you can discover when you enable it.

This article was posted by Matty on 2016-10-26 15:58:00 -0400 EDT