Using Linux auditing to observe program executions and file accesses

Linux systems have a rather nice auditing framework built into the kernel. This framework consists of a kernel component that inspects and filters events and a userland daemon (auditd) that takes the events and archives them to persistent storage. These audit log events can then be searched and summarized with the ausearch and aureport utilities. The following ausearch command will dump all audit records for UID 7005:

$ ausearch --uid-all 7005 | head -4
time->Thu Oct 20 15:01:49 2016
type=PROCTITLE msg=audit(1476990109.442:153031): proctitle="-bash"
type=SYSCALL msg=audit(1476990109.442:153031): arch=c000003e syscall=1 success=yes exit=1024 a0=1 a1=55b32ff2fe40 a2=400 a3=6553726567616e61 items=0 ppid=18901 pid=19071 auid=7005 uid=7005 gid=7005 euid=7005 suid=7005 fsuid=7005 egid=7005 sgid=7005 fsgid=7005 tty=pts9 ses=6031 comm="ps" exe="/usr/bin/ps" key=(null)

Each audit event contains the type of event (system call executed, file read, etc.) as well as a bunch of user and system context. To control the types of events that get audited you can create an audit.rules file in /etc/audit/rules.d. This file can contain control rules, file system rules and system call rules. Control rules are used to control the behavior of the audit subsystem (e.g., what to do when the disk fills up, how many audit buffers to allocate, etc.). File system rules allow you to audit when files or directories are accessed and system call rules allow you to audit system calls (full list in /usr/include/asm/unistd_64.h).

To illustrate how useful auditing is lets say you are tasked with figuring out what a black box application does. Furthermore, lets assume this application runs as the user fmep with the UID 7005. To log EVERY system call generated by processes with UID 7005 we can add the following audit rule (**WARNING** DO NOT ENABLE THIS ON PRODUCTION SYSTEMS WITHOUT PROPER TESTING ***END WARNING***):

$ auditctl -a always,exit -S all -F auid=7005

The “-a” option tells auditd to append the rule to the chain, “always,exit” audits system call events at exit, “-S all” audits all system calls and the “-F auid=7005” filters events where the loginuid is set to 7005 (more info on AUID is available here). To verify the rule was added we can run auditctl with the “-l” option:

$ auditctl -l
-a always,exit -S all -F auid=7005

Once the rules are activated you will start seeing events similar to the following in /var/log/audit/audit.log:

type=SYSCALL msg=audit(1476990167.749:160759): arch=c000003e syscall=14 success=yes exit=0 a0=2 a1=7ffe53ea8fc0 a2=0 a3=8 items=0 ppid=18884 pid=18900 auid=7005 uid=7005 gid=7005 euid=7005 suid=7005 fsuid=7005 egid=7005 sgid=7005 fsgid=7005 tty=(none) ses=6031 comm=”sshd” exe=”/usr/sbin/sshd” key=(null)

Each system call audit event contains the system call number as well as a good deal of context (pid, uid, ppid, etc.) to describe the event. To better understand the list of commands and files being accessed by a given UID I wrote auditloginfo. This script parses the audit log and produces a report listing the commands executed, files accessed and a summary of the system calls made:

$  sudo auditloginfo.py --uid 7005
Audit report for AUID 7005

System calls made:
   read 207
   stat 179
   close 147
   open 142
   rt_sigprocmask 141
   clock_gettime 120
   write 88
   
Commands executed:
   clear  2
   /bin/basename bash  2
   uname -m  1
   ps -ef  1
   ls --color=auto -la  1

Files opened:
   /etc/ld.so.cache 17
   /lib64/libc.so.6 15
   /etc/passwd 14
   /usr/lib/locale/locale-archive 12
   /lib64/libpthread.so.0 8

I’ve found auditing rather useful and it’s amazing what you can discover when you enable it.

Configuring a Linux NFS server in a SELinux-managed environment

The Linux kernel has supported NFS for as long as I can remember. All of the major distributions (Redhat, CentOS, Fedora, Suse, Ubunut) ship with NFS client and server support and have all of the user land daemons and tools needed to configure and debug NFS. I spent some time this past weekend bringing up a new NFS server in a SELinux-managed environment, and thought I would share my experience with my readers.

Setting up a Linux NFS server with SELinux can be done in just a few simple steps:

1. Configure SELinux to allow remote hosts to access content that is exported through NFS.

2. Add file systems that you want to export to the /etc/exports file.

3. Enable the NFS services.

SELinux does not allow remote content to be accessed by default. This can easily be fixed by enabling one of the three SELinux booleans listed below:

nfs_export_all_ro — allows file systems to be exported read-only
nfs_export_all_rw — allows file systems to be exported read-write
use_nfs_home_dirs — allows home directories to be exported over NFS

To set a boolean you can use the setsebool utility:

$ setsebool -P nfs_export_all_rw 1

Once SELinux has been instructed to allow NFS exports, you can add the file systems you want to export to /etc/exports. This file has the following format:

FILE_SYSTEM_TO_EXPORT CLIENT_LIST(EXPORT_OPTIONS)

To export the file system/exports/nfs to clients on the 192.168.1.0/24 network or in the prefetch.net domain, we can add an entry similar to the following to /etc/exports

/export/nfs 192.168.1.0/255.255.255.0(rw,sync) *.prefetch.net(rw,sync)

To start the NFS services on your server, you will need to enable the portmap and nfs services. This can be accomplished with chkconfig and service:

$ chkconfig portmap on

$ chkconfig nfs on

$ chkconfig nfslock on

$ service portmap start

Starting portmap:                                          [  OK  ]

$ service nfs start

Starting NFS services:                                     [  OK  ]
Starting NFS quotas:                                       [  OK  ]
Starting NFS daemon:                                       [  OK  ]
Starting NFS mountd:                                       [  OK  ]

$ service nfslock start

Starting NFS statd:                                        [  OK  ]

The file systems listed in /etc/exports should now be exported, and you can verify this with the exportfs utility:

$ exportfs

/export/nfs   	192.168.1.0/255.255.255.0
/export/nfs   	*.prefetch.net

To verify a mount is functioning, you can try mounting it from a client that falls inside the ACL:

$ mount server:/export/nfs /export/nfs

If this fails or you receive a permission denied error you can check the following things:

1. Check setroubleshootd to see if SELinux is blocking access.

2. Verify that the portmap and nfs services are running.

3. Check nfsstat and exportfs to view the NFS statistics and exported file systems.

4. If a firewall is configured, verify that iptables is allowing access.

If everything is working as expected, you should pat yourself on the back for a job well done!

How to debug SELinux service issues

I have been spending a good bit of time playing with SELinux, which has required me to learn how to debug issues that arise due to its use. The most common issues that arise are usually related to security context mismatches. This can occur because somebody modified a file and didn’t bother to chcon a file, or because somebody changed the contexts that were once present on a file. To see the SELinux context assigned to a file or directory, you can use the ls “-Z” option:

$ ls -lZ /var/www

-rw-r--r--. root root system_u:object_r:httpd_sys_content_t:s0 index.html

Now say a user approached you because they were getting 403 forbidden messages when they tried to retrieve a file on your Apache web server:

$ wget localhost/index.txt

--2010-10-29 14:44:22--  http://localhost/index.txt
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2010-10-29 14:44:22 ERROR 403: Forbidden.

The apache error log indicated that there was indeed a permission issue:

[Fri Oct 29 14:25:53 2010] [error] [client 127.0.0.1] (13)Permission denied: access to /index.txt denied
[Fri Oct 29 14:44:22 2010] [error] [client 127.0.0.1] (13)Permission denied: access to /index.txt denied

Though the UNIX permissions on the file were correct. To see if the permission issue is caused by SELinux, you can fire up the setroubleshootd daemon and then tail /var/log/messages:

$ /usr/sbin/setroubleshootd

$ grep httpd /var/log/messages

Oct 29 14:44:26 theshack setroubleshoot: SELinux is preventing /usr/sbin/httpd "getattr" access to /var/www/html/index.txt. For complete SELinux messages. run sealert -l f02ee96b-75fb-40d6-a5e4-dd2bc91f52e2
Oct 29 14:44:26 theshack setroubleshoot: SELinux is preventing /usr/sbin/httpd "getattr" access to /var/www/html/index.txt. For complete SELinux messages. run sealert -l f02ee96b-75fb-40d6-a5e4-dd2bc91f52e2

We can see that SELinux was the reason the request failed, and we can get some additional information by running sealert with the “-l” option and the alert number:

$ sealert -l f02ee96b-75fb-40d6-a5e4-dd2bc91f52e2

Summary:

SELinux is preventing /usr/sbin/httpd "getattr" access to
/var/www/html/index.txt.

Detailed Description:

SELinux denied access requested by httpd. /var/www/html/index.txt may be a
mislabeled. /var/www/html/index.txt default SELinux type is httpd_sys_content_t,
but its current type is var_t. Changing this file back to the default type, may
fix your problem.

File contexts can be assigned to a file in the following ways.

  * Files created in a directory receive the file context of the parent
    directory by default.
  * The SELinux policy might override the default label inherited from the
    parent directory by specifying a process running in context A which creates
    a file in a directory labeled B will instead create the file with label C.
    An example of this would be the dhcp client running with the dhclient_t type
    and creating a file in the directory /etc. This file would normally receive
    the etc_t type due to parental inheritance but instead the file is labeled
    with the net_conf_t type because the SELinux policy specifies this.
  * Users can change the file context on a file using tools such as chcon, or
    restorecon.

This file could have been mislabeled either by user error, or if an normally
confined application was run under the wrong domain.

However, this might also indicate a bug in SELinux because the file should not
have been labeled with this type.

If you believe this is a bug, please file a bug report against this package.

Allowing Access:

You can restore the default system context to this file by executing the
restorecon command. restorecon '/var/www/html/index.txt', if this file is a
directory, you can recursively restore using restorecon -R
'/var/www/html/index.txt'.

Fix Command:

/sbin/restorecon '/var/www/html/index.txt'

Additional Information:

Source Context                unconfined_u:system_r:httpd_t:s0
Target Context                unconfined_u:object_r:var_t:s0
Target Objects                /var/www/html/index.txt [ file ]
Source                        httpd
Source Path                   /usr/sbin/httpd
Port                          
Host                          theshack
Source RPM Packages           httpd-2.2.16-1.fc13
Target RPM Packages           
Policy RPM                    selinux-policy-3.7.19-65.fc13
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Plugin Name                   restorecon
Host Name                     theshack
Platform                      Linux theshack 2.6.34.7-61.fc13.x86_64 #1 SMP Tue
                              Oct 19 04:06:30 UTC 2010 x86_64 x86_64
Alert Count                   4
First Seen                    Fri Oct 29 14:25:53 2010
Last Seen                     Fri Oct 29 14:44:22 2010
Local ID                      f02ee96b-75fb-40d6-a5e4-dd2bc91f52e2
Line Numbers                  

Raw Audit Messages            

node=theshack type=AVC msg=audit(1288377862.457:22628): avc:  denied  { getattr } for  pid=2798 comm="httpd" path="/var/www/html/index.txt" dev=dm-0 ino=525951 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file

node=theshack type=SYSCALL msg=audit(1288377862.457:22628): arch=c000003e syscall=6 success=no exit=-13 a0=7f609700e588 a1=7fffabd48f80 a2=7fffabd48f80 a3=1 items=0 ppid=2794 pid=2798 auid=500 uid=48 gid=487 euid=48 suid=48 fsuid=48 egid=487 sgid=487 fsgid=487 tty=(none) ses=1 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)

The error message is pretty self explanatory, and you can use restorecon to fix the security context of the file:

$ /sbin/restorecon ‘/var/www/html/index.txt’

Once the correct security context has been assigned to the file, we can verify that all is well:

$ ls -lZ /var/www/html/index.txt
-rw-r–r–. root root system_u:object_r:httpd_sys_content_t:s0 /var/www/html/index.txt

$ wget -q -O – localhost/index.txt
hi

If you aren’t able to fire up setroubleshootd, you can still debug this issue. The first thing to look at is the audit log (audit.log) and messages file, which should both be located under /var/log. Let’s see if the httpd process caused any alerts to be generated:

$ grep httpd /var/log/audit/audit.log

type=AVC msg=audit(1288376753.574:22621): avc:  denied  { getattr } for  pid=2803 comm="httpd" path="/var/www/html/index.txt" dev=dm-0 ino=525951 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file
type=SYSCALL msg=audit(1288376753.574:22621): arch=c000003e syscall=4 success=no exit=-13 a0=7f609700e4b8 a1=7fffabd48f80 a2=7fffabd48f80 a3=0 items=0 ppid=2794 pid=2803 auid=500 uid=48 gid=487 euid=48 suid=48 fsuid=48 egid=487 sgid=487 fsgid=487 tty=(none) ses=1 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1288376753.575:22622): avc:  denied  { getattr } for  pid=2803 comm="httpd" path="/var/www/html/index.txt" dev=dm-0 ino=525951 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file
type=SYSCALL msg=audit(1288376753.575:22622): arch=c000003e syscall=6 success=no exit=-13 a0=7f609700e588 a1=7fffabd48f80 a2=7fffabd48f80 a3=1 items=0 ppid=2794 pid=2803 auid=500 uid=48 gid=487 euid=48 suid=48 fsuid=48 egid=487 sgid=487 fsgid=487 tty=(none) ses=1 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)

In the messages above, we can see that SELinux denied access to the index.txt file due to an incorrect security context (I’ll do a follow up post on security contexts and how to tell which ones should be used for a given file). To fix this, we can once again run the restorecon command to recursively restore the context of all of the files in /var/www:

$ restorecon -R /var/www

$ ls -lZ

-rw-r--r--. root root system_u:object_r:httpd_sys_content_t:s0 index.html

SELinux is a pretty cool technology, and I think it’s gotten a bad wrap due to its complexity. I’m going to write up a number of additional SELinux posts in the coming weeks, which will hopefully demystify some things and make debugging SELinux issues a piece of cake!