Using TCP Wrappers to protect Linux and Solaris services

I have been using tcp wrappers for years, and it’s a very simple way to allow and deny network access to applications. TCP wrapper functionality is built into the system libwrap.so module, which various applications are linked against. To see if a given application supports tcp wrappers, you can use the ldd utility:

$ ldd `which sshd` | grep wrap
libwrap.so.0 => /lib64/libwrap.so.0 (0x00002ac16fe0f000)

TCP wrappers is configured through the /etc/hosts.allow and /etc/hosts.deny files. The hosts.allow file allows you to control which services will be accepted, and the hosts.deny file allows you to control which services will be denied. Both files use a format similar to the following:

DAEMON_LIST : CLIENT_LIST [ : SHELL_COMMAND ]

The DAEMON_LIST contains the name of the executable you are protecting, which could be sshd, sendmail or any other daemon that you are trying to protect. The CLIENT_LIST contains the hosts or domain names you wish to allow or deny access to, and they can take various forms:

ALL — matches everything
.prefetch.net — matches everything in the prefetch.net domain
192.168.0.0/255.255.0.0 — matches everything in the 192.168 /16 IP address space
192.168.1.1 — matches a single IP address

SHELL_COMMAND allows you to run a command when the rule matches. This could be used to run a notification script, block an IP with iptables or to provide some more extensive logging. To put this into action, we can set up our hosts.allow and hosts.deny files to limit access to our SSH daemon. The following hosts.allow will allow connections from the IP 192.168.1.100, and deny access from everyone else:

$ cat /etc/hosts.allow
sshd : 192.168.1.100

$ cat /etc/hosts.deny
ALL : ALL

When libwrap processes these files, it will first look for matches in /etc/hosts.allow by sequentially evaluating the rules. If a match isn’t found, it will then consult the hosts.deny file. If a connection is denied, you should see a message similar to the following in the messages file:

Apr 16 13:16:18 localhost sshd[3628]: refused connect from ::ffff:192.168.1.8 (::ffff:192.168.1.8)

TCP wrappers is an invaluable tool, and provides a simple and intuitive way to secure your services. It’s no substitute for a properly functioning host firewall, but an additional tool that can be used to protect your critical services.

How to debug SELinux service issues

I have been spending a good bit of time playing with SELinux, which has required me to learn how to debug issues that arise due to its use. The most common issues that arise are usually related to security context mismatches. This can occur because somebody modified a file and didn’t bother to chcon a file, or because somebody changed the contexts that were once present on a file. To see the SELinux context assigned to a file or directory, you can use the ls “-Z” option:

$ ls -lZ /var/www

-rw-r--r--. root root system_u:object_r:httpd_sys_content_t:s0 index.html

Now say a user approached you because they were getting 403 forbidden messages when they tried to retrieve a file on your Apache web server:

$ wget localhost/index.txt

--2010-10-29 14:44:22--  http://localhost/index.txt
Resolving localhost... 127.0.0.1
Connecting to localhost|127.0.0.1|:80... connected.
HTTP request sent, awaiting response... 403 Forbidden
2010-10-29 14:44:22 ERROR 403: Forbidden.

The apache error log indicated that there was indeed a permission issue:

[Fri Oct 29 14:25:53 2010] [error] [client 127.0.0.1] (13)Permission denied: access to /index.txt denied
[Fri Oct 29 14:44:22 2010] [error] [client 127.0.0.1] (13)Permission denied: access to /index.txt denied

Though the UNIX permissions on the file were correct. To see if the permission issue is caused by SELinux, you can fire up the setroubleshootd daemon and then tail /var/log/messages:

$ /usr/sbin/setroubleshootd

$ grep httpd /var/log/messages

Oct 29 14:44:26 theshack setroubleshoot: SELinux is preventing /usr/sbin/httpd "getattr" access to /var/www/html/index.txt. For complete SELinux messages. run sealert -l f02ee96b-75fb-40d6-a5e4-dd2bc91f52e2
Oct 29 14:44:26 theshack setroubleshoot: SELinux is preventing /usr/sbin/httpd "getattr" access to /var/www/html/index.txt. For complete SELinux messages. run sealert -l f02ee96b-75fb-40d6-a5e4-dd2bc91f52e2

We can see that SELinux was the reason the request failed, and we can get some additional information by running sealert with the “-l” option and the alert number:

$ sealert -l f02ee96b-75fb-40d6-a5e4-dd2bc91f52e2

Summary:

SELinux is preventing /usr/sbin/httpd "getattr" access to
/var/www/html/index.txt.

Detailed Description:

SELinux denied access requested by httpd. /var/www/html/index.txt may be a
mislabeled. /var/www/html/index.txt default SELinux type is httpd_sys_content_t,
but its current type is var_t. Changing this file back to the default type, may
fix your problem.

File contexts can be assigned to a file in the following ways.

  * Files created in a directory receive the file context of the parent
    directory by default.
  * The SELinux policy might override the default label inherited from the
    parent directory by specifying a process running in context A which creates
    a file in a directory labeled B will instead create the file with label C.
    An example of this would be the dhcp client running with the dhclient_t type
    and creating a file in the directory /etc. This file would normally receive
    the etc_t type due to parental inheritance but instead the file is labeled
    with the net_conf_t type because the SELinux policy specifies this.
  * Users can change the file context on a file using tools such as chcon, or
    restorecon.

This file could have been mislabeled either by user error, or if an normally
confined application was run under the wrong domain.

However, this might also indicate a bug in SELinux because the file should not
have been labeled with this type.

If you believe this is a bug, please file a bug report against this package.

Allowing Access:

You can restore the default system context to this file by executing the
restorecon command. restorecon '/var/www/html/index.txt', if this file is a
directory, you can recursively restore using restorecon -R
'/var/www/html/index.txt'.

Fix Command:

/sbin/restorecon '/var/www/html/index.txt'

Additional Information:

Source Context                unconfined_u:system_r:httpd_t:s0
Target Context                unconfined_u:object_r:var_t:s0
Target Objects                /var/www/html/index.txt [ file ]
Source                        httpd
Source Path                   /usr/sbin/httpd
Port                          
Host                          theshack
Source RPM Packages           httpd-2.2.16-1.fc13
Target RPM Packages           
Policy RPM                    selinux-policy-3.7.19-65.fc13
Selinux Enabled               True
Policy Type                   targeted
Enforcing Mode                Enforcing
Plugin Name                   restorecon
Host Name                     theshack
Platform                      Linux theshack 2.6.34.7-61.fc13.x86_64 #1 SMP Tue
                              Oct 19 04:06:30 UTC 2010 x86_64 x86_64
Alert Count                   4
First Seen                    Fri Oct 29 14:25:53 2010
Last Seen                     Fri Oct 29 14:44:22 2010
Local ID                      f02ee96b-75fb-40d6-a5e4-dd2bc91f52e2
Line Numbers                  

Raw Audit Messages            

node=theshack type=AVC msg=audit(1288377862.457:22628): avc:  denied  { getattr } for  pid=2798 comm="httpd" path="/var/www/html/index.txt" dev=dm-0 ino=525951 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file

node=theshack type=SYSCALL msg=audit(1288377862.457:22628): arch=c000003e syscall=6 success=no exit=-13 a0=7f609700e588 a1=7fffabd48f80 a2=7fffabd48f80 a3=1 items=0 ppid=2794 pid=2798 auid=500 uid=48 gid=487 euid=48 suid=48 fsuid=48 egid=487 sgid=487 fsgid=487 tty=(none) ses=1 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)

The error message is pretty self explanatory, and you can use restorecon to fix the security context of the file:

$ /sbin/restorecon ‘/var/www/html/index.txt’

Once the correct security context has been assigned to the file, we can verify that all is well:

$ ls -lZ /var/www/html/index.txt
-rw-r–r–. root root system_u:object_r:httpd_sys_content_t:s0 /var/www/html/index.txt

$ wget -q -O – localhost/index.txt
hi

If you aren’t able to fire up setroubleshootd, you can still debug this issue. The first thing to look at is the audit log (audit.log) and messages file, which should both be located under /var/log. Let’s see if the httpd process caused any alerts to be generated:

$ grep httpd /var/log/audit/audit.log

type=AVC msg=audit(1288376753.574:22621): avc:  denied  { getattr } for  pid=2803 comm="httpd" path="/var/www/html/index.txt" dev=dm-0 ino=525951 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file
type=SYSCALL msg=audit(1288376753.574:22621): arch=c000003e syscall=4 success=no exit=-13 a0=7f609700e4b8 a1=7fffabd48f80 a2=7fffabd48f80 a3=0 items=0 ppid=2794 pid=2803 auid=500 uid=48 gid=487 euid=48 suid=48 fsuid=48 egid=487 sgid=487 fsgid=487 tty=(none) ses=1 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(1288376753.575:22622): avc:  denied  { getattr } for  pid=2803 comm="httpd" path="/var/www/html/index.txt" dev=dm-0 ino=525951 scontext=unconfined_u:system_r:httpd_t:s0 tcontext=unconfined_u:object_r:var_t:s0 tclass=file
type=SYSCALL msg=audit(1288376753.575:22622): arch=c000003e syscall=6 success=no exit=-13 a0=7f609700e588 a1=7fffabd48f80 a2=7fffabd48f80 a3=1 items=0 ppid=2794 pid=2803 auid=500 uid=48 gid=487 euid=48 suid=48 fsuid=48 egid=487 sgid=487 fsgid=487 tty=(none) ses=1 comm="httpd" exe="/usr/sbin/httpd" subj=unconfined_u:system_r:httpd_t:s0 key=(null)

In the messages above, we can see that SELinux denied access to the index.txt file due to an incorrect security context (I’ll do a follow up post on security contexts and how to tell which ones should be used for a given file). To fix this, we can once again run the restorecon command to recursively restore the context of all of the files in /var/www:

$ restorecon -R /var/www

$ ls -lZ

-rw-r--r--. root root system_u:object_r:httpd_sys_content_t:s0 index.html

SELinux is a pretty cool technology, and I think it’s gotten a bad wrap due to its complexity. I’m going to write up a number of additional SELinux posts in the coming weeks, which will hopefully demystify some things and make debugging SELinux issues a piece of cake!

Concert Review: KISS

When I was growing up, the people around me were constantly exposing me to various forms of Rock & Roll. It seems like just yesterday I was driving to the store with my cousin, who put a KISS cassette in this stereo and told me give it a good listen. The music resinated with me, and something about the band hit a happy musical note with me. I bought one of their albums, and played it over and over and over … Eventually I outgrew KISS, but a piece of them has always stuck with me.

Now fast forward 20-years, and we have Gene Simmon’s Family Jewels and a version of KISS without Ace Freely and Peter Criss. This brought back memories of me mozying around with my cousin jamming out to their music, and I knew I had to see these guys live. I got my chance a few weeks back, and the entire concert experience was amazing!

The show started out in crazy KISS fashion, which plenty of pyrotechnics and Gene and Paul putting on an amazing stage presence. Paul was running around the stage getting the crow into the show, Gene was struting his stuff and wagging his tongue, and the newbies in teh band looked to be having a blast as well. A ton of awesome music was performed, including hits such as “Cold Gin”, “Crazy, Crazy Nights”, “Calling Dr. Love”, “Love Gun”, “Black Diamond”, “Detroit Rock City”, “Lick It Up” and their biggest hit IMHO “Rock And Roll All NIght”. The fans at the show were insanely awesome, and a number of them painted themselves up for the event. Nice!

I can summarize this show with the following two words. KICK ASS!

Finding orphaned RPMs on Linux hosts

If you use an RPM-based Linux distribution, you may have run into one or more cases were your system contains orphaned packages. An orphaned package is a package that doesn’t have any packages that depend on it, and in a number of cases the package is no longer required for the system to function correctly. To view the list of orphaned packages on your Linux host, you can use the incredibly useful rpmorphan tool. To install it, you can fire up yum:

$ yum install rpmorphan

Once installed, you can run it to view the list of orphaned packages on your system:

$ rpmorphan
libdwarf
libertas-usb8388-firmware
libsane-hpaio

To get additional information on what rpmorphan is doing, you can run it with the “-verbose” option. This is a super useful utility, and one that will make its way into my SA utility belt!

A couple useful tidbits about the Linux /dev/random and /dev/urandom devices

Linux contains two devices that provide a source of entropy for the system. The first device is /dev/random, and the second is /dev/urandom. /dev/random is a character special device that provides a source of entropy until the system-wide entropy pool is exhausted, at which time it will block until additional entropy is available. /dev/urandom is a character device that uses the system entropy pool until it is depleted, then falls back to a pseudo-random number generator.

To gain access to the system wide entropy pool, you can use the openssl utilities “rand” option:

$ openssl rand -base64 16
4T+aLG9TA5hGoa7pPhWhJQ==

Or dump out the /dev/random and /dev/urandom devices with cat and company.

How to learn everything you ever wanted to know about Linux sockets

Viewing network socket data is something SysAdmins do often. We could be called on to see if a connection is established to a host, if an application is listening on a given port, or we may need to review the network connection table as a whole to see what a server is doing (this is especially valuable when DDOS attacks occur). The netstat and lsof tools provide quite a bit of visibility into this area, but I’ve recently started firing up the ss (socket stat) tool when I need to view socket information. Socket stat can display pretty much everything you ever wanted to know about the connections on your server. To get a basic breakdown of ports that applications are listening on, you can run ss with the “-l” option:

$ ss -l

Recv-Q Send-Q                         Local Address:Port                             Peer Address:Port   
0      128                                       :::ssh                                        :::*       
0      128                                        *:ssh                                         *:*       
0      128                                127.0.0.1:ipp                                         *:*       
0      128                                      ::1:ipp                                        :::*       

To view the processes that are using each listening socket, you can run ss with the “-p” option:

$ ss -p

State      Recv-Q Send-Q      Local Address:Port          Peer Address:Port   
CLOSE-WAIT 1      0          192.168.1.1:57666         192.168.1.2:http     users:(("gvfsd-http",16992,14))

To display the amount of memory being consumed by the socket buffers, you can use the ss “-m” option (this is quite handy!):

$ ss -e -m

State       Recv-Q Send-Q                    Local Address:Port                        Peer Address:Port   
CLOSE-WAIT  1      0                        192.168.1.1:57666                       192.168.1.2:http     uid:500 ino:40834026 sk:ffff88022d3b2080
	 mem:(r360,w0,f3736,t0)

Additionally, you can use the ss “-s” option to summarize all of the socket states:

$ ss -s

Total: 571 (kernel 589)
TCP:   17 (estab 10, closed 0, orphaned 0, synrecv 0, timewait 0/0), ports 0

Transport Total     IP        IPv6
*	  589       -         -        
RAW	  0         0         0        
UDP	  10        6         4        
TCP	  17        14        3        
INET	  27        20        7        
FRAG	  0         0         0       

There are also options to display information about specific socket types (UNIX domain, UDP, TCP, etc), and to dig deep into the connection table information (see the “-i” option for further details). If you have a current release of CentOS, RHEL or Fedora, this awesome tool should be on your system. It’s part of the iproute package.