Using fail2ban to lock out unwanted visitors to your SSH daemon

I have a number of digital ocean droplets and AWS instances that I use for personal projects. For convenience I leave SSH open to the world so I can access these systems wherever I’m at. This DEFINITELY isn’t a best practice but my personal instances don’t contain anything sensitive and can be rebuilt in minutes with ansible. A wide open TCP port 22 doesn’t come without issues though. If you leave your SSH daemon open to the world you are going to see a myriad of failed log attempts in /var/log/secure:

Jan 25 16:10:35 oscar sshd[10450]: Failed password for root from port 23476 ssh2

To enhance the security of my SSH daemon I enforce key-based authentication, update openssh hourly w/ an ansible playbook and enable fail2ban to blacklist IPs that try to log into my servers too many times. Setting up fail2ban on a CentOS 7 Linux host is extremely easy. First, you need to install the packages:

$ yum -y install fail2ban fail2ban-systemd jwhois

Once the packages are installed you will should review the default fail2ban configuration settings in /etc/fail2ban/jail.conf. This file shouldn’t be edited directly since system updates will replace it and flush away any changes you made. Customizations should go into /etc/fail2ban/jail.local. Changes made to this file will override the system defaults. Here is a sample /etc/fail2ban/jail.local file to get you started:

# Ban hosts for one hour:
bantime = 3600

# Use the multiport iptables action
banaction = iptables-multiport

# Who to send e-mail to when an IP is banned
sendername = fail2ban@oscar
mta = sendmail

# Where to send info if an IP is banned (e-mail, logs, etc.)
action = %(action_mwl)s

enabled = true

Once this file is in place you will need to enable and start the service:

$ systemctl enable fail2ban
$ systemctl enable fail2ban

If you didn’t encounter an error the fail2ban-client should list the jail it created for the SSH service:

$ fail2ban-client status
|- Number of jail:      1
`- Jail list:   sshd

To see IPs that are being blacklisted you can grep for Ban in /var/log/fail2ban.log:

$ grep Ban fail2ban.log
2017-01-25 15:57:25,365 fail2ban.actions [10213]: NOTICE [sshd] Ban

Or run iptables to list the IPs it added to the f2b-sshd chain:

$ sudo iptables -S f2b-sshd
-N f2b-sshd
-A f2b-sshd -s -j REJECT –reject-with icmp-port-unreachable
-A f2b-sshd -j RETURN

If e-mail notifications are configured via the “action” directive you will receive an e-mail each time fail2ban blacklists an IP. Now that you see how easy it is to set this up I would like to provide a bit more detail on the terminology I used above. A fail2ban “jail” is a combination of filters and actions. Filters are the criteria fail2ban uses to figure out if something bad is happening and are defined in /etc/fail2ban/filter.d/.conf. Actions describe what to do (add an iptables rule to reject traffic from a source IP, add a blackhole route, etc.) if a filter matches. All of the actions are defined in /etc/fail2ban/action.d. The manual pages are terrific and all of the files in /etc/fail2ban are THOROUGHLY commented. If you use ansible you can automate the fail2ban installation with a fail2ban.yml playbook. This is a great utility and super easy to use!

Using Linux auditing to observe program executions and file accesses

Linux systems have a rather nice auditing framework built into the kernel. This framework consists of a kernel component that inspects and filters events and a userland daemon (auditd) that takes the events and archives them to persistent storage. These audit log events can then be searched and summarized with the ausearch and aureport utilities. The following ausearch command will dump all audit records for UID 7005:

$ ausearch --uid-all 7005 | head -4
time->Thu Oct 20 15:01:49 2016
type=PROCTITLE msg=audit(1476990109.442:153031): proctitle="-bash"
type=SYSCALL msg=audit(1476990109.442:153031): arch=c000003e syscall=1 success=yes exit=1024 a0=1 a1=55b32ff2fe40 a2=400 a3=6553726567616e61 items=0 ppid=18901 pid=19071 auid=7005 uid=7005 gid=7005 euid=7005 suid=7005 fsuid=7005 egid=7005 sgid=7005 fsgid=7005 tty=pts9 ses=6031 comm="ps" exe="/usr/bin/ps" key=(null)

Each audit event contains the type of event (system call executed, file read, etc.) as well as a bunch of user and system context. To control the types of events that get audited you can create an audit.rules file in /etc/audit/rules.d. This file can contain control rules, file system rules and system call rules. Control rules are used to control the behavior of the audit subsystem (e.g., what to do when the disk fills up, how many audit buffers to allocate, etc.). File system rules allow you to audit when files or directories are accessed and system call rules allow you to audit system calls (full list in /usr/include/asm/unistd_64.h).

To illustrate how useful auditing is lets say you are tasked with figuring out what a black box application does. Furthermore, lets assume this application runs as the user fmep with the UID 7005. To log EVERY system call generated by processes with UID 7005 we can add the following audit rule (**WARNING** DO NOT ENABLE THIS ON PRODUCTION SYSTEMS WITHOUT PROPER TESTING ***END WARNING***):

$ auditctl -a always,exit -S all -F auid=7005

The “-a” option tells auditd to append the rule to the chain, “always,exit” audits system call events at exit, “-S all” audits all system calls and the “-F auid=7005” filters events where the loginuid is set to 7005 (more info on AUID is available here). To verify the rule was added we can run auditctl with the “-l” option:

$ auditctl -l
-a always,exit -S all -F auid=7005

Once the rules are activated you will start seeing events similar to the following in /var/log/audit/audit.log:

type=SYSCALL msg=audit(1476990167.749:160759): arch=c000003e syscall=14 success=yes exit=0 a0=2 a1=7ffe53ea8fc0 a2=0 a3=8 items=0 ppid=18884 pid=18900 auid=7005 uid=7005 gid=7005 euid=7005 suid=7005 fsuid=7005 egid=7005 sgid=7005 fsgid=7005 tty=(none) ses=6031 comm=”sshd” exe=”/usr/sbin/sshd” key=(null)

Each system call audit event contains the system call number as well as a good deal of context (pid, uid, ppid, etc.) to describe the event. To better understand the list of commands and files being accessed by a given UID I wrote auditloginfo. This script parses the audit log and produces a report listing the commands executed, files accessed and a summary of the system calls made:

$  sudo --uid 7005
Audit report for AUID 7005

System calls made:
   read 207
   stat 179
   close 147
   open 142
   rt_sigprocmask 141
   clock_gettime 120
   write 88
Commands executed:
   clear  2
   /bin/basename bash  2
   uname -m  1
   ps -ef  1
   ls --color=auto -la  1

Files opened:
   /etc/ 17
   /lib64/ 15
   /etc/passwd 14
   /usr/lib/locale/locale-archive 12
   /lib64/ 8

I’ve found auditing rather useful and it’s amazing what you can discover when you enable it.

Tracking Linux UIDs across process creation

When you support mission critical systems it is critically important to know WHAT is running on your systems and WHO is running it. In the past it was sometimes a chore to tell which UID ran a program. This was especially true in the early days when programs fork()’ed and execve()’ed after escalating privileges.

The which UID ran a program mystery has been mostly solved (I say mostly because user namespaces throw a few kinks into the equation) by auditing and the PAM loginuid module. PAM loginuid helps solve this problem by maintaining a /proc/PID/loginuid entry with the UID of the user that interacted with a service (e.g., ssh) that is configured to use pam_loginuid. You can get a list of PAM loginuid’ed services with grep:

$ grep --files-with-matches loginuid /etc/pamd/* | paste - - - - -
atd	crond	gdm-autologin	gdm-fingerprint	gdm-password
gdm-pin	gdm-smartcard	lightdm	lightdm-autologin	login
lxdm	pluto	remote	sshd	systemd-user

Once the loginuid is set the original UID will follow process creation:

$ id
uid=1000(matty) gid=1000(matty) .....

$ cat /proc/$$/loginuid 

$ sudo -i

$ id
uid=0(root) gid=0(root) groups=0(root)

# cat /proc/$$/loginuid 

# bash

# cat /proc/$$/loginuid 

Now this gets especially interesting when you enable auditing. If auditing is enabled and privileges are escalated each event will contain an AUID with the value of the actual user that ran a privileged command:

type=SYSCALL msg=audit(1477500276.323:92205): arch=c000003e syscall=231 a0=1 a1=3c a2=1 a3=3 items=0 ppid=30493 pid=30544 auid=7005 uid=7005 gid=7005 euid=7005 suid=7005 fsuid=7005 egid=7005 sgid=7005 fsgid=7005 tty=pts1 ses=1196 comm=”dmidecode” exe=”/usr/sbin/dmidecode” key=(null)

No more seeing uid=0 and having to track down the underlying user or application by digging through logs. This is also incredibly useful when conducting RCAs and doing postmordems. To learn more about this feature check out pam_loginuid(8), the pam_loginuid source and the following link from the samhein team.

Using the rsyslog MySQL plug-in to send syslog data to a SQL database

I have been experimenting with ways to better manage the logs my servers generate. Depending on who you ask, folks will recommend sending your logs to a remote syslog server that writes the logs to disk, some may recommend sending it to a log analysis tool similar to splunk, and others would recommend feeding it to a SQL database. I’ve talked before about setting up syslog-ng for remote logging, and in this case I wanted to experiment with something new. I also didn’t have money to buy a tool like splunk, so I decided to start experimenting with funneling syslog data into a MyQSL database.

Setting up syslog to write messages to a MySQL database is crazy easy to do on CentOS 6. The built-in syslog daemon (rsyslog) has database plug-ins for several opensource databases, which can be installed with the yum package manager:

$ yum install rsyslog-mysql

Once the plug-in is installed you can run the provided createDB.sql script (this is part of the rsyslog-mysql package) to create a database (the default database will be named Syslog, though you can edit the createDBL.sql file if you want to call it something else) as well as the tables the log entries will be stored in:

$ rpm -q -l rsyslog-mysql-4.6.2-12.el6.x86_64 | grep createDB.sql

$ mysql -u root -h localhost –password

mysql> source /usr/share/doc/rsyslog-mysql-4.6.2/createDB.sql

If this completes successfully you should have two brand spanking new tables:

mysql> use Syslog;

mysql> show tables;

| Tables_in_Syslog       |
| SystemEvents           |
| SystemEventsProperties |

The SystemEvents table is where log data is stored and has the following structure:

mysql> desc SystemEvents;

| Field              | Type             | Null | Key | Default | Extra          |
| ID                 | int(10) unsigned | NO   | PRI | NULL    | auto_increment |
| CustomerID         | bigint(20)       | YES  |     | NULL    |                |
| ReceivedAt         | datetime         | YES  |     | NULL    |                |
| DeviceReportedTime | datetime         | YES  |     | NULL    |                |
| Facility           | smallint(6)      | YES  |     | NULL    |                |
| Priority           | smallint(6)      | YES  |     | NULL    |                |
| FromHost           | varchar(60)      | YES  |     | NULL    |                |
| Message            | text             | YES  |     | NULL    |                |
| NTSeverity         | int(11)          | YES  |     | NULL    |                |
| Importance         | int(11)          | YES  |     | NULL    |                |
| EventSource        | varchar(60)      | YES  |     | NULL    |                |
| EventUser          | varchar(60)      | YES  |     | NULL    |                |
| EventCategory      | int(11)          | YES  |     | NULL    |                |
| EventID            | int(11)          | YES  |     | NULL    |                |
| EventBinaryData    | text             | YES  |     | NULL    |                |
| MaxAvailable       | int(11)          | YES  |     | NULL    |                |
| CurrUsage          | int(11)          | YES  |     | NULL    |                |
| MinUsage           | int(11)          | YES  |     | NULL    |                |
| MaxUsage           | int(11)          | YES  |     | NULL    |                |
| InfoUnitID         | int(11)          | YES  |     | NULL    |                |
| SysLogTag          | varchar(60)      | YES  |     | NULL    |                |
| EventLogType       | varchar(60)      | YES  |     | NULL    |                |
| GenericFileName    | varchar(60)      | YES  |     | NULL    |                |
| SystemID           | int(11)          | YES  |     | NULL    |                |

The column descriptions are pretty much self explanatory, and you can reference the rsyslog documentation to get more specifics on the purpose of each column. In order for rsyslog to be able to write to the database, you will need to create a user (using root is not recommended) and grant the privileges to allow the user to INSERT new data. For my purposes I created an rsyslog user and restricted them to INSERT’ing data into just the Syslog database:

mysql> grant INSERT on Syslog.* to rsyslog identified by “PASSWORD_HERE”;

This completes the MySQL configuration. To tell rsyslog to start sending messages to the Syslog database, you will need to add directives similar to the following to /etc/rsyslog.conf. You will also need to restart the rsyslog service:

$ModLoad ommysql

The first line loads the MySQL plug-in, and the second line tells rsyslog to send all of the log entries (you can pair this down to specific facilities and priorities) to the Syslog database on It will use the provided user (rsyslog in this example) and password (PASSWORD_HERE in this example) to login to the database. If everything worked as expected you should be able to view the log entries in the SystemEvents table:

$ mysql -u root -h localhost –password

mysql> use Syslog;

mysql> select * from SystemEvents limit 5;

| ID | CustomerID | ReceivedAt          | DeviceReportedTime  | Facility | Priority | FromHost  | Message                                                                                                | NTSeverity | Importance | EventSource | EventUser | EventCategory | EventID | EventBinaryData | MaxAvailable | CurrUsage | MinUsage | MaxUsage | InfoUnitID | SysLogTag | EventLogType | GenericFileName | SystemID |
|  1 |       NULL | 2012-02-11 15:42:01 | 2012-02-11 15:42:01 |        0 |        6 | centos6-1 | imklog 4.6.2, log source = /proc/kmsg started.                                                         |       NULL |       NULL | NULL        | NULL      |          NULL |    NULL | NULL            |         NULL |      NULL |     NULL |     NULL |          1 | kernel:   | NULL         | NULL            |     NULL |
|  2 |       NULL | 2012-02-11 15:42:01 | 2012-02-11 15:42:01 |        5 |        6 | centos6-1 |  [origin software="rsyslogd" swVersion="4.6.2" x-pid="3891" x-info=""] (re)start |       NULL |       NULL | NULL        | NULL      |          NULL |    NULL | NULL            |         NULL |      NULL |     NULL |     NULL |          1 | rsyslogd: | NULL         | NULL            |     NULL |
|  3 |       NULL | 2012-02-11 15:42:25 | 2012-02-11 15:42:25 |        0 |        6 | centos6-2 | imklog 4.6.2, log source = /proc/kmsg started.                                                         |       NULL |       NULL | NULL        | NULL      |          NULL |    NULL | NULL            |         NULL |      NULL |     NULL |     NULL |          1 | kernel:   | NULL         | NULL            |     NULL |
|  4 |       NULL | 2012-02-11 15:42:25 | 2012-02-11 15:42:25 |        5 |        6 | centos6-2 |  [origin software="rsyslogd" swVersion="4.6.2" x-pid="5932" x-info=""] (re)start |       NULL |       NULL | NULL        | NULL      |          NULL |    NULL | NULL            |         NULL |      NULL |     NULL |     NULL |          1 | rsyslogd: | NULL         | NULL            |     NULL |
|  5 |       NULL | 2012-02-11 15:42:22 | 2012-02-11 15:42:22 |        1 |        5 | centos6-1 |  test                                                                                                  |       NULL |       NULL | NULL        | NULL      |          NULL |    NULL | NULL            |         NULL |      NULL |     NULL |     NULL |          1 | matty:    | NULL         | NULL            |     NULL |
5 rows in set (0.00 sec)

Awesome! Once you have all of your hosts pointing to your database you can use the power of SQL to sift through the data and correlate events (I wonder how useful this would be when applied to digital computer forensics). You can also use tools like LogAnalyzer to visualize your log data. I’ve thought of hundreds of things I can do with my log data, I just need to spend some time coding them up! :)

Integrating ssh-agent into your login process

Most of my readers utilize SSH keys to access remote systems. The security benefits are well known, and key-based authentication makes automating remote tasks a whole lot easier. When you use key-based authentication it becomes imperative to protect your private key, since a third party could access your systems if they were able to gain access to your account. The SSH key generator (ssh-keygen) will attempt to encrypt your private key by default, and can also be used ssh-keygen to add a password to a private key after the fact.

With passwords comes prompts, and with prompts comes frustration. To alleviate this frustration you can use the ssh-agent process to minimize the number of times you need to type your password. Ssh-agent stores your private keys securely in memory, and hands them out to the ssh process when you attempt to connect to remote systems. Keys are added to ssh-agent through the ssh-add command line utility, which will prompt you for your private key password prior to adding them to the keys held in memory by ssh-agent.

Each time you access a remote system the ssh client will contact the ssh-agent process to acquire your private keys. If you start ssh-agent and run ssh-add to add your private keys when you login to a server, you will now be able to access other hosts using key-based authentication without a password for the length of the shell session. You will find this especially useful when you are using tools like clusterit to manage remote machines.

To automate the process above, I like to modify my bash environment to prompt me for the password when I login to my servers. The ssh-add prompt I get looks similar to this:

$ ssh

Last login: Tue Jan 17 20:37:54 2012 from
Starting an ssh-agent process
Enter passphrase for /home/matty/.ssh/id_dsa:
Identity added: /home/matty/.ssh/id_dsa (/home/matty/.ssh/id_dsa)

Once I’ve input the correct password I can then access other systems freely and without a password. So what exactly did I do to integrate ssh-agent into my shell environment? First I added an exec statement to create an ssh-agent process and make the bash process a child of it (the reasons why this is required are documented in the SSH FAQ):

$ grep ssh-agent .bash_profile

# Start an ssh-agent process and start bash as a child of it
echo “Starting an ssh-agent process”
exec ssh-agent bash

Once ssh-agent is tied into the environment I call ssh-add from my .bashrc to add my private keys:

$ grep ssh-add .bashrc

# Add the encrypted private keys to my in-memory key store with ssh-add.
echo “Calling ssh-add to add my private key to ssh-agent”

The second entry is what causes the following password prompt when I login to my servers:

Enter passphrase for /home/matty/.ssh/id_dsa:

I always try to do everything I can to improve security, and this is definitely one of those every admin should do to protect their beloved private key(s). :) If you are doing something differently, please share your thoughts in the comment section below.

How to figure out if a processes has been chroot()’ed

A number of applications (e.g., custom chroot jails, openssh, vsftp, apache) support the ability to chroot themselves. To find out if a process called chroot() at startup, you can check the /proc/<pid>/root entry for the process. For non-chrooted processes this entry will point to /:

$ ps auxwww | grep [s]endmail

root      3643  0.0  0.1  69032  2344 ?        Ss    2011   0:01 sendmail: accepting connections
smmsp     3651  0.0  0.0  59784  1780 ?        Ss    2011   0:01 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue

$ cd /proc/3643

$ ls -lad root

lrwxrwxrwx 1 root root 0 Jan 22 10:23 root -> /

For a chrooted process the root directory will point to the directory passed to the chroot() system call:

$ ps auxwww | grep [n]amed

named    18298  0.0  2.3 243632 49084 ?        Ssl   2011  15:16 /usr/sbin/named -u named -t /var/named/chroot

$ cd /proc/18298

$ ls -lad root

lrwxrwxrwx 1 named named 0 Jan 22 10:19 root -> /var/named/chroot

Chroot environments can be made secure, especially if you follow the coding practices discussed in Building Secure Software and Using Chroot Securely. These are must reads for anyone who plans to use chroot()!