Integrating ssh-agent into your login process

Most of my readers utilize SSH keys to access remote systems. The security benefits are well known, and key-based authentication makes automating remote tasks a whole lot easier. When you use key-based authentication it becomes imperative to protect your private key, since a third party could access your systems if they were able to gain access to your account. The SSH key generator (ssh-keygen) will attempt to encrypt your private key by default, and can also be used ssh-keygen to add a password to a private key after the fact.

With passwords comes prompts, and with prompts comes frustration. To alleviate this frustration you can use the ssh-agent process to minimize the number of times you need to type your password. Ssh-agent stores your private keys securely in memory, and hands them out to the ssh process when you attempt to connect to remote systems. Keys are added to ssh-agent through the ssh-add command line utility, which will prompt you for your private key password prior to adding them to the keys held in memory by ssh-agent.

Each time you access a remote system the ssh client will contact the ssh-agent process to acquire your private keys. If you start ssh-agent and run ssh-add to add your private keys when you login to a server, you will now be able to access other hosts using key-based authentication without a password for the length of the shell session. You will find this especially useful when you are using tools like clusterit to manage remote machines.

To automate the process above, I like to modify my bash environment to prompt me for the password when I login to my servers. The ssh-add prompt I get looks similar to this:

$ ssh

Last login: Tue Jan 17 20:37:54 2012 from
Starting an ssh-agent process
Enter passphrase for /home/matty/.ssh/id_dsa:
Identity added: /home/matty/.ssh/id_dsa (/home/matty/.ssh/id_dsa)

Once I’ve input the correct password I can then access other systems freely and without a password. So what exactly did I do to integrate ssh-agent into my shell environment? First I added an exec statement to create an ssh-agent process and make the bash process a child of it (the reasons why this is required are documented in the SSH FAQ):

$ grep ssh-agent .bash_profile

# Start an ssh-agent process and start bash as a child of it
echo “Starting an ssh-agent process”
exec ssh-agent bash

Once ssh-agent is tied into the environment I call ssh-add from my .bashrc to add my private keys:

$ grep ssh-add .bashrc

# Add the encrypted private keys to my in-memory key store with ssh-add.
echo “Calling ssh-add to add my private key to ssh-agent”

The second entry is what causes the following password prompt when I login to my servers:

Enter passphrase for /home/matty/.ssh/id_dsa:

I always try to do everything I can to improve security, and this is definitely one of those every admin should do to protect their beloved private key(s). :) If you are doing something differently, please share your thoughts in the comment section below.

A couple of gotchas with the OpenSSH chroot() implementation

I previously discussed the OpenSSH Match directive, and how it can be used to chroot SSH and SFTP users. Over the past couple of months I’ve encountered some gotchas with the chroot implementation in OpenSSH. Since I had to figure these items out myself, I figured I would share my findings here so folks wouldn’t need to spend hours looking at source code (if you want to geek out and see how this works, check out session.c in the OpenSSH source code).

The first gotcha occurs when the users home directory doesn’t have the correct permissions. The directory the user is chroot()’ed into needs to be root owned, and in order for the user to see the contents of the top level directory the group permissions need to be read/execute. A user that is going to be chroot()’ed into /chroot/user will need the following permissions:

$ mkdir /chroot/user

$ chmod 750 /chroot/user

$ chown root:user /chroot/user

If the permissions aren’t set 100% correctly you will be immediately disconnected:

$ sftp -o Port=222 user@
Connecting to…
user@’s password:
Read from remote host Connection reset by peer
Connection closed

And the following error will be written to your system logs:

Apr 23 14:06:27 vern sshd[13714]: fatal: bad ownership or modes for chroot directory “/chroot/user”

There is also an issue with directory write-ability. Users will not be able to write to the top-level directory of the chroot() due to the strong permissions that are required. If you want your user to be able to create files and directories they will need to change into a directory off of “/” prior to doing so. If you want to place the user into this directory you can replace the home directory field in /etc/passwd with this directory:

$ grep user /etc/passwd

In this example user will be chroot’ed into /chroot/user, then a chdir() will occur and the user will be placed into the directory home:

$ sftp -o Port=222 user@
Connecting to…
user@’s password:
sftp> pwd
Remote working directory: /home

With a little understanding the OpenSSH chroot() implementation can definitely be made quite usable.

How to encrypt an SSH private key

If you are using SSH key-based authentication you should be encrypting your private key. This ensures that if someone breaks into your server and steals your keys, they won’t be able to utilize them to access other systems. If your private key isn’t encrypted you can use the ssh-keygen utilities “-p” option to do so:

$ ssh-keygen -p -f id_dsa
Enter old passphrase:
Key has comment ‘id_dsa’
Enter new passphrase (empty for no passphrase):
Enter same passphrase again:
Your identification has been saved with the new passphrase.

This option can be used to change the password used to encrypt a private key, and to add a password to an existing private key. Viva la OpenSSH!

Speeding up SSH (SCP) data transfers

I’ll be the first to admit that I’m an SCP addict. It doesn’t matter what kind of data I’m working with, if it can be turned into an object that I move around with scp I’m in! One thing I’ve always noticed with scp is the dismal out of the box performance. I read quite some time back on the openssh mailing list that there were some fixed buffers inside openssh that prevented copy operations from fully utilizing high speed network links.

This past week my fellow bloging partner Mike sent me a link to the high performance ssh project. This nifty project created a patch that replaces the fixed length buffers with values determined at runtime. This is pretty fricking cool, and I hope they get their changes merged into the openssh source code! The site that currently hosts the HPN-SSH code also has a good article on tuning your TCP/IP network stack. Good stuff!

A simple and easy way to restart dropped SSH sessions on Linux servers

I frequently use OpenSSH port forwards to move around my various networks, and there is nothing worse than an SSH connection dropping when you make heavy use of them. Recently I came across the autossh utility, which provides a crazy easy way to monitor ssh sessions and restart them when they are reset or dropped. To use this awesome little tool, you can invoke autossh with the options you would normally pass to your ssh client:



$ autossh -D -L 8000

In the example above I am creating a dynamic local port forward that will accept connections on localhost:8000. To change how autossh manages the sessions and logs it creates, you can set one of more environment variables. The variables are documented in the autossh(1) manual page, and allow you control how data is logged, connection attributes and how keepalives operate. If autossh is in use and your ssh connection dies for any reason, it will be restarted and your port forwards will be transparently restored. Bada boom, bada bing!

Chroot’ing users with openssh

I recently learned about the new ChrootDirectory in OpenSSH 5.2, and wanted to play around with it to see what it was capable of. To begin my quest, I started off by creating a couple of users that would be chroot’ed to their home directories when they logged into the server with sftp. Once the users were created, I added the following configuration stanza to my sshd_config file to chroot these users when they logged in with their sftp client:

Subsystem       sftp    internal-sftp

Match user u1,u2,u3
         ChrootDirectory /home/%u
         X11Forwarding no
         AllowTcpForwarding no
         ForceCommand internal-sftp

Once these directives where added, I started up the daemon in debug mode:

$ /usr/local/sbin/sshd -ddd -f /usr/local/etc/sshd_config

Debug mode will cause the daemon to log verbosely to stdout, which is extremely useful for locating problems with new configuration directives. Now that the daemon was running, I tried to login with the user u1:

$ sftp -oPort=222 u1@
Connecting to…
u1@’s password:
Read from remote host Connection reset by peer
Connection closed

The first attempt was a no go, but luckily verbose logging made debugging this issue a snap:

debug3: mm_get_keystate: Getting compression state
debug3: mm_get_keystate: Getting Network I/O buffers
debug3: mm_share_sync: Share sync
debug3: mm_share_sync: Share sync end
debug3: safely_chroot: checking ‘/’
debug3: safely_chroot: checking ‘/home/’
debug3: safely_chroot: checking ‘/home/u1’
bad ownership or modes for chroot directory “/home/u1”

After changing /home/u1 to be owned by root, I was able to login and poke around:

$ sftp -oPort=222 u1@
Connecting to…
u1@’s password:
sftp> pwd
Remote working directory: /
sftp> ls -l
drwxr-xr-x 2 1001 1001 4096 Mar 15 15:03 uploads
sftp> cd uploads
sftp> ls -l
-rw-r–r– 1 1001 1001 39655552 Mar 15 15:04 techtalk1.mp3
sftp> put techtalk2*
Uploading techtalk2.mp3 to /uploads/techtalk2.mp3
techtalk2.mp3 3% 3776KB 2.3MB/s 00:39 ETA^
sftp> ls -l
-rw-r–r– 1 1001 1001 5046272 Mar 15 15:11 techtalk2.mp3
-rw-r–r– 1 1001 1001 39655552 Mar 15 15:04 techtalk1.mp3

This is super useful, though building chroot jails for normal SSH sessions will require a bit more work (i.e., you need to populate the chroot directory with all the config files and binaries needed to run a typical shell session). Makejail can make this a WHOLE lot easier, and I am about to submit a patch to the makejail developers to allow it to work on Solaris hosts. OpenSSH rocks!