OpenSSH’s VPN

Most SSH clients have the ability to perform local and remote port forwarding.  This is a pretty neat use of SSH if you haven’t ever seen it before.  OpenSSH can take it one step further and provide a full VPN solution encrypting all network traffic on all ports between two machines.  This is pretty powerful stuff.  This is useful for a quick-and-dirty way to encrypt all traffic between two machines.  For a longer term solution, you might want to check out how to configure IPsec or use OpenVPN.  All three solutions have some really cool features and benefits.

OpenSSH is the most widely deployed open source SSH client / server solution today.  Most Linux/BSD hosts I have encountered will use this as the client/server by default.  Sun’s ssh packages are based off of the OpenSSH distribution with some tweaks and modifications, but its pretty close to OpenSSH’s implementation.

Anyways, to create a VPN tunnel between two machines, two variables in sshd_config need to be tweaked as well as the presence of the tun/tap kernel module.  This kernel module is available on most Linux/BSD distributions.  It may have to be compiled and inserted into the Solaris kernel, or you can download it here.

[root@locutus ~]# uname -a
Linux locutus 2.6.25-14.fc9.i686 #1 SMP Thu May 1 06:28:41 EDT 2008 i686 i686 i386 GNU/Linux
[root@locutus ~]# lsmod | grep tun
tun                    11776  2
[root@locutus ~]# modinfo tun
filename:       /lib/modules/2.6.25-14.fc9.i686/kernel/drivers/net/tun.ko
alias:          char-major-10-200
license:        GPL
author:         (C) 1999-2004 Max Krasnyansky <maxk@qualcomm.com>
description:    Universal TUN/TAP device driver
srcversion:     12C02361DF16200902CDE64
depends:
vermagic:       2.6.25-14.fc9.i686 SMP mod_unload 686 4KSTACKS
So, the tun module has already been inserted into my running kernel.  Next, set these two variables in sshd_config and have sshd re-read its configuration files…

[root@locutus ~]# egrep ‘PermitTunnel|PermitRootLogin’ /etc/ssh/sshd_config
PermitRootLogin yes
PermitTunnel yes

[root@locutus ~]# service sshd reload
Reloading sshd:                                            [  OK  ]
[root@locutus ~]#

Make sure you don’t mess up sshd_config and reload the daemon as your only way to access the machine!  Console access is always a good thing.

Now all we need to do is to open the VPN tunnel itself.  Here, I open a VPN tunnel to localhost (not really useful) but you can get the idea…

[root@locutus ~]# ssh -w any:any root@localhost

The any:any defines the local:remote “tun” device.  We could have put 0:1 here, (tun0 as the local, tun1 as the remote) but any:any takes care of it for us in case there are any pre-existing tun devices in use.) or 0:0 if we were accessing a real remote machine.  (I can’t define two “tun0” devices on localhost)

So, after my SSH session connects, sure enough the tun devices exist..

[root@locutus ~]# ifconfig tun0
tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
POINTOPOINT NOARP MULTICAST  MTU:1500  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[root@locutus ~]# ifconfig tun1
tun1      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
POINTOPOINT NOARP MULTICAST  MTU:1500  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

Now all we have to do is assign them some network addresses and let them know that its a point-to-point connection between the two…

[root@locutus ~]# ifconfig tun0 10.0.0.10 pointopoint 10.0.0.11
[root@locutus ~]# ifconfig tun1 10.0.0.11 pointopoint 10.0.0.10

[root@locutus ~]# ifconfig tun0
tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.0.0.10  P-t-P:10.0.0.11  Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)

[root@locutus ~]# ifconfig tun1
tun1      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00
inet addr:10.0.0.11  P-t-P:10.0.0.10  Mask:255.255.255.255
UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
RX packets:0 errors:0 dropped:0 overruns:0 frame:0
TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:0 (0.0 b)  TX bytes:0 (0.0 b)
Nice!  Now all network traffic between the machines using those newly created addresses will be tunneled through SSH!  Could this be a solution for NFSv3 and firewalls?  ;-)

If there are multiple machines on the “other side” of the VPN that you would want to connect to, you will also need to add a route..

[root@locutus ~]# route add -net 10.0.0.0 netmask 255.255.255.0 gw 10.0.0.10 tun0

[root@locutus ~]# netstat -rn
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
10.0.0.11       0.0.0.0         255.255.255.255 UH        0 0          0 tun0
10.0.0.10       0.0.0.0         255.255.255.255 UH        0 0          0 tun1
10.0.0.0        10.0.0.10       255.255.255.0   UG        0 0          0 tun0

So any traffic destined for 10.0.0.0/24 is gonna go out tun0 to be routed onwards and upwards.

We’ve also got to set up an arp entry..

[root@locutus ~]# arp -sD 10.0.0.11 eth0 pub

[root@locutus ~]# arp -an
? (10.0.0.11) at * PERM PUP on eth0

Ryan McGuire wrote a pretty cool blog entry about not only this feature, but some other really neat things with OpenSSH.  I based a lot of this article after learning them from his site.  Check out his python script that will automate a lot of this for you.  Thanks Ryan!

Respect my ~/.Xauthority !#@$!

South Park is a hilarious show, and I think that Cartman is the best character.  One of Cartman’s classic lines is “YOU WILL RESPECT MY AUTHORITAH!#!”

So Cartman wasn’t a unix geek and wasn’t talking about X11 Forwarding / SSH, but maybe there is a moral to the story.

You have to execute some sort of GUI program on a remote host and it requires root access in order to execute (or you have to change to a different user to execute the GUI with correct permissions)… 

At first, when you logged into the machine for the first time without X11 forwarding enabled, your ~/.Xauthority file doesn’t exist…

cartman@locutus:~$ ls -l ~/.Xauthority
ls: /home/cartman/.Xauthority: No such file or directory

So you log back out, and when you ssh back into the remote machine, you remember to forward X11 by issuing..

$ ssh -X <user>@<remote box>

i.e.

$ ssh -X cartman@locutus
Linux locutus 2.6.22-14-generic #1 SMP Sun Oct 14 23:05:12 GMT 2007 i686
cartman@locutus:~$ xclock

Sure enough, you fire up /usr/bin/xclock (or /usr/openwin/bin/xclock) and verify that the GUI program displays back on your local desktop.  

cartman@locutus:~$ echo $DISPLAY
localhost:10.0

Sweet.  Next, when you change users..

cartman@locutus:~$ su –
Password

:
root@locutus:~# id
uid=0(root) gid=0(root) groups=0(root)


root@locutus:~# xclock
Error: Can’t open display:

 

you loose your X11 forwarding.  DOH!

So whats the solution here?  You can’t log in directly to the box as the root user (this should always be disabled.  Its really bad practice if it isn’t) — and you don’t really want to throw a SSH key into /root/.ssh/authorized_keys for obvious reasons — so what’s there to do?

When you SSH into a machine with X11 forwarding, it opens a TCP port, tunnels it through SSHD, and stores this information into a MIT cookie file in your home directory called ~/.Xauthority

All we have to do is “move” this information along with us when we change users.  We can use the xauth command to manipulate this for us.  First, lets display what the value of our cookie is.  Note the :10 matching up to our $DISPLAY variable…

cartman@locutus:~$ xauth list
locutus/unix:10  MIT-MAGIC-COOKIE-1  e2cba22d040f0e75dcbd203ee40736de

Now lets change users..

cartman@locutus:~$ su –
Password:

root@locutus:~# ls -l /root/.Xauthority
ls: /root/.Xauthority: No such file or directory

root@locutus:~# xauth list

So no MIT cookies currently exist… That makes sense because we didn’t X11 port forward into the root account.. Lets add one.  Don’t forget the “/unix” after the FQDN..
root@locutus:~# xauth add locutus/unix:10 MIT-MAGIC-COOKIE-1
d203ee40736de0e75dcb

xauth:  creating new authority file /root/.Xauthority

Exceeeelent….
root@locutus:~# xauth list
localhost/unix:10  MIT-MAGIC-COOKIE-1  e2cba22d040f0e75dcbd203ee40736de

We’re not done yet… The last thing we have to do is to set our $DISPLAY variable to the same display as above..  Right now it may be set to null…

root@locutus:~# echo $DISPLAY

root@locutus:~# xclock
Error: Can’t open display:

So lets set it to localhost:10.0

root@locutus:~# export DISPLAY=localhost:10
root@locutus:~# xclock

Sure enough, we get xclock to display.  We didn’t have to be the root user in this example.  Any local user could perform the same function.

Alternativly, xauth also has a “merge” function to where you can read an existing ~/<user/.Xauthority file to merge with another.  This really is only going to work for the root user (unless you chmod) because the permissions on this file is octal 600…

root@locutus:~# ls -l /home/cartman/.Xauthority
-rw——- 1 cartman cartman 53 2008-04-05 00:22 /home/cartman/.Xauthority

Lets remove the previous Xauthority we had in place…

root@locutus:~# xauth list
locutus/unix:10  MIT-MAGIC-COOKIE-1  e2cba22d040f0e75dcbd203ee40736de

root@locutus:~# xauth remove locutus/unix:10

 

And then we’ll use the merge function pointing at a specific .Xauthority file…
root@locutus:~# xauth merge /home/cartman/.Xauthority

Sure enough, it imported correctly..

root@locutus:~# xauth list
locutus/unix:10  MIT-MAGIC-COOKIE-1  e2cba22d040f0e75dcbd203ee40736de

Our DISPLAY variable matches the display above and xclock starts up without any errors.
root@locutus:~# echo $DISPLAY
localhost:10
root@locutus:~# xclock

 

When xauth is invoked without any options, it brings up a menu based configuration utility thats pretty neat… Here’s “xauth info” in action…

root@locutus:~# xauth
Using authority file /root/.Xauthority
xauth> help
    add dpyname protoname hexkey   add entry
    exit                           save changes and exit program
    extract filename dpyname…    extract entries into file
    help [topic]                   print help
    info                           print information about entries
    list [dpyname…]              list entries
    merge filename…              merge entries from files
    nextract filename dpyname…   numerically extract entries
    nlist [dpyname…]             numerically list entries
    nmerge filename…             numerically merge entries
    quit                           abort changes and exit program
    remove dpyname…              remove entries
    source filename                read commands from file
    ?                              list available commands
    generate dpyname protoname [options]  use server to generate entry
    options are:
      timeout n    authorization expiration time in seconds
      trusted      clients using this entry are trusted
      untrusted    clients using this entry are untrusted
      group n      clients using this entry belong to application group n
      data hexkey  auth protocol specific data needed to generate the entry

xauth> info
Authority file:       /root/.Xauthority
File new:             no
File locked:          no
Number of entries:    1
Changes honored:      yes
Changes made:         no
Current input:        (stdin):2

There are also all sorts of security implecations surrounding ~/.Xauthority where the root user or administrator could hijack X11 sessions.  This article is a great read and I suggest taking a look at it when you have a chance.  It also goes into better detail on the steps of how the X11 forward occurs and security hazards surrounding it.

 

When SSH permissions bite!

Last week I set up several Linux and Solaris hosts to use key based authentication. For some reason two of the hosts continued to prompt me for a password, even though the server and client were configured correctly to used DSA keys (I was using the same config on all of the servers, so I knew it worked). When I traced the sshd daemon on one of the hosts that was misbehaving, I saw the following just before the password prompt was displayed:

$ strace -f -p `pgrep sshd`
<.....>
stat("/home/matty/.ssh/authorized_keys", {st_mode=S_IFREG|0664, st_size=1026, ...}) = 0
open("/home/matty/.ssh/authorized_keys", O_RDONLY) = 4
lstat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/matty", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
lstat("/home/matty/.ssh", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
lstat("/home/matty/.ssh/authorized_keys", {st_mode=S_IFREG|0664, st_size=1026, ...}) = 0
lstat("/home", {st_mode=S_IFDIR|0755, st_size=4096, ...}) = 0
lstat("/home/matty", {st_mode=S_IFDIR|0700, st_size=4096, ...}) = 0
fstat(4, {st_mode=S_IFREG|0664, st_size=1026, ...}) = 0

The strace output made me realize that $HOME/.ssh might not be set to 0700, or the authorized_keys file might not be set to 0600. It turns out the permissions on both entries were set incorrectly, and after adjusting the permissions (which got borked by an incorrect umask entry in /etc/profile), everything worked as expected. As a side note, I am curious why the SSH daemon doesn’t log the permission errors when run with multiple debug flags. This would make a fantastic RFE! :)

Limiting access to OpenSSH features with the Match keyword

With the introduction of OpenSSH 4.3p2, Darren Tucker introduced the “Match” keyword. This super nifty keyword can be used to limit features to specific users, hosts and groups, and allows administrators to enforce granular feature access (e.g., key-based authentication can only be used from specific hosts or subnets). To use the Match feature, the Match directive needs to added to the sshd_config configuration file with a criteria to enforce, and a set of directives to allow or deny. The Criteria can be either “User,” “Group,” “Address,” or “Host,” and wildcards are supported when the Host criteria is used. The following example shows how the Match keyword can be used to limit TCP port forwarding and X11 port forwarding to the user stew:

AllowTcpForwarding no
X11Forwarding no

Match User stew
         AllowTcpForwarding yes
         X11Forwarding yes

The full list of directives that are supported inside a Match block are listed in OpenSSH bug #1180. This is an incredible feature, and something I have wanted for years. Thanks Darren for adding this!

Verifying SSH host keys

If you use SSH to access remote servers, the ssh client will prompt you each time you connect to a new server, and ask you to accept the servers host key:

$ ssh mail.me.net

The authenticity of host 'mail.me.net (1.2.3.4)' can't be established.
RSA key fingerprint is 72:c6:5f:e7:85:c8:23:5f:c6:c9:99:88:dd:aa:bb:dd.
Are you sure you want to continue connecting (yes/no)? yes 
Warning: Permanently added 'mail.me.net,1.2.3.4' (RSA) to the list of known hosts.
matty@mail.me.net's password: 

How do you determine if the fingerprint presented is valid? The easiest way to validate the key is to login to the server through the console and run the ssk-keygen utility with the “-l” (list fingerprint) and “-f” (file to check) options:

$ ssh-keygen -l -f /etc/ssh/ssh_host_rsa_key
2048 72:c6:5f:e7:85:c8:23:5f:c6:c9:99:88:dd:aa:bb:dd ssh_host_rsa_key.pub

If you can’t access the console, you can ask someone who already has access to read off the ssh-keygen output. While it’s a bit of a pain, it is a small step in the name of secure access.