Chroot’ing users with openssh

I recently learned about the new ChrootDirectory in OpenSSH 5.2, and wanted to play around with it to see what it was capable of. To begin my quest, I started off by creating a couple of users that would be chroot’ed to their home directories when they logged into the server with sftp. Once the users were created, I added the following configuration stanza to my sshd_config file to chroot these users when they logged in with their sftp client:

Subsystem       sftp    internal-sftp

Match user u1,u2,u3
         ChrootDirectory /home/%u
         X11Forwarding no
         AllowTcpForwarding no
         ForceCommand internal-sftp



Once these directives where added, I started up the daemon in debug mode:

$ /usr/local/sbin/sshd -ddd -f /usr/local/etc/sshd_config

Debug mode will cause the daemon to log verbosely to stdout, which is extremely useful for locating problems with new configuration directives. Now that the daemon was running, I tried to login with the user u1:

$ sftp -oPort=222 u1@192.168.1.15
Connecting to 192.168.1.15…
u1@192.168.1.15’s password:
Read from remote host 192.168.1.15: Connection reset by peer
Connection closed

The first attempt was a no go, but luckily verbose logging made debugging this issue a snap:

debug3: mm_get_keystate: Getting compression state
debug3: mm_get_keystate: Getting Network I/O buffers
debug3: mm_share_sync: Share sync
debug3: mm_share_sync: Share sync end
debug3: safely_chroot: checking ‘/’
debug3: safely_chroot: checking ‘/home/’
debug3: safely_chroot: checking ‘/home/u1’
bad ownership or modes for chroot directory “/home/u1”

After changing /home/u1 to be owned by root, I was able to login and poke around:

$ sftp -oPort=222 u1@192.168.1.15
Connecting to 192.168.1.15…
u1@192.168.1.15’s password:
sftp> pwd
Remote working directory: /
sftp> ls -l
drwxr-xr-x 2 1001 1001 4096 Mar 15 15:03 uploads
sftp> cd uploads
sftp> ls -l
-rw-r–r– 1 1001 1001 39655552 Mar 15 15:04 techtalk1.mp3
sftp> put techtalk2*
Uploading techtalk2.mp3 to /uploads/techtalk2.mp3
techtalk2.mp3 3% 3776KB 2.3MB/s 00:39 ETA^
sftp> ls -l
-rw-r–r– 1 1001 1001 5046272 Mar 15 15:11 techtalk2.mp3
-rw-r–r– 1 1001 1001 39655552 Mar 15 15:04 techtalk1.mp3

This is super useful, though building chroot jails for normal SSH sessions will require a bit more work (i.e., you need to populate the chroot directory with all the config files and binaries needed to run a typical shell session). Makejail can make this a WHOLE lot easier, and I am about to submit a patch to the makejail developers to allow it to work on Solaris hosts. OpenSSH rocks!

Monitoring network bandwidth with bwm-ng

There are a bunch of utilities available to monitor bandwidth utilization on Linux hosts, and I’ve touched on a few in previous posts. I recently came across bwm-ng while perusing the Debian package repository, and decided to try it out. When bwm-ng is executed without any arguments, it provides a relatively simple curses interface with throughput statistics for each interface in the system:

$ bwm-ng

  bwm-ng v0.6 (probing every 0.500s), press 'h' for help
  input: /proc/net/dev type: rate
  |         iface                   Rx                   Tx                Total
  ==============================================================================
               lo:           0.00 KB/s            0.00 KB/s            0.00 KB/s
             eth0:        2275.89 KB/s           57.56 KB/s         2333.45 KB/s
  ------------------------------------------------------------------------------
            total:        2275.89 KB/s           57.56 KB/s         2333.45 KB/s



But the simplicity of the tool stops there, since there are a SLEW of options to control the output format, and whether or not sampled data is written to a file. This is a nifty utility, but I think I will stick with iftop.

Using paste to create columns from input data

I periodically need to take input data from various utilities and convert it to columnar data. There are a million ways to do this, but I have come to rely on the paste utility to perform this task:

$ ls

1	11	13	15	17	19	20	4	6	9
10	12	14	16	18	2	3	5	78


$ ls | paste – – –

1	10	11
12	13	14
15	16	17
18	19	2
20	3	4
5	6	78
9		


In the output above, paste will take the input given to it and print the data in 3 columns (you can add more hyphens to get more columns of data). If anyone has some interesting little tidbits such as this, feel free to add them to the comments section. Thanks!

Parallelizing shell tasks with project middleman and xargs

While poking around the Internet, I came across a link to project middleman. The project provides an easy way for administrators to parallelize tasks inside shell scripts, and is described rather nicely in the README file that comes with the source code:

“The philosophy behind mdm is that users should benefit from their multi-core systems without making drastic changes to their shell scripts. With mdm, you annotate your scripts to specify which commands might benefit from parallelization, and then you run it under the supervision of the mdm system. At runtime, the mdm system dynamically discovers parallelization opportunities and run the annotated commands in parallel as appropriate.”

And when they mention annotating a shell script, it really is as simple as placing the “mdm-run” binary in front of tasks that can be parallelized (you can also define an I/O profile if tasks will interfere with each others I/O streams):

mdm-run convert2ogg *.mp3

This is pretty sweet, and I need to play around with this a bit more on my quad core desktop. Rock on!

*** UPDATE ***

I just came across Parallelizing Jobs with xargs, which describes how to use the xargs “-P” option to parallelize tasks:

$ ls *.mp3 | xargs -P 8 -n 1 convert2ogg

The “-P” argument to xargs will cause 8 convert2ogg processes to be kicked off, and the “-n” option will ensure that only one argument of the ls output is passed to each process that is created. This is sweet, and I can DEFINITELY see myself using this super useful argument in the future!!!!!

Observing Linux performance with dstat

I mentioned a few posts back that I was playing with a bunch of Linux utilities. One of these utilities is dstat, which allows you to view a number of system statistics (disk utilization, network utilization, page activity, etc.) from the command line:

$ dstat

----total-cpu-usage---- -dsk/total- -net/total- ---paging-- ---system--
usr sys idl wai hiq siq| read  writ| recv  send|  in   out | int   csw 
  0   0 100   0   0   0|1591B  134k|   0     0 |   0   0.6B|   1     4 
  1   7  90   0   1   1|   0     0 |2939k   69k|   0     0 |2784  3190 
  2   6  88   0   2   2|   0     0 |2882k   68k|   0     0 |2738  3079 
  0   9  90   0   0   1|   0   552k|2724k   64k|   0     0 |2591  2859 
  1  10  71  13   2   4|   0    22M|2782k   67k|   0     0 |2693  2773 
  0   4  92   0   2   2|   0     0 |2646k   63k|   0     0 |2487  2781 
  0   4  95   0   0   1|   0     0 |1712k   42k|   0     0 |1696  2069 
  1   7  91   0   0   1|   0     0 |2593k   63k|   0     0 |2489  2781 
  2   6  89   0   2   1|   0  2896k|2695k   64k|   0     0 |2555  2918 
  1   6  65  23   2   3|   0    21M|2760k   65k|   0     0 |2688  2942 



The default output contains a number of useful statistics, but additional stats can be displayed if you pass a few options to dstat:

$ dstat -tcdgimnsy

-----time----- ----total-cpu-usage---- -dsk/total- ---paging-- -interrupts ------memory-usage----- -net/total- -swp/total- ---system--
  date/time   |usr sys idl wai hiq siq| read  writ|  in   out |  17    18 | used  buff  cach  free| recv  send| used  free| int   csw 
08-03 09:43:42|  0   0 100   0   0   0|1607B  134k|   0   0.6B|   1     1 |  19M   12M  104M  116M|   0     0 |  92k  729M|   1     5 
08-03 09:43:43|  0   0 100   0   0   0|   0     0 |   0     0 |   0     5 |  19M   12M  104M  116M| 198B  580B|  92k  729M|   5    14 
08-03 09:43:44|  0   1  99   0   0   0|   0     0 |   0     0 |   0     4 |  19M   12M  104M  116M| 132B  580B|  92k  729M|   4     8 
08-03 09:43:45|  0   0 100   0   0   0|   0     0 |   0     0 |   0     4 |  19M   12M  104M  116M| 132B  580B|  92k  729M|   4     8 
08-03 09:43:46|  0   0 100   0   0   0|   0     0 |   0     0 |   0     4 |  19M   12M  104M  116M| 132B  580B|  92k  729M|   4    14 
08-03 09:43:47|  0   1  99   0   0   0|   0     0 |   0     0 |   0     4 |  19M   12M  104M  116M| 132B  580B|  92k  729M|   4     7 
08-03 09:43:48|  0   0 100   0   0   0|   0   240k|   0     0 |   2     4 |  19M   12M  104M  116M| 132B  580B|  92k  729M|   6    13 
08-03 09:43:49|  0   0 100   0   0   0|   0     0 |   0     0 |   0     4 |  19M   12M  104M  116M| 132B  580B|  92k  729M|   4     8 
08-03 09:43:50|  0   0 100   0   0   0|   0     0 |   0     0 |   0     4 |  19M   12M  104M  116M| 132B  580B|  92k  729M|   4     8 



The output above will overflow a standard 80 character wide display, so you may need to adjust the size of your terminal window to see everything. Dstat is an amazing utility, and yet another tool I plan to add to my performance analysis toolkit.

Viewing process utilization on Linux hosts with pidstat

While playing around with my Debian 5.0 host last week, I came across the pidstat utility. Pidstat allows you to display a number of statistics for processes running on a system (and threads inside that process), which can be incredibly useful for troubleshooting performance problems. To use pidstat to get a breakdown of how a given process is using the processors on a system, pidstat can be run with the “-p” option and the process ID to interrogate:

$ pidstat -p 3297 2

Linux 2.6.26-1-686 (disarm) 	03/07/2009 	_i686_

08:07:18 AM       PID   %user %system    %CPU   CPU  Command
08:07:20 AM      3297   11.06   74.37   85.43     0  dd
08:07:22 AM      3297   13.00   70.00   83.00     0  dd
08:07:24 AM      3297   10.95   72.64   83.58     0  dd



If you are looking to see how much I/O a specific process is responsible for, you can run pidstat with the “-d” option:

$ pidstat -d -p 3288 2

Linux 2.6.26-1-686 (disarm) 	03/07/2009 	_i686_

08:06:35 AM       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
08:06:37 AM      3288      3.98 113319.40      0.00  dd
08:06:39 AM      3288      4.00 112514.00      0.00  dd
08:06:41 AM      3288      4.00  81454.00      0.00  dd



And finally, to view paging activity per process, you can run pidstat with the “-r” option (and optionally the “-t” flag if you want to see thread activity):

$ pidstat -r -t -p 3338 2

Linux 2.6.26-1-686 (disarm) 	03/07/2009 	_i686_

08:10:20 AM       PID       TID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
08:10:22 AM      3338        -       0.00      0.00    3180    596   0.23  dd
08:10:22 AM        -       3338      0.00      0.00    3180    596   0.23  |__dd

08:10:22 AM       PID       TID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
08:10:24 AM      3338        -       0.00      0.00    3180    596   0.23  dd
08:10:24 AM        -       3338      0.00      0.00    3180    596   0.23  |__dd

08:10:24 AM       PID       TID  minflt/s  majflt/s     VSZ    RSS   %MEM  Command
08:10:26 AM      3338        -       0.00      0.00    3180    596   0.23  dd
08:10:26 AM        -       3338      0.00      0.00    3180    596   0.23  |__dd



Pidstat is pretty darn cool, and I will definitely be using this in the future! Nice!