Speeding up SSH (SCP) data transfers

I’ll be the first to admit that I’m an SCP addict. It doesn’t matter what kind of data I’m working with, if it can be turned into an object that I move around with scp I’m in! One thing I’ve always noticed with scp is the dismal out of the box performance. I read quite some time back on the openssh mailing list that there were some fixed buffers inside openssh that prevented copy operations from fully utilizing high speed network links.

This past week my fellow bloging partner Mike sent me a link to the high performance ssh project. This nifty project created a patch that replaces the fixed length buffers with values determined at runtime. This is pretty fricking cool, and I hope they get their changes merged into the openssh source code! The site that currently hosts the HPN-SSH code also has a good article on tuning your TCP/IP network stack. Good stuff!

2 thoughts on “Speeding up SSH (SCP) data transfers”

  1. For real-world usage scp has at least three other annoying problems.

    1. it doesn’t copy into tmp file and use rename, but just rewrite the file. Thats ugly and dangerous as it’s not atomic and e.g doesn’t work for running executables (gives ebusy) on linux.

    2. it can’t copy the content of a directory (not -r of the directory but just the content of the directory into another directory).
    Most users therefore use shell globbing, and most of the time forget about dotfiles.

    3. it hangs on directory loops.

    all of which rsync doesn’t have.

  2. I dug through the mailing lists at one point, and it looks like the decision was made a while back to just increase the static buffer size rather than add the complexity of the HPN-SSH patch. It did help, but not as much. I’ve actually had surprisingly good results just piping things through ssh instead of using scp.

Leave a Reply

Your email address will not be published. Required fields are marked *