One of my colleagues came to me with an ssh problem a few weeks back, and I thought I would share my findings here. The issue centered around ssh hanging when you tried to invoke a command on a remote server, similar to this:
$ ssh host "ls -la"
When I added the “-vv” (verbose output) option to ssh to root cause the problem, I got the following information:
debug1: authentications that can continue: publickey,password
debug3: start over, passed a different list
debug3: authmethod_lookup publickey
debug3: authmethod_is_enabled publickey
debug1: next auth method to try is publickey
debug1: key does not exist: //.ssh/id_rsa
debug1: try pubkey: //.ssh/id_dsa
debug1: read SSH2 private key done: name dsa w/o comment success 1
debug3: sign_and_send_pubkey
debug1: sig size 20 20
debug2: we sent a publickey packet, wait for reply
debug1: Remote: Forced command: /usr/local/bin/yikes
debug1: ssh-userauth2 successfull: method publickey
debug1: fd 6 setting O_NONBLOCK
debug1: fd 7 setting O_NONBLOCK
debug1: fd 8 IS O_NONBLOCK
debug1: channel 0: new [client-session]
debug1: send channel open 0
debug1: Entering interactive session.
debug2: callback start
debug1: client_init id 0 arg 0
debug1: channel request 0: shell
debug2: callback done
debug1: channel 0: open confirm rwindow 0 rmax 16384
debug2: channel 0: rcvd adjust 32768
debug1: channel 0: read< =0 rfd 6 len 0
debug1: channel 0: read failed
debug1: channel 0: input open->drain
debug1: channel 0: close_read
debug1: channel 0: input: no drain shortcut
debug1: channel 0: ibuf empty
debug1: channel 0: input drain->closed
debug1: channel 0: send eof
debug1: channel 0: rcvd eof
debug1: channel 0: output open->drain
debug1: channel 0: obuf empty
debug1: channel 0: output drain->closed
debug1: channel 0: close_write
debug1: channel 0: send close
debug1: channel: 0 rcvd request for exit-status
debug1: cb_fn 267a4 cb_event 91
debug1: channel 0: rcvd close
debug1: channel 0: full closed2
debug1: channel_free: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i8/0 o128/0 fd -1/-1)
Upon examining the output, I started to wonder why the message “The following conenctions are open” was displayed. Upon examing the OpenSSH source code, it looks like the ssh client examines the active sessions prior to exiting, and displays this message if the session doesn’t exit cleanly (this is grossly oversimplified, and I will refer you to channel.c and clientloop.c. if you are interested in the gory details).
I took my knowledge from the code and started searching the OpenSSH archives for issues related to commands hanging. I immediately found a thread describing an SSH command hang problem, and saw that Darren Tucker had posted a patch to address the issue. Since we were using the SSH client provided by Sun, I wanted to see if this patch had been integrated into the Sun SSH client. To get the current patch revision of the SSH client, I used the Solaris what(1) command:
$ what /usr/bin/ssh
/usr/bin/ssh:
$ `OpenBSD: ssh.c, v 1.69 2000/10/27 07:32:19 markus Exp `
$ `OpenBSD: sshconnect.c, v 1.79 2000/09/17 15:52:51 markus Exp `
$ `OpenBSD: sshconnect1.c, v 1.8 2000/10/12 09:59:19 markus Exp `
$ `OpenBSD: sshconnect2.c, v 1.27 2000/10/19 16:45:16 provos Exp `
$ `OpenBSD: clientloop.c,v 1.39 2000/10/27 07:48:22 markus Exp `
$ `OpenBSD: atomicio.c,v 1.7 2000/10/18 18:04:02 markus Exp `
$ `OpenBSD: authfd.c,v 1.29 2000/10/09 21:51:00 markus Exp `
$ `OpenBSD: authfile.c,v 1.20 2000/10/11 20:27:23 markus Exp `
$ `OpenBSD: bufaux.c,v 1.13 2000/09/07 20:27:50 deraadt Exp `
$ `OpenBSD: buffer.c,v 1.8 2000/09/07 20:27:50 deraadt Exp `
$ `OpenBSD: channels.c,v 1.72 2000/10/27 07:48:22 markus Exp `
$ `OpenBSD: cipher.c,v 1.37 2000/10/23 19:31:54 markus Exp `
$ `OpenBSD: cli.c,v 1.2 2000/10/16 09:38:44 djm Exp `
$ `OpenBSD: compat.c,v 1.27 2000/10/31 09:31:58 markus Exp `
$ `OpenBSD: dispatch.c,v 1.5 2000/09/21 11:25:34 markus Exp `
$ `OpenBSD: hostfile.c,v 1.20 2000/09/07 20:27:51 deraadt Exp `
$ `OpenBSD: key.c,v 1.11 2000/09/07 20:27:51 deraadt Exp `
$ `OpenBSD: kex.c,v 1.12 2000/10/11 20:27:23 markus Exp `
$ `OpenBSD: log.c,v 1.11 2000/09/30 16:27:43 markus Exp `
$ `OpenBSD: mac.c,v 1.1 2001/02/11 12:59:24 markus Exp `
$ `OpenBSD: match.c,v 1.9 2000/09/07 20:27:52 deraadt Exp `
$ `OpenBSD: util.c, v 1.6 2000/10/27 07:32:19 markus Exp `
$ `OpenBSD: mpaux.c,v 1.14 2000/09/07 20:27:52 deraadt Exp `
$ `OpenBSD: nchan.c, v 1.19 2000/09/07 20:27:52 deraadt Exp `
$ `OpenBSD: packet.c, v 1.38 2000/10/12 14:21:12 markus Exp `
$ `OpenBSD: readpass.c, v 1.12 2000/10/11 20:14:39 markus Exp `
$ `OpenBSD: readconf.c, v 1.49 2000/10/11 20:27:23 markus Exp `
$ `OpenBSD: rsa.c, v 1.16 2000/09/07 20:27:53 deraadt Exp `
$ `OpenBSD: tildexpand.c, v 1.8 2000/09/07 20:27:55 deraadt Exp `
$ `OpenBSD: ttymodes.c, v 1.8 2000/09/07 20:27:55 deraadt Exp `
$ `OpenBSD: uidswap.c, v 1.9 2000/09/07 20:27:55 deraadt Exp `
$ `OpenBSD: uuencode.c, v 1.7 2000/09/07 20:27:55 deraadt Exp `
$ `OpenBSD: xmalloc.c, v 1.8 2000/09/07 20:27:55 deraadt Exp `
$ `OpenBSD: canohost.c,v 1.26 2001/04/18 14:15:00 markus Exp `
$ `OpenBSD: compress.c,v 1.9 2000/09/07 20:27:50 deraadt Exp `
$ `OpenBSD: crc32.c,v 1.7 2000/09/07 20:27:51 deraadt Exp `
$ `OpenBSD: ssh-dss.c, v 1.2 2000/12/19 23:17:58 markus Exp `
$ `OpenBSD: ssh-rsa.c, v 1.4 2001/01/16 19:20:06 markus Exp `
SunOS 5.9 Generic 114356-01 Jan 2003
When I searched the sunsolve patch database for patch identifier 114356, I immediately saw the following information:
[4763864 Non-interactive ssh can hang after remote command has
exited](http://sunsolve.sun.com/search/document.do?assetkey=urn:cds:docid:1-21-113273-06-1)
This matched the issue we were experiencing! I immediately applied the patch, and everything started working! I love debugging issues like this, and hope Sun incorporates this into the Recommended patch cluster in the future!!!