Running commands across multiple servers with clusterssh

I periodically need to perform repetitive maintenance operations (e.g., patching systems) on groups of servers, which typically requires me to run a similar set of commands on multiple hosts. To make my life easier, I use the super useful clusterssh utility to interactively run commands across a group of servers.

Cluster SSH is super easy to configure, and uses the concept of a “cluster” to define a group of similar nodes. To get up and running with clusterssh, you will first need to run cssh with the “-u” option to generate a configuration file (this step is optional, but creating a config file reduces start up time):

$ cssh -u > $HOME/.csshrc

Once the configuration file is created, you can open the file in a text editor, and change the settings to fit your administration preferences. Some of the settings that can be changed are ssh connection parameters, the placement of xterms on your screen, the placement of the master command window, window titles, etc. In additional to modifying look and feel type items, you also need to add one or more clusters, which are logical groupings of machines. On one of my desktops, I have a “backup” cluster that I use to connect to our master and media servers, a “web” cluster to connect to our web servers, an “app” cluster to connect to our application servers, and a “db” cluster to connect to our database servers. To define a new cluster, you can add a group description followed by an equal sign and one or more hosts, and then register that grouping using the “clusters” key word. Here are the entries from my .csshrc file:

clusters = backup web app db
backup = nbmed01 nbmed02 nbmed03 nbmed04 nbmas01 nbmas02
web = web01 web02 web03 web04 web05 web06 web07 web08
app = app01 app02
db = db01 db02

After the configuration settings are adjusted and one ore more clusters are defined, you can run the cssh utility with the name of the cluster to connect to:

$ cssh web

This wil open up one xterm window per server, and create an SSH connection to the server. If you have RSA or DSA keys set up, each xterm window will display a prompt on the server. In addition to opening one xterm window per server, cssh will also create a command window to send commands to all of the servers that you are connected to (you can also type into individual xterms, which is useful for running commands on just one system). This makes patching, host files updates (when you can’t completely depend on DNS) and the such super super easy! This is some serious bling yizos! Jeah!

5 thoughts on “Running commands across multiple servers with clusterssh”

  1. If you don’t require interactive SSH access, “dsh” (http://www.netfort.gr.jp/~dancer/software/dsh.html.en) is another great cluster SSH program. It has similar concepts of groups, and you can run the same command concurrently all hosts in a group (dsh -cMg [group] [cmd]) or with a max of [n] concurrent processes (dsh -F [n] -Mg [group] [cmd]). It works great for us for installing software (dsh -cMg group apt-get install -y imagemagick) or even for tailing logfiles (dsh -cMg group tail -f /opt/logs/apache.log).

    We actually drive a lot of our machine configurations through DSH groups – software deployments, lists of which apps get started where, and hopefully soon we will also key our monitoring off of this data. (Since the group lists control which applications get started where, we know exactly which apps need to be monitrored on which hosts).

    I will have to check out cssh for those times where an interactive shell is required on the systems, thanks for the tip!

    -Andy

  2. I have been working with the big AIX virtualized servers and they use a ibm version of dsh also, I have just found the open source replacement its called pdsh, Parallel Distributed Shell, http://sourceforge.net/projects/pdsh . The main difference with this one that i like is that it comes with a dcp, which will let you distributed copy files from your master node to all the child nodes etc… I have used all of the ones mentioned above, but i like this one the best :D

    cheers, great blog btw :D

    /rod

  3. Or if you use KDE Konsole, open all the sessions you want and go to “View” menu, click “Send Input to All Sessions”

  4. I’m getting a “font missing: ISO8859-15 fro Window” error after finally getting it compiled. Anybody got this working on Solaris 10?

Leave a Reply

Your email address will not be published. Required fields are marked *