My ssh story

One of the first things that get under your skin when you deal with hundreds of servers daily is the need to enter the password every time you log in to a remote machine. ssh has an immediate answer to this pain: a key-based authentication which does not require much advertising these days. ^[1]

The challenge is that when you support a really large fleet of machines that belong to separate organizational branches, it’s very often that — for various reasons — you will not be allowed to install personal ssh keys on each and every unix box. And so you are stuck with entering passwords at the prompt.

SAs are a resourceful bunch with a visceral aversion to repetitive and dull tasks. Obviously, no one wanted to type in their password, that had to adhere to security requirements of having lower and upper case letters, numbers, non-alphanumberic characters, and a minimal length,^[2] literally hundreds of times every day.

Hence, someone quickly came up with a solution in the form of expect script. expect, for those not old enough to remember, is (still is!) a Tcl extension that lets automate interactions with text-based interfaces. At the very basic level, expect allows you to specify what output to expect from a program, and what input to provide in response. In our modern “DevOpsy” days, I don’t see it much used, but back then it was a quite popular tool, especially for establishing dial-up connections.

This script:

Now, with such script you can assign your password to a variable, and connect to hosts quickly as if you had ssh keys configured:

That was definitely a win! And it didn’t stop there. By modifying the script one could easily perform mass-actions over a list of hosts. For example, if we replace interact with something like:

We then can collect the uptime statistics across many hosts.

Granted, the output is a bit messy, but you can work with it. In many cases, a simple grep would do the trick, but for more complex cases, the awk is the most wondrous of things:

Surprisingly — to me back then, but not anymore — organizations with a sizeable server estate often struggle to know what is that they actually have there. Which machines have cronjobs configured, and what are they? Which servers have locally added user accounts? What are versions of web servers in production?

If there is no automated and resonably credible way to collect information off servers to answer such questions, the only alternative is to ssh in and capture files or command outputs with necessary details.

This is why it was so important to be able to automatically log in, execute commands, and capture their output. The expect script, although far from being perfect, allowed us to carry out quite sizable queries in an automated fashion.

Sometime after getting used to the expect+ssh combination, the “approved” repository for our linux boxes received the sshpass tool. This tool is a beauty — small, simple, smart, and reliable.

sshpass is essentially a better version of the above expect script — more reliable and easier to handle. As soon as it became available, it replaced the expect script almost completely:

The life was bright with sshpass, ssh’ing was a breeze, and I could fairly quickly collect large sets of data and sift through it to get the information I needed.

As the number of systems I was supporting grew, so did the time required to gather information or execute some command across them. Naturally, some way of parallelizing ssh execution was in order. After initial attempts to write a script that would loop over the list of hosts, and control the number of processes working in the background, managing their stdout, doing error handling, and so on and so forth, the appreciation for the amount of time necessary to write such instrument properly descended on me. To save time, I looked for existing tools to do this job.

The first tool I found was GNU Parallel — an amazing work of art with countless features, making it essentially a swiss-army-knife for parallelizing processes. With it you could do something like:

I could not use the GNU parallel in my environment, though, as it was absent from the “approved” repository of tools. A secondary concern involved the fact that owing to its power and flexibility, parallel required spending time on the documentation again and again, and still, it was easy to make mistakes. And when you deal with lots of servers, you don’t want to make mistakes. So I looked for some other tool and discovered the pdsh.

pdsh is yet another absolutely fantastic tool, which is very thoughtfully made. You can use it to run commands in parallel over a list of hosts. In combination with sshpass it is a tool of tremendous power:

As you can see, pdsh automatically prepends lines with the hostname which produced it. This makes processing the output with awk a picnic. The proper handling of stdout and stderr allows to nicely handle cases when you need to collect the information present only on a subset of hosts:

In the example above, grep on hosts which didn’t have /apache/conf/tokens.conf generated errors like:

pdsh is smart in that it redirects to local stderr file any output that the command it runs generated to the remote host’s stderr. This really helps a lot in handling such output, and simple redirection as in the example can filter out unnecessary noise.

  TODO : improve example to show why scp won’t work

An interesting side-use of pdsh was an improved mass-collection of files. And its ability to run any binary, not just ssh, allowed to use it in a very similar way to GNU parallel:

How could the problem with 2FA be solved? This time I had to really start digging into ssh documentation. I learnt about connection multiplexing that ssh supports. Turns out, once you establish your first ssh connection to a host, you can reuse this connection for the all subsequent ssh connections to that same host. And because the connection is already authenticated, you will not be asked for the password again!

This is a perfect solution to logging in to 2FA-enabled jump hosts: establish one connection to it, pass the authentication, and leave it running in the background. Reuse that connection for all new ssh connections to the same jump host.

Moreover, we can even avoid remembering and using -M and -S options all the time by adding them to the default ssh configuration for all or a group of hosts. This can be done by creating an entry in ~/.ssh/config like this:

In the above example, we enable connection multiplexing and specify that the socket files should be created in ~/.ssh/ directory. %C is substituted with a hash of a string that consists of local and remote hostnames, destination port, and remote username. This way, you get a new socket for each new connection to a different host, or with a different username.

Furthermore, to avoid keeping a terminal occupied by the initial ssh connection that we need for authentication, or accidentally closing it, we can run it in the background, leaving it there doing nothing:

The ssh process started and asked for the password, and upon authenticating went into the background as was directed by -f option. -N option tells ssh not to execute any remote command, just maintain the connection.

Now this is wonderful! An ordinary ssh command would take the advantage of an existing connection, and ask for a password just once, when it connects to a host behind the jump host. Surely, we then could use sshpass to supply password like this:

To my chagrin, I discovered that this trick doesn’t work, and I was still getting a password prompt. The reason appeared to be that:

So, in my naive attempt above, sshpass fools the first ssh process, which is the ssh to bastion host, whereas the second ssh, which is the one asking for a password, is not handled by sshpass.

Okay, so how do we deal with that password prompt coming from the second ssh command? We can surely revive the expect script and modify it accordingly. But we have got this far not to turn back to where we once were. Instead, we need that the ssh process we run on our local machine was the one asking for the password to the final destination host.

The ssh has an interesting option called ProxyCommand which specifies the command to use to connect to the server on a TCP level and provide network transport for ssh. If we can somehow establish a TCP connection between a workstation and destination server over the jump host, we can then tell ssh to use it. When it comes to establishing connections between hosts, the standard unix toolbox has an outstanding utility for that — netcat.

Netcat can establish a tcp connection to ssh port on a remote host, and tie it to stdin/stdout of a process that invoked it. We can’t connect directly through to the host behind a jump host, right? But we can use ssh to connect to jump host and execute netcat (nc) there:

This command is effectively establishing a TCP connection between the source and destination hosts via a jump host. All that is left is to tell another ssh to use this connection:

Here, ssh bastion nc %h %p uses a previously established multiplexed connection to connect to a jump host without asking for a password, and then executes nc to connect to the destination host. %h and %p inside ProxyCommand option are substituted with the hostname and port from the parent ssh command, in our example tb-c0 and 22 (default).

Once this is done, the parent (leftmost) ssh command uses the stdin/stdout of the proxy command, to talk directly to the destination host that sits behind the jump host! Next, it will negotiate a connection, and ask us for the password. But since it is the ssh process that runs on a source machine that asks for a password, the sshpass can detect this and supply the password from the environment variable defined previously.

To avoid specifying ProxyCommand as a parameter on a command line, we can again turn to ssh configuration file, and direct the group of hosts which require jumping to use it automatically:

With this setup we now can:

As a result, we can use pdsh again to run commands across many hosts:

Once again my life was bright, and I could access and run commands across hundreds of servers in a very efficient manner, yay!

The combination of ssh and netcat proved to be so convenient and widely used, that in OpenSSH 5.4 a native implementation was added that didn’t require netcat to be installed. All you had to do was to use -W option, so instead of

you had to specify

with exactly the same result.

As configurations with jump hosts proliferated, the use of them has been further simplified in OpenSSH 7.3 by an addition of -J command line parameter and ProxyJump configuration file option. With it, the necessary incantation for jumping became simply ProxyJump bastion, nothing more.

The new -J parameter has a neat feature worthy particular mentioning. If you have not just one, but a chain of jump hosts, earlier you had to specify ProxyCommand for each jump host and it was getting messy. With -J you can simply specify them over a comma, like so:

Very convenient.

Working with a large fleet of machines means that often you need to move information between servers. It can be files, scripts, some other bits and pieces. One of the first commands that I have learnt of this ilk was:

This command moves a directory tree with files from one host to another. On the machine you are logged in you run tar that produces a zipped tarball to its stdout, which is piped to stdin of an ssh process. Ssh connects to the target host and passes own stdin to stdin of a command it spawns remotely, which in our case is tar. Finally, tar reads stdin and unpacks it to /home/user.

Once you start using pipes and redirection with ssh it becomes your second nature. Need to append a local file with a file from a remote box?

Backup a database, gzip it, and store it on a remote host, all of that without using disk space on the database server?

Check what is recorded in mp3 file on a server?

Execute a script that you have locally without copying it to the remote server? This is useful when you need to run it across many servers, or if the command you want to execute becomes too complex for a one-liner:

Or maybe debug HTTP traffic on a server with whireshark installed on your workstation:

At some point, I realised that such commands really let me connect processes that run on separate hosts in the very same manner as I do it when dealing with processes running on the same host. In other words, ssh let me create process pipes that worked across the network! This is when ssh stopped being a “remote shell” to me, and became a “network interprocess pipe”.

Not everyone knows, though, that there are configuration options that can be specified in authorized_keys file to limit who can use this key, and what can be done while using it.

Such options are specified in front of the public key line in authorized_keys file. The most useful options to me are:

There are more options supported, for more details check AUTHORIZED_KEYS FILE FORMAT section of sshd man page.

There are situations when you need to do something on a remote server with your private ssh keys. For example, you might use a jumphost which requires a different key than the destination host. Or you might need to pull a repository from github for deployment using your personal ssh keys.

Copying and leaving the keys to the remote server might not the best way to handle such situations. ssh supports a better way to feature called Agent Forwarding. Agent is a daemon that holds ssh keys and allows clients to verify them. You can tell ssh to forward verification requests from a remote system to your workstation where you keep private keys and run ssh-agent.

  TODO : Improve description, single key is enough.

Let’s look at a diagram how this works:

For this process to work, ssh connection between workstation and host A has to allow Agent Forwarding. This can be achived by specifying -A switch to ssh when connecting to host A.

Even less know is the fact that ssh supports certificate-based authentication. SSH identity certificates are very useful when you manage a large fleet of servers. The present of Certificate Authority (CA) mean that you do not need to distribute keys for people across the servers. The ssh client will present to a server with a signed personal certificate which can be validated with CA certificate, confirming user’s credentials without keeping public keys on servers.

The access revocation becomes easier too; ssh server supports Key Revocation Lists (KRL) which allow to manage centrally the list of revoked certificates.

ssh has a built-in SOCKS proxy server. SOCKS is a protocol that allows to pass traffic between two hosts via a proxy server. Client applications need to support this protocol to be able to use SOCKS proxy. Thankfully, all major web-browsers implement such support.

To establish a local SOCKS proxy, you need to use -D <port> option. It is convenient to combine it with -N -f to make ssh go into background after authentication.

As you can see, when we use the proxy the web server thinks we are coming from Germany, which is where the tbdemo-ext is located.

The SOCKS proxy that ssh creates is not necessarily limited to the machine where it runs. It is a default behaviour if no IP address is specified, but it can be easily overriden by prepending port with a colon:

This is a shortcut for specifying a *:8888 to bind and listen to all network interfaces; alternatively, you can explicitly specify the network interface IP address you want the SOCKS proxy to listen on.

As you have seen, dynamic proxying is a handy feature, but requires the client application to support SOCKS protocol. Unfortunatelly, vast majority of tools will not support it. There are some tools that attempt to implement a transparent wrapper around programs and redirect all their network connections through a SOCKS server, for example proxychains. Such wrappers, however, bear not insignificant limitations; they require the programs to be dynamically linked, with the same linker as the tool itself.

ssh provides a functionality that allows to transparently forward traffic of applications that don’t support proxy servers (or if you do not want to pass their traffic via a proxy). First, let’s explore Local port forwarding.

Local port forwarding allows to designate a port, or a set of ports, on the same host where ssh is executed, and bind them to some destination host, passing the traffic via possibly another host to which ssh connection is established.

In the example, the workstation is behind a firewall which does not allow it to connect on port 25 directly. The firewall permits ssh connections to an external server, though. In this case, we can use ssh to set up local port forwarding and reach a specific mailserver on port 25 via external server. To do it, we only need to run the following command:

This command connects to remote-server, starts listening on workstation’s port 8025, and forward all connections to that port via ssh connection to remote-server, to mail-server port 25.

Couple of things to remember:

A very common scenario where Local port forwarding is used is to gain access to a service running on a remote server which is not available via direct connection, because say it is running behind a corporate firewall. In this case, if you can ssh in to a machine that is on the network behind the firewall, then all you have to do is:

Now, by running rdesktop to localhost:3389, you can connect to a windows machine behind the firewall running a remote desktop service.

Similarly, this approach is used to secure access over insecure protocols. For example, VNC protocol is not encrypted. As a best practice, VNC service is configured on, and allows access from, localhost interface only. In order to connect to it remotely, a user need to set up port forwarding with:

And then use localhost:5901 as the destination address for VNC client. The traffic gets routed via ssh connection to vnc-server, where it is forwarded to localhost:5901 where VNC service is listening for connections.

Local port forwarding allows to create so to speak “outbound” connections. They are useful when you want to reach something that is hidden behind a firewall, or runs on remote machine’s localhost inteface.

ssh can also create a reverse configurtion, where you connect to remote host and bind a port on a remote machine to a local host. This can be confusing at first because the forwarding happens in the opposite direction than the ssh connection. Here is an example command:

Now let’s look at a diagram of what it does:

If GatewayPorts is enabled, then you can expose a service running on your workstation (or even network) to the remote server’s network by using the external IP address:

  TODO : Remote + local fw for home-based window laptop

Talking of forwarding, I cannot avoid mentioning ssh ability to forward X11 sessions. This means two things:

To enable X11 forwarding, you just need to specify -X option on the command line. In the example below, I connect from a host that has X server (bastion) to a host that doesn’t (tb-c14). Then I run a program on tb-c14 that requires X-server, and because I used the option -X, the application connects to X-server on bastion and displays the UI there.

I remember one time I needed to install Oracle on a linux server remotely. Back then, Oracle installer for Linux only supported the GUI mode installation. Thanks to ssh I redirected X11 session from the server to my linux laptop and performed the installation.

Again, ssh can do this for you. You can request ssh to create tun devices on machines between which the connection is established, and then use standard linux networking tools to configure a tunnel between machines.

To do this, you need to use -w option that specifies the number on tun device on local and remote machines:

As soon as we established ssh connection, tun0 device appears on both machines. Now, the only thing left is to configure IP addresses on tun0 from both ends:

That’s it! We now have a tunnel encrypted with ssh between two hosts. If we want to let the outside host access to internal network, we can add routing and optionally NAT’ing:

This really seems to be a quick and easy way to setup a VPN. However, you don’t see it used widely, and that is because it has a number of considerable downsides:

What can we do then, if we don’t have the root access on the destination host? Also is there any way to make TCP-over-TCP problem go away? Well, again the answer is yes.

There is an ingenious tool sshuttle that can help with this. sshuttle implements quite a complex mechanics to achieve the end goal, but it is extremly simple to use. All you need to do is:

It requires root access on the originating host, but since it’s usually your laptop or desktop, it is not a problem. Once it establishes connection, it starts intercepting tcp traffic, and reconstructs data for all connections that should go via ssh connection. This allows sshuttle to send raw data via connection, avoiding TCP-over-TCP problem. At the other end, a component reconstructs tcp connections from the datastream.

It is amazing how such a complex tool is so easy to use! As an additional benefit, sshuttle supports forwarding DNS requests via the connection, and you can use internal DNS names instead of figuring out the IP addresses.

The support of secure HTTPS connections via proxy is implemented using CONNECT command, which essentially establishes direct connection between client requesting the connection, and the destination server via proxy server. This is very similar to what we discussed in the Jump hosts section above, but instead of netcat,^[7] or built-in -W functionality. We just will need a tool that knows the HTTP proxy protocol, can connect and request direct connection to the destination server, and then map this connection to its standard input and output.

There is in fact a multitude of tools which do exactly this: corkscrew, httptunnel, proxytunnel, socat. They are very similar in fetures, so I will cover only one that is available in widely-used (and allowed by security policies) EPEL repository — corkscrew.

The command is very simple to use, we just need to specify it in ProxyCommand option to ssh, and we then can overcome a firewall:

This technique has a serious drawback, though. As used in the example above, it requires HTTP proxy to allow CONNECT command to be used for port 22. As the main purpose of CONNECT is to support HTTPS connections, it frequently gets limited to just port 443, or a few more, but it doesn’t usually include the ssh port 22.

The easiest way around it is to start ssh daemon on port 443, if you can do this. You can also use port forwarding on the server, to route traffic from port 443 to 22 (see socat section). Admittedly, this is a bit of a hack, and not a particularly nice. There is another issue with this method; although ssh traffic is obviously encrypted, the initial phase includes protocol negotiation, and it is evident that the traffic that is carried, is not HTTPS, but rather SSH:

When a TCP connection is established between SSH client and server, both must exchange identification strings. This string is in plain text, starts with SSH-, and includes software name, so it is obvious what kind of traffic will flow inside this particular connection.

What can we do to mask SSH traffic going via an HTTP proxy? Well, it would be great if we could establish an encrypted connection similar to how a browser does it, and then continue to use it for ssh traffic.

A perfect tool to do this is stunnel. It establishes a TCP connection, and then negotiates an SSL encyption, which is exactly what a browser does. After the connection is established and encrypted, client stunnel listens for connections on a specified port. Once someone connects to this port, stunnel forwards this connection to the server side, where it is passed to a service defined in the configuration file.

See a diagram:

Here is what is going on in this diagram:

As a result of this configuration, the traffic passing the HTTP proxy is pretty much indistinguishable from that of HTTPS. There are a few other telltale signs that this connection isn’t really a regular connection HTTP connection, like length of the connection, amount of traffic passing, etc; but this is a different issue for a separate discussion.

Let’s look at actual implementation of this setup. Start with stunnel configuration files:

This example client configuration file defines a connection to a server via an HTTP proxy:

Now, let’s see how it all works together:

It is not always possible to dedicate port 443 for trickery with ssh. Sometimes it has to be used by a legitimate http server! Also, it would be nice to show a webpage for any curious soul who decided to poke at our server, instead of confusing them with a handing browser.

As you might have guessed by now, this is again possible. There is a tool, named sslh, which is an ssl/ssh demultiplexer. It accepts connections, analyses incoming transmission, and depending on its type passes down to a corresponding application.

It natually fits into our setup with stunnel: on the server side, the stunnel will forward not to ssh, but to sslh; sslh will probe the traffic and direct it either to nginx for browsers, or to ssh for remote shell.

Here is a diagram:

In Case 1, when a browser connects to port 443 of the server, stunnel takes care of setting up SSL encryption just like a normal webserver would do.^[8] Next, it passes the dencrypted TCP connection down to sslh which detects that this is an HTTP connection, and forwards it to port 80, where nginx is configured to listen for unencrypted HTTP traffic. nginx produces a webpage which is rendered by the browser.

In Case 2, we use stunnel to establish SSL connection, and then ssh via it. Once ssh client attempts to connect to ssh server, stunnel on the server passes this traffic to sslh which after determining it’s an ssh connection, forwards it to port 22. This finalizes the process of establishing an ssh connection.

This setup requires only a slight change, compared with the previous one. First, we need to update server-side stunnel configuration to pass the connection to sslh:

Next, we need to start sslh:

Let’s see how this all works together:

socat is a next-generation version and superset of netcat. SO stands for socket, and in essence, socat relays data between two sources, for example between two network ports, or between a file and a network connection, etc.

In particular, I wanted to mention a few usecases I used it for in production environments.

The last example is interesting, but clearly cumbersome. The upcoming version 2 of socat supports an intriguing functionality of piping sockets. With it, you can combine two socat commands from the example above into a fairly elegant construction:

As the next step, you can simply then use socat as a ProxyCommand in ssh_config:

mosh is an extremely handy replacement for ssh as a remote shell in modern mobile settings. It allows you to move between locations without losing the connection. You can start working at home, suspend or hibernate the laptop, go to office, open the laptop, and your remote shell will seamlessly continue to work.

I have been using mosh for the past few years instead of ssh whenever I can. There are, however, a few gotchas with mosh:

  TODO : A must-have subsystem for ssh. Surprisingly, often it is overlooked.

  TODO : A simplified ssh on demand. Useful for cases when you permit connections from the internet