Using a ProxyCommand to Leap Frog Your Bastions

5 minute read Published: 2012-10-15

I do most of my work over SSH. Even when I'm working in my browser or pgAdminIII, I'm usually doing that over SSH tunnels. VPN Software has been around for quite some time and it's still mostly disappointing and usually run by the least competent group in any IT department. I developed a workflow using SSH from my laptop, either on the corporate network or at home, I can ssh /directly/ to the server I'm interested in working on.

In order to accomplish this, I have made some compromises. First off, if I'm SSH-ing from my home, I am /required/ to type the fully qualified domain names (FQDN) when workign remotely. I use the presence of the domain name to activate the proper leap frogging. I also decided to use ControlMaster's with SSH that can leave me with a terminal without a prompt when I forget which shell is my master. Overall, the pros outweigh the cons and I'm more productive because of it.

ControlMaster

Using a ControlMaster with ssh allows multiple connections to the same tcp connection. This means subsequent connections are much faster to open, but places a limit on the original connection that all connections riding on it must be closed before the ControlMaster connection closes. This may or may not be desirable, but does come in handy when using ProxyCommand to bounce around through jump hosts as the connection establishment overhead is removed.

Adding this line to your ~/.ssh/config will enable ControlMaster for all connections:

# Use Control Master so we type passwords less
ControlMaster auto
ControlPath ~/.ssh/ssh_control_%h_%p_%r

Estasblishing a Jump Host

I find it's best to alias the jump hosts to host names that don't exist in DNS. Ideally, I'll never log in to these hosts directly, so I can even forget these names. Let's create an alias for our bastion host, lets call it 'bastion' and it run ssh on 65022.

# Bastion Host
Host bastion
  Hostname corporate-bastion.example.com
  Port 65022

Using Your Aliases

It's really simple:

# Use the bastion to connect to internal resources
Host *.internal.example.com
    ProxyCommand ssh bastion nc %h %p

If you're configuring your networks in a way that make sense, this configuration will work from home or work. Usually, the internal.example.com zones live inside your DNS search path while you're on the corporate network. You probably also have SSH access directly to your servers from the corporate network, so while at work you:

$ ssh webserver-001

And then once you go home you can access the same server, directly by:

$ ssh webserver-001.internal.example.com

Sure, it's more typing, but it gets you exactly where you want quickly. Chances are you are using zsh or bash with autocompletion and you can just hit /tab/ when you get to the first '.' to have the autocomplete work it's magic.

It's Complicated ..

Sure it is. It's always more complicated. And we can achieve the same things with your insanely complicated series of jump hosts. Chances are, you've got some "high security" shit going on with your network, and you need to use jump hosts internally. Maybe your external bastion machine only provides SSH access to the other internal bastions on your network. Well, that's plenty O.K.

Multiple Jump Hosts

Again, I like to pick aliases not in DNS to avoid confusion. DNS should be the authoritative place for things on your network, so don't collide with it. I can't help you if you insist on being stupid. Let's say we have a setup where we need to connect to an external bastion, then to our internal bastion host from the outside. This gives us multiple layers of security, but can drive a man insane with all the non-sense required to scp, rsync, or tunnel to those hosts behind two bastions.

When you have multiple jump hosts in play, ControlMaster comes in handy. It dramatically reduces connection time and complexity. Should one of the bastion hosts require a two-factor authentication scheme, ControlMaster will make you life incredibly easy. Here's an example of how I might set this up:

# Use Control Master so we type passwords less
ControlMaster auto
ControlPath ~/.ssh/ssh_control_%h_%p_%r

# External Bastion Host
Host extbastion
  Hostname external-bastion.example.com
  Port 65022

# Internal Bastion Host
Host intbastion
    ProxyCommand ssh extbastion nc internal-bastion.example.com 22

So everything looks the same as we saw earlier. So to utilize the two jump hosts together, we can just chain to the internal bastion host!

# Use the internal bastion from the external bastion to connect to internal resources
Host *.internal.example.com
    ProxyCommand ssh intbastion nc %h %p

And there you go, keep chaining on the jump hosts and it will keep working. Again, I don't like to overlap my Host aliases with DNS, so make good decisions while naming your aliases.

Bonus Round!

As an FYI, you can use environment variables in your ProxyCommands! So, maybe you do this in your ~/.bashrc:

export SSH_PROXY='int'
alias extssh="SSH_PROXY=ext ssh"

And add these entries to your ~/.ssh/config:

# External Bastion Host
Host jumper
  Hostname external-bastion.example.com
  Port 65022

# Internal Bastion Host
Host bastion-int
    Hostname internal-bastion.example.com

# External Access to Internal Bastion Host
Host bastion-ext
    ProxyCommand ssh jumper nc internal-bastion.example.com 22

# Proxy based on environment variable SSH_PROXY
Host *.internal.example.com
    ProxyCommand ssh bastion-$SSH_PROXY nc %h %p

Now you can retrain choose how to use your ssh jump hosts using shell environment variables:

$ ssh webserver-001.internal.example.com
# Desktop -> internal-bastion.example.com -> webserver-001.internal.ample.com

$ extssh webserver-001.internal.example.com
# Desktop -> external-bastion.example.com -> internal-bastion.example.com -> webserver-001.internal.example.com

Enjoy!

find me: