Sunday, April 24, 2022

Dynamic SSH jump hosts


Introduction

As a road warrior we all have likely created a jump host configuration which allows us to automatically connect to a machine behind a jump host.  In the past I used a convention like the following to connect to a host on my home network while I was away:

ssh hostA.mynet.red-tux

This would ssh to hostA on my home network via the externally accessable jump box.  However this begins to have trouble if you have a git server on your internal network.  So I did some digging and found some people talking about the ability to create a dynamic jump configuration, but not too much detail.  This is what I came up with.  

SSH Configuration file

First we need to edit our local client side ssh configuration file.  Below is an example of what I use.  This configuration will work for all hosts under the "mynet.red-tux.net" domain, and that entry is the entry which does the magic as it calls a script which determines if ssh should go directly to the host or via a jump host.

~/home/.ssh/config

compression yes
tcpkeepalive yes
serveraliveinterval 15
serveralivecountmax 6
ForwardAgent yes

host bungee.red-tux.net
 ForwardAgent yes
 ControlPath ~/.ssh/control-%r@%h:%p
 ControlMaster auto
 ControlPersist 1

host *.lab.mynet.red-tux.net
 StrictHostKeyChecking no
 UserKnownHostsFile /dev/null

host *.mynet.red-tux.net
 ProxyCommand ~/bin/proxy_ssh.sh %h %p   

Proxy Script 

In my case this script sits in my home directory inside bin with a name of "proxy_ssh.sh".  This script receives two parameters from ssh, the host you are attempting to connect to and the port you are attempting to connect to on that host.  First the whole file, then I'll break down the what it does.  Essentially this script looks for the internal IP address of the jump host because I have an internal DNS server I manage for my hosts (running on Red Hat IdM/FreeIPA).  If it determines the IP address is the internal one it connects directly and sets up a netcat proxy for the ProxyCommand to work correctly, otherwise it will connect to the jump host and then setup the needed netcat proxy.

~/bin/proxy_ssh.sh

#!/bin/bash

#Time to cache lookup in seconds
CACHE_LIMIT=900

#Cache file to use
CACHE_FILE=/home/nelsonab/.ssh/network_lookup

#Jump host
JUMP_HOST=jump.example.com
JUMP_IP=192.168.55.5
JUMP_LOOKUP="$(host $JUMP_HOST | awk '{print $4}')"


if [[ "$1" =~ .*red-tux\.net$ ]]; then
# red-tux.net found in hostname  
 CURRENT_TIME=$(date +%s)

 CACHE_UPDATE_NEEDED=/bin/false

 if [[ -f $CACHE_FILE ]]; then
   echo "Cache found"
   # # Read Cache
   # . $CACHE_FILE
   #Get age of cache
   CACHE_AGE=$(stat -c %Y $CACHE_FILE)

   if (( $(( $CURRENT_TIME - $CACHE_AGE )) > $CACHE_LIMIT )); then
   #|| "$JUMP_LOOKUP" != "$SSH_LOCATION_IP" ]]; then
     #The age of the cache is greater than the limit or , update needed
     CACHE_UPDATE_NEEDED=/bin/true
   fi
 else
   #Cache could not be found, update needed
   CACHE_UPDATE_NEEDED=/bin/true
 fi

 if $CACHE_UPDATE_NEEDED; then
   JUMP_HOST_IP=$(host $JUMP_HOST | awk '{print $4}')
   if [[ "$JUMP_HOST_IP" == "$JUMP_IP" ]]; then
     echo "SSH_LOCATION=internal" > $CACHE_FILE
   else
     echo "SSH_LOCATION=external" > $CACHE_FILE
   fi
   echo "SSH_LOCATION_IP='$JUMP_LOOKUP'" >> $CACHE_FILE
   logger "PROXY_SSH Cache file updated"
 fi
 . $CACHE_FILE

 if [[ "$SSH_LOCATION" == "internal" ]]; then
   exec nc $1 $2
 else
   exec ssh bungee.red-tux.net nc $1 $2
 fi
else
# red-tux.net not found in hostname
 exec nc $1 $2
fi

Breakdown of proxy_ssh script

First up we set a variable for how long a lookup cache is good for.  This script will perform a DNS lookup, hence the desire to cache the lookup.  The chances of me roaming between inside and outside my network within the timeout period is overall low.

#Time to cache lookup in seconds

CACHE_LIMIT=900

#Cache file to use
CACHE_FILE=/home/nelsonab/.ssh/network_lookup

Next we setup some information about the Jump host, and how to look it up.  It is likely you can change the lookup done for the "JUMP_LOOKUP" variable and not need to change the rest of the script, but I have not tested this. 

#Jump host
JUMP_HOST=jump.example.com
JUMP_IP=192.168.55.5
JUMP_LOOKUP="$(host $JUMP_HOST | awk '{print $4}')"

Then we do a regex on the host name passed in, if it matches our dynamic domain, then we proceed.  This is more of a fallback to ensure we're only jumping for hosts in the given domain, it is likely this isn't needed as the host configuration performs this selection. 

if [[ "$1" =~ .*red-tux\.net$ ]]; then
# red-tux.net found in hostname  
 CURRENT_TIME=$(date +%s)

We set a variable assuming that the cache does not need to be updated, and then perform actions based on this variable later, after numerous checks have been performed.  It is likely this could be greatly simplified, but I was wanting to have a fallback to invalidate the cache if the IP address of of the host changed during the timeout period, but never got this finished.

 CACHE_UPDATE_NEEDED=/bin/false

 if [[ -f $CACHE_FILE ]]; then
   echo "Cache found"
   # # Read Cache
   # . $CACHE_FILE
   #Get age of cache
   CACHE_AGE=$(stat -c %Y $CACHE_FILE)

   if (( $(( $CURRENT_TIME - $CACHE_AGE )) > $CACHE_LIMIT )); then
   #|| "$JUMP_LOOKUP" != "$SSH_LOCATION_IP" ]]; then
     #The age of the cache is greater than the limit or , update needed
     CACHE_UPDATE_NEEDED=/bin/true
   fi
 else
   #Cache could not be found, update needed
   CACHE_UPDATE_NEEDED=/bin/true
 fi

Next we perform the lookup and set the information in the cache file.  I haven't fully included the JUMP_LOOKUP logic, I guess that's a #TODO. 

 if $CACHE_UPDATE_NEEDED; then
   JUMP_HOST_IP=$(host $JUMP_HOST | awk '{print $4}')
   if [[ "$JUMP_HOST_IP" == "$JUMP_IP" ]]; then
     echo "SSH_LOCATION=internal" > $CACHE_FILE
   else
     echo "SSH_LOCATION=external" > $CACHE_FILE
   fi
   echo "SSH_LOCATION_IP='$JUMP_LOOKUP'" >> $CACHE_FILE
   logger "PROXY_SSH Cache file updated"
 fi

Next we source the cache file since it sets the variable "SSH_LOCATION".  If the cache did not need an update all the steps to this point execute quickly. 

 . $CACHE_FILE

If we're internal then we go directly to the host 

 if [[ "$SSH_LOCATION" == "internal" ]]; then
   exec nc $1 $2

For external we first go to the jump host 

 else
   exec ssh bungee.red-tux.net nc $1 $2
 fi

Alternatively we go directly to the host as a fallback. 

else
# red-tux.net not found in hostname
 exec nc $1 $2
fi

Conclusion

As you can see overall the process is straight forward.  Hopefully this helps you create a dynamic ssh jump configuration of your own.