In this article I describe how I setup, configure and use my Linux Workstation and SSH Bastion/Jumphosts for maximimum awesomness and ease of use.
For illustration purpose we asume to have three very simple and mostly identical environments named test, stage and live each having a Jumphost and a bunch of machines only accessible via their corosponding Jumphost.
Where each environment contains the following hosts:
[prx]
moep-prx01 ansible_host=10.10.10.11
moep-prx02 ansible_host=10.10.10.12
[web]
moep-web01 anisble_host=10.10.10.101
moep-web02 anisble_host=10.10.10.102
moep-web03 anisble_host=10.10.10.103
moep-web04 anisble_host=10.10.10.104
moep-web05 anisble_host=10.10.10.105
moep-web06 anisble_host=10.10.10.106
[sto]
moep-sto01 anisble_host=10.10.10.21
moep-sto02 anisble_host=10.10.10.22
[sql]
moep-sql01 anisble_host=10.10.10.31
moep-sql02 anisble_host=10.10.10.32
moep-sql03 anisble_host=10.10.10.33
Use sshuttle instead of ssh port fowarding
Use sshuttle! It is awesome!
Lets imagine you want to access the MySQL Service running in Test on the moep-sql01, moep-sql02 and moep-sql03 machines. For this you normally would have to setup SSH Port forwarding, for the port 3306 for each target machine, or would have to ssh into the Jumphost and run commands on there. Both options are tedious, especially running stuff directly on the jumphost, as this undermines the function and purpose of the Jumphost and turns it into a remote workstation.
Instead of Port forwarding, or running comands on the Jumphost, use sshuttle.
Sshuttle is a "poor mans VPN" that allows you to essentially route traffic for a given Network transparently through a Jumphost without the need to forward every single Port.
To access all Test machines, and their services simply open up a sshuttle like this.
sshuttle -r your_user@192.168.1.20 10.10.10.0/24
This will allow you to acces any machine inside the Network 10.10.10.0/24 that is reachable via the Jumphost.
Now you can use all the Tools installed on your Workstation to access Machines in the 10.10.10.0/24, i.E. youse your Browser, Api Tools to check the Web Services, use DBeaver to access the SQL Databases, and so on.
SSH Bastion/Jumphost configuration
If we where to run a Playbook or single task on all Machines contained in the inventory file we can encounter a problem where Some SSH Connections fail, this is because of the default configuration of the sshd daemon regarding parallel sessions.
To work around this issue we must modify the sshd configuration on our Jumphost to allow more parallel SSH Sessions/Connections.
For this purpose we add the following configuration to the /etc/ssh/sshd.config on our Jumphosts.
MaxSessions 50
MaxStartups 50:30:80
The MaxSessions option specifies the maximum number of open shell, login or subsystem (e.g. sftp) sessions permitted per network connection. And is increased to allow for better Multiplexing.
The MaxStartups option specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Leaving this on its default vaue is the main cause of connection problems with big inventories.
ssh bastion host ssh configuration.
To connect via SSH to a target System in i.E. the Stage environment, you can, of course, open up a sshuttle and tunel all trafic for the target network, but what if you wanted to connect to a live System at the same time, now you got the Problem where live and Stage share the same Network (10.10.10.0/24) and you cannot reach the Systems in either live or Test while your Stage sshuttle is established.
To work around this (atleast for SSH connections) you can use the Jumphosts as bastion hosts for their corosponding environments.
The simplest way to do this is by running this command:
ssh -o ProxyCommand='ssh -W %h:%p your_user@bastion' your_user@target
Because this is not simple at all we want to specify these setting in our openssh client configuration and add easy to remember aliases for the environments as well.
We will add the following to our ~/.ssh/config file
# Bastion Hosts
Host test-moep-*
StrictHostKeyChecking no
User your_user
ProxyCommand ssh -W %h:%p -q your_user@192.168.1.20
Host stage-moep-*
StrictHostKeyChecking no
User your_user
ProxyCommand ssh -W %h:%p -q your_user@192.168.1.30
Host prod-moep-*
StrictHostKeyChecking no
User your_user
ProxyCommand ssh -W %h:%p -q your_user@192.168.1.40
# moep Hosts
Host *-moep-prx01
HostName 10.10.10.11
Host *-moep-prx02
HostName 10.10.10.12
Host *-moep-web01
HostName 10.10.10.101
Host *-moep-web02
HostName 10.10.10.102
Host *-moep-web03
HostName 10.10.10.103
Host *-moep-web04
HostName 10.10.10.104
Host *-moep-web05
HostName 10.10.10.105
Host *-moep-web06
HostName 10.10.10.106
Host *-moep-sto01
HostName 10.10.10.21
Host *-moep-sto02
HostName 10.10.10.22
Host *-moep-sql01
HostName 10.10.10.31
Host *-moep-sql02
HostName 10.10.10.32
Host *-moep-sql03
HostName 10.10.10.33
This configuration allows us to ssh into any node of any envirionment, at the same time, without the limitations that come with opening up a sshuttle. If you i.E. want to ssh into a Testing Machine, use the corosponding prefix test- in your SSH command like this:
ssh test-moep-web03
add a bastion host configuration to your ansible inventory
To be able to run ansible against all the environments similtaniously we want to setup a Bastion Host configuration for each environment, this can be done by setting the Variable ansible_ssh_common_args.
For each environment we need to set this variable acordingly like this:
Testing
ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q your_user@192.168.1.20"'
Staging
ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q your_user@192.168.1.30"'
Live
ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q your_user@192.168.1.40"'
This can i.E. be done with variables inside the inventorie file itself or, anywhere else, where variables can be set.
How and where you set the ansible_ssh_common_args depends mainly on your teams style.