In this article I describe how I setup, configure and use my Linux Workstation and SSH Bastion/Jumphosts for maximimum awesomness and ease of use.
For illustration purpose we asume to have three very simple and mostly identical environments named test, stage and live each having a Jumphost and a bunch of machines only accessible via their corosponding Jumphost.
Where each environment contains the following hosts:
[prx] moep-prx01 ansible_host=10.10.10.11 moep-prx02 ansible_host=10.10.10.12 [web] moep-web01 anisble_host=10.10.10.101 moep-web02 anisble_host=10.10.10.102 moep-web03 anisble_host=10.10.10.103 moep-web04 anisble_host=10.10.10.104 moep-web05 anisble_host=10.10.10.105 moep-web06 anisble_host=10.10.10.106 [sto] moep-sto01 anisble_host=10.10.10.21 moep-sto02 anisble_host=10.10.10.22 [sql] moep-sql01 anisble_host=10.10.10.31 moep-sql02 anisble_host=10.10.10.32 moep-sql03 anisble_host=10.10.10.33
Use sshuttle instead of ssh port fowarding
Use sshuttle! It is awesome!
Lets imagine you want to access the MySQL Service running in Test on the
moep-sql03 machines. For this you normally would have to setup SSH Port forwarding, for the port 3306 for each target machine, or would have to ssh into the Jumphost and run commands on there. Both options are tedious, especially running stuff directly on the jumphost, as this undermines the function and purpose of the Jumphost and turns it into a remote workstation.
Instead of Port forwarding, or running comands on the Jumphost, use sshuttle.
Sshuttle is a "poor mans VPN" that allows you to essentially route traffic for a given Network transparently through a Jumphost without the need to forward every single Port.
To access all Test machines, and their services simply open up a sshuttle like this.
sshuttle -r firstname.lastname@example.org 10.10.10.0/24
This will allow you to acces any machine inside the Network
10.10.10.0/24 that is reachable via the Jumphost.
Now you can use all the Tools installed on your Workstation to access Machines in the
10.10.10.0/24, i.E. youse your Browser, Api Tools to check the Web Services, use DBeaver to access the SQL Databases, and so on.
SSH Bastion/Jumphost configuration
If we where to run a Playbook or single task on all Machines contained in the inventory file we can encounter a problem where Some SSH Connections fail, this is because of the default configuration of the sshd daemon regarding parallel sessions.
To work around this issue we must modify the sshd configuration on our Jumphost to allow more parallel SSH Sessions/Connections.
For this purpose we add the following configuration to the
/etc/ssh/sshd.config on our Jumphosts.
MaxSessions 50 MaxStartups 50:30:80
MaxSessions option specifies the maximum number of open shell, login or subsystem (e.g. sftp) sessions permitted per network connection. And is increased to allow for better Multiplexing.
MaxStartups option specifies the maximum number of concurrent unauthenticated connections to the SSH daemon. Leaving this on its default vaue is the main cause of connection problems with big inventories.
ssh bastion host ssh configuration.
To connect via SSH to a target System in i.E. the Stage environment, you can, of course, open up a sshuttle and tunel all trafic for the target network, but what if you wanted to connect to a live System at the same time, now you got the Problem where live and Stage share the same Network (
10.10.10.0/24) and you cannot reach the Systems in either live or Test while your Stage sshuttle is established.
To work around this (atleast for SSH connections) you can use the Jumphosts as bastion hosts for their corosponding environments.
The simplest way to do this is by running this command:
ssh -o ProxyCommand='ssh -W %h:%p your_user@bastion' your_user@target
Because this is not simple at all we want to specify these setting in our openssh client configuration and add easy to remember aliases for the environments as well.
We will add the following to our
# Bastion Hosts Host test-moep-* StrictHostKeyChecking no User your_user ProxyCommand ssh -W %h:%p -q email@example.com Host stage-moep-* StrictHostKeyChecking no User your_user ProxyCommand ssh -W %h:%p -q firstname.lastname@example.org Host prod-moep-* StrictHostKeyChecking no User your_user ProxyCommand ssh -W %h:%p -q email@example.com # moep Hosts Host *-moep-prx01 HostName 10.10.10.11 Host *-moep-prx02 HostName 10.10.10.12 Host *-moep-web01 HostName 10.10.10.101 Host *-moep-web02 HostName 10.10.10.102 Host *-moep-web03 HostName 10.10.10.103 Host *-moep-web04 HostName 10.10.10.104 Host *-moep-web05 HostName 10.10.10.105 Host *-moep-web06 HostName 10.10.10.106 Host *-moep-sto01 HostName 10.10.10.21 Host *-moep-sto02 HostName 10.10.10.22 Host *-moep-sql01 HostName 10.10.10.31 Host *-moep-sql02 HostName 10.10.10.32 Host *-moep-sql03 HostName 10.10.10.33
This configuration allows us to ssh into any node of any envirionment, at the same time, without the limitations that come with opening up a sshuttle. If you i.E. want to ssh into a Testing Machine, use the corosponding prefix
test- in your SSH command like this:
add a bastion host configuration to your ansible inventory
To be able to run ansible against all the environments similtaniously we want to setup a Bastion Host configuration for each environment, this can be done by setting the Variable
For each environment we need to set this variable acordingly like this:
ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q firstname.lastname@example.org"'
ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q email@example.com"'
ansible_ssh_common_args: '-o ProxyCommand="ssh -W %h:%p -q firstname.lastname@example.org"'
This can i.E. be done with variables inside the inventorie file itself or, anywhere else, where variables can be set.
How and where you set the
ansible_ssh_common_args depends mainly on your teams style.