Day 22 - Building a secure bastion host, or, 50 ways to kill your server

By: Anna Kennedy (@anna_ken_)
Edited By: Gillian Gunson (@shebang_the_cat)

Bastion (noun) 1. A projecting part of a fortification 2. A special purpose computer on a network specifically designed and configured to withstand attacks

If you deploy servers to a private network, then you also need a way to connect to them. The two most common ways methods are to use a VPN, or to ssh through a bastion host (also known as a jump box). Shielding services this way massively reduces your attack surface, but you need to make sure that the server exposed to the internet is as secure as you can make it.

At Telenor Digital we have about 20 federated AWS accounts, and we wanted to avoid having to set up a complex system of VPNs. Additionally, we wanted to be able to connect to any account from anywhere, and not just from designated IP ranges. Deploying a bastion host to each account would allow us to connect easily to instances via ssh forwarding. Our preferred forwarding solution is sshuttle, a "transparent proxy server / poor man's VPN".

This is where it got... interesting. At the time of writing, Amazon AWS did not have a designated bastion host instance type available. Nor in fact do any of the other main cloud providers, nor did there appear to be any other trustworthy bastions available from other sources. There didn’t even seem to be any information about how other people were solving this problem.

I knew that we wanted a secure bastion such that:

1. Only authorised users can ssh into the bastion
2. The bastion is useless for anything BUT ssh'ing through

How hard could making such a bastion possibly be?

Constraints and processes

Our technology stack uses Ubuntu exclusively, and we wanted the bastion to be compatible with the various services we already deploy, such as Consul, Ansible, and Filebeat. Beyond that, I personally have a lot more experience with Ubuntu than I do with any other OS.

For these reasons we decided to base the bastion on a minimal Ubuntu install, strip out as many packages as possible, add some extra security, and make a golden image bastion AMI. Had it not been for these constraints, there might be better OSs to start with, such as Alpine Linux.

Additionally, we run everything in AWS so one or two points of the following are AWS-specific, but based on a lot of conversations it seems that the bastion problem is one that affects a much wider range of architectures.

We use Packer to build our AMIs, Ansible to set them up and Serverspec to test them, so building AMIs is a pretty fast process, typically taking about five minutes. After that we deploy everything using Terraform, so it's a quick turnaround from code commit to running instance.

Starting point: Ubuntu minimal-server

My first port of call: what packages are pre-installed in an Ubuntu minimal-server? Inspection via $apt list --installed or $ dpkg-query -W showed over 2000 packages, and of those I was surprised how many I'd never heard of. And of the ones I had heard of, I was further surprised how many seemed, well, superfluous.

I spent some time and made a few spreadsheets trying to figure out what all the mystery packages were before I got bored and had the bright idea of leveraging Ubuntu's package rating system: all packages are labelled as one of: required, important, standard, optional, or extra.

$ dpkg-query -Wf '${Package;-40}${Priority}\n'
apt                             important
adduser                         required
at                              standard
a11y-profile-manager-indicator  optional
adium-theme-ubuntu              extra

Remove optional and extra packages

Those optional and extra packages sounded very nonessential. I was pretty sure I could rip those out with a nice one-liner and be done.

dpkg-query -Wf '${Package;-40}${Priority}\n' | awk '$2 ~ /optional|extra/ { print $1 }' | xargs -I % sudo apt-get -y purge %

Turns out this was not my best ever idea. All sorts of surprising packages were marked optional or extra and were thus unceremoniously removed, including:

• cloud-init
• grub
• linux-base
• openssh-server
• resolvconf
• ubuntu-server (meta-package)

It doesn't take a genius to realise that removing grub, open-ssh or resolvconf is colossally ill-advised, but even after I tried not removing these but uninstalling the rest I had no luck. On every build I got an unstable and/or unusable image, often not booting at all. Interestingly it broke in a different way each time, possibly something to do with how fast it was uprooting various dependencies before it got to an unbootable state. After quite a lot of experimenting with package-removal lists and getting apparently nondeterministic results, it was time for a new strategy.

Remove a selected list of packages

I revised my plan somewhat in the realisation that maybe blindly removing lots of packages wasn't the best of ideas. Maybe I could look through the package list and remove the ones that seemed the most 'useful' and remove them. Some obvious candidates for removal were the various scripting languages, plus tools like curl and net-tools. I was pretty sure these were just peripherals to a minimal server.

Package name	Ok to remove?	Dependency
curl	no	Consul
ed	yes
ftp	yes
gawk	yes
nano	yes
net-tools	no	sshuttle
perl	no	ssh
python 2.7	no	Ansible
python 3	no	AWS instance checks
rsync	yes
screen	yes
tar	no	Ansible
tmux	yes
vim	yes
wget	yes

It turns out I was incorrect. Due to the various restrictions placed upon the system because we use Consul, sshuttle, Ansible and AWS, about half of my hitlist was unremovable.

To compensate for the limitations in my "remove all the things" strategy, I decided to explore limiting user powers.

Restrict user capabilities

Really, I didn't want users to be able to do anything - they should only be allowed to ssh tunnel or sshuttle through the bastion. Therefore locking down the specific commands a user could issue ought to limit potential damage. To restrict user capabilities, I found four possible methods:

• Change all user shells to /bin/nologin
• Use rbash instead of bash
• Restrict allowed commands in authorized_keys
• Remove sudo from all users

All seemed like good ideas - but on testing I discovered that the first three options only work for pure ssh tunnelling, and don’t work in conjunction with sshuttle.

Remove sudo

I disabled sudo by removing all users from the sudo group, which worked perfectly apart from introducing a new dimension to bastion troubleshooting - without sudo it’s not possible to read the logs nor perform any meaningful investigation on the instance.

We offset most of the pain by having the bastion export its logs to our ELG (Elasticsearch, Logstash, Graylog) logging stack, and export metrics to Prometheus. Between these, most issues were easily identified without needing direct sudo access directly. For the couple of bigger build issues that I had in the later stages of development, I built two versions of the bastion at a time, with and without sudo. A little clunky, but only temporarily.

With the bastion locked down as much as possible, I then added in a few more restrictions to finalise the hardening.

Install fail2ban

An oldie but a goodie, fail2ban is fantastic at restricting logon attempts by locking out anyone who fails to login three times in a row for a determined time period.

Use 2FA and port knocking

Some clever folks in my team ended up making a version of sshuttle that invokes AWS two-factor authentication for our users, and implements a port knocking capability which only opens the ssh port in response to a certain request. The details of this are outside the scope of this article, but we hope to make this open-source in the near future.

Finally! A bastion!

After a lot of experiments and some heavy testing, the bastion was declared production-ready and deployed to all accounts. The final image is t2.nano sized, but we’ve not seen any problems with performance so far as the ssh forwarding is so lightweight.

It's now been in use for a least half a year, and it's been surprisingly stable. We have a Jenkins job that builds and tests the bastion AMI on any change to the source AMI or on code merge, and we redeploy our bastions every few weeks.

I still lie awake in bed sometimes and try to work out how to build a bastion from the ground up, but to all intents and purposes I think the one we have is just fine.

References

• https://www.cyberciti.biz/faq/linux-bastion-host/
• https://askubuntu.com/questions/79665/keep-only-essential-packages
• http://docs.aws.amazon.com/quickstart/latest/linux-bastion/architecture.html
• http://annaken.github.io/testing-packer-builds-with-serverspec

Related content

• https://www.youtube.com/watch?v=Kd4OMVYTwnY
• https://www.slideshare.net/AnnaKennedy11/building-a-secure-bastion-or-50-ways-to-kill-your-server-81551575

Day 17 - Stupid SSH tricks

Written by: Corey Quinn (@kb1jwq)
Edited by: Ben Cotton (@funnelfiasco)

Every year or two, I like to look back over my client's SSH configuration file and assess what I've changed.

This year's emphasis has been on a few options that center around session persistence. I’ve been spending a lot of time on the road this year, using SSH to log into remote servers over terrible hotel wireless networks. As a result, I’ve found myself plagued by SSH session resets. This can be somewhat distracting when I’m in the midst of a task that requires deep concentration— or in the middle of editing a configuration file without the use of screen or tmux.

ServerAliveInterval 60

This triggers a message from the client to the server every sixty seconds requesting a response, in the event that data haven’t been received from the server in that time. This message is sent via SSH’s encrypted channel.

ServerAliveCountMax 10

This sets the number of server alive messages that will be sent. Combined with ServerAliveInterval, this means that the route to the server can vanish for 11 minutes before the client will forcibly disconnect. Note that in many environments, the system’s TCP timeout will be reached before this.

TCPKeepAlive no

Counterintuitively, setting this results in fewer disconnections from your host, as transient TCP problems can self-repair in ways that fly below SSH's radar. You may not want to apply this to scripts that work via SSH, as "parts of the SSH tunnel going non-responsive" may work in ways you neither want nor expect!

ControlMaster auto
ControlPath ~/.ssh/%r@%h:%p
ControlPersist 4h

These three are a bit interesting. ControlMaster auto permits multiple SSH sessions to opportunistically reuse an existing connection, the socket for which lives at ControlPath (in this case, a socket file that lives at ~/.ssh/$REMOTE_LOGIN_USENAME@$HOST:$SSH_PORT). Should that socket not exist, it will be created— and thanks to ControlPersist, it will continue to exist for four hours. Taken as a whole, this has the effect of causing subsequent SSH connections (including scp, rsync (provided you’re using SSH as a transport), and sftp) to be able to skip the SSH session establishment.

As a quick test, my initial connection with these settings takes a bit over 2 seconds to complete. Subsequent connections to that same host complete in 0.3 seconds -- almost an order of magnitude faster. This is particularly useful when using a configuration management that’s establishing repeated SSH connections to the same host, such as ansible, or salt-ssh. It’s worth mentioning that ControlMaster was introduced in OpenSSH 4.0, whereas ControlPersist didn’t arrive until OpenSSH 5.6.

The last trick is a bit off the topic of SSH, as it’s not (strictly speaking) SSH based. Mosh (from “mobile shell”) is a project that uses SSH for its initial authentication, but then falls over to a UDP-based transport. It offers intelligent local echoing over latent links (text that the server hasn’t acknowledged shows up as underlined locally), and persists through connection changes. Effectively, I can start up a mosh session, close my laptop, and go to another location. When I connect to a new wireless network, the session resumes seamlessly. This has the effect of making latent links far more comfortable to work with; I’m typing this post in vim on a server that’s currently 6000 miles and 150ms away from my laptop, for instance.

As an added benefit, mosh prioritizes Ctrl-C; if you’ve ever accidentally catted a 3GB log file, you’ll appreciate this little nicety! Ctrl-C stops the flood virtually instantly.

I will say that mosh is relatively new, and implements a different cryptography scheme than SSH does. As a result, you may not be comfortable running this across the open internet. Personally, I run it over OpenVPN only; while I have no reason to doubt its cryptography implementation, I tend to lean more toward a paranoid stance when it comes to new cryptographic systems.

Ideally this has been enlightening; SSH has a lot of strange options that allow for somewhat nifty behavior in the face of network issues, and mosh is a bit of a game-changer around this space as well.

Day 24 - Terminal Multiplexers

Written by Jon Heise

In the grand world of the command line and in dealing with servers, persistence is key. To that end, there are ways to run stuff that lives beyond your current ssh session, this article will focus on screen and tmux. These tools help you run shell sessions that persist across logins (among other awesome capabilities).

GNU Screen is by far the older of the two being first released in 1987. Tmux was released much more recently in 2007.

Similarities

First off, let's talk similarities. They both have the core functionality of being able to “detach” a session from active usage by a user. Let's say a user wants to run irssi (an irc client) and detach it - the workflow would be as follows:

Note: screen and tmux are generally controlled by multiple keystrokes. In the examples below, C-a means pressing "control + a" keys. A sequence 'C-a d' means press "control + a", release, then press 'd'. This syntax is common in screen and tmux documentation.

goal	screen	tmux
run irssi	screen -S irc irssi	tmux new irssi
detach	C-a d	C-b d
attach again	screen -r irc	tmux attach

FYI: The default screen command key is C-a, while the default tmux command key is C-b. Most things you do in screen and tmux, by default, will always start by pressing the command key.

The above examples create a screen session called 'irc' and an unnamed tmux session. Both tmux and screen also support multiple terminals in one session, that is, you can run multiple commands in separate windows in the same screen or tmux session.

goal	screen	tmux
create new window	C-a c	C-b c
go to previous window	C-a p	C-b p

Using multiple windows in tmux is a bit easier, by default, as there is an omnipresent status bar at the bottom of the terminal showing open windows.

You can get a list of all known sessions:

screen -ls

tmux ls

That's the basics of each - first: creating, attaching, and detaching sessions, and second: creating new windows in a session.

Differences

Having listed features that they both share, it is time to discuss features that one posses that the other does not.

Screen has special features like serial port support and multiuser access. Screen can directly open serial connections from other machines, network gear, or anything that wants to spew data forward on a serial terminal. This feature can be handy for sysadmins with macbooks that need to configure some network gear as there is no good mac specific terminal emulator software. Connection to a serial port is simple:

screen /dev/ttyN

Additionally, you can share your screen sessions with other users on the system with screen's "multiuser" feature. This allow you to share terminals with remote coworkers to do pair debugging or shadowing without having to use a user shared account. See the screen documentation for the following commands: multiuser and acladd.

Tmux's main feature is the client/server model it uses. Earlier it was mentioned that not being able to specify a specific instance of tmux was less of a nuisance, this is due to the fact that tmux runs in a client server model. As long as there is a session of tmux running (even in the background), the tmux server will exist to manage it. Since there is a central server dealing with each tmux client and session, it is far simpler for the client that the user has launched to be aware of these. If you have multiple tmux sessions running (via tmux new), you can ask any tmux session to list them by typing 'C-b s', the result looks something like this:

From the above session list, you can switch to any other open session.

Until recently, tmux (vs screen) was the only one supporting both vertical and horizontal splits. You can create horizontal splits with C-b " and C-b % for vertical splits. In screen, horizontals are made with C-a S and verticals made with C-a | (pipe).

Tricks

You can nest screens within screens, or tmux within tmux. This is most common when running tmux or screen, sshing to another server, and running screen/tmux from there.

goal	screen	tmux
start up	screen -S main	tmux new -s main
ssh somewhere	ssh ...	ssh ...
create a new session on remote host	screen -S foo	tmux new -s foo

The main problem with nesting is knowing how to talk to the nested session. To detach from the 2nd screen session (nested) you'll have to send 'C-a a' which will send a litteral C-a from the first screen to the running program, which is another screen session. Detaching from the nested screen, then, would be 'C-a a d'

This is similar with tmux, though your tmux may not be configured similarly. You may have to add this to your ~/.tmux.conf:

bind-key C-b send prefix

Now pressing 'C-b C-b d' will detach from the nested tmux

The above only applies if you nest screen-in-screen and tmux-in-tmux. If you have screen-in-tmux, you would just press the normal C-a to talk to screen. Same with tmux.

Making them similar again

Screen's defaults don't usually include a status bar, but you can make one similar to tmux by adding this to your ~/.screenrc:

hardstatus alwayslastline "%w"

You can also tune tmux to behave more like screen by changing the command key. In your ~/.tmux.conf:

set-option -g prefix C-a  # make the command key C-a
unbind-key C-b            # unbind the old command key
bind-key a send-prefix    # 'C-a a' sends literal 'C-a'

Why would you do this? If you've used screen for years, and want some of your muscle memory to function in tmux, doing the above is a good start.

Cheat sheet

In closing, here is a recap of commands in convenient cheat sheet form:

action	screen	tmux
new named session	screen -S foobar	tmux new -s foobar
detach session	`C-a d`	`C-b d`
reattach session	screen -dr foobar	tmux a -t foobar
new terminal	`C-a c`	`C-b c`
next terminal	`C-a a` or `C-a n`	`C-b n`
lock terminal	`C-a x`	lock (from command prompt)
large clock	not supported	`C-b t`
not as good smaller clock	`C-a t`	not supported
split screen horizontal	`C-a S`	`C-b “`
split screen vertical	`C-a |`	`C-b %`
change to other portion of a split	`C-a tab`	`C-b arrowkey` (up or down)
send prefix	`C-a a`	`C-b C-b` (if

Happy tmux'ing and screen'ing!

Further reading:

• tmux homepage
• GNU Screen homepage
• OpenBSD FAQ on tmux

Day 2 - Going Parallel

This article was written by Brandon Burton, aka @solarce.

As system administrators, we are often faced with tasks that need to run against a number of things, perhaps files, users, servers, etc. In most cases, we resort of a loop of some sort, often, just a for loop in the shell. The drawback to this seemingly obvious approach, is that we are constrained by the fact that this approach is serial and so the time it will take increases linearly with the number of things we are running the task against.

I am here to tell you there is a better way, it is the path of going parallel!

Tools for your shell scripts

The first place to start is with tools that can replace that for loop you usually use and add some parallelism to the task you are running.

The two most well known tools that are available are:

1. xargs
2. gnu parallel

xargs is a tool used to build and execute command lines from standard input, but one of its great features is that it can execute those command lines in parallel via its -P argument. A quick example of this is:

seq 10 20 | xargs -n 1 -P 5 sleep

This will send a sequence of numbers to xargs and divide it into chunks of one argument (-n 1) at a time and fork off 5 parallel processes (-P 5) to execute each. You can see it in action:

$ ps -eaf | grep sleep
baron     5830  5482  0 11:12 pts/2    00:00:00 xargs -n 1 -P 5 sleep
baron     5831  5830  0 11:12 pts/2    00:00:00 sleep 10
baron     5832  5830  0 11:12 pts/2    00:00:00 sleep 11
baron     5833  5830  0 11:12 pts/2    00:00:00 sleep 12
baron     5834  5830  0 11:12 pts/2    00:00:00 sleep 13
baron     5835  5830  0 11:12 pts/2    00:00:00 sleep 14

Some further reading on xargs is available at:

• http://www.semicomplete.com/blog/articles/week-of-unix-tools/day-5-xargs.html
• http://www.xaprb.com/blog/2009/05/01/an-easy-way-to-run-many-tasks-in-parallel/
• http://www.linuxask.com/questions/run-tasks-in-parallel-with-xargs
• http://stackoverflow.com/questions/3321738/shell-scripting-using-xargs-to-execute-parallel-instances-of-a-shell-function

gnu parallel is a lesser known tool, but has been gaining popularity recently. It is written with the specific focus on executing processes in parallel. From the home page description: "GNU parallel is a shell tool for executing jobs in parallel locally or using remote machines. A job is typically a single command or a small script that has to be run for each of the lines in the input. The typical input is a list of files, a list of hosts, a list of users, a list of URLs, or a list of tables."

A quick example of using parallel is:

% cat offlineimap-cron5min.plist | parallel --max-procs=8 --group 'echo "Thing: {}"'
Thing:       <string>offlineimap-cron5min</string> 
Thing:     <key>Label</key> 
Thing:       <string>solarce</string> 
Thing:     <key>UserName</key> 
Thing:   <dict> 
Thing:     <key>ProgramArguments</key> 
Thing:       <string>admin</string> 
...

This plist file is xml, but the output of parallel is unordered above because each line of input is processed by one of the 8 workers and output occurs (--group) as each worker finishes an input (line) and not necessarily in the order of input.

Some further reading on parallel is available at:

• http://www.gnu.org/software/parallel/man.html
• http://psung.blogspot.com/2010/08/gnu-parallel.html
• http://unethicalblogger.com/posts/2010/11/gnuparallelchangedmy_life

Additionally, there is a great screencast on it.

Tools for multiple machines

The next step in our journey is to progress from just running parallel processes to running our tasks in parallel on multiple machines.

A common approach to this is to use something like the following:

for server in $(cat list_of_servers.txt); do
    ssh $server command argument
done

While this approach is fine for small tasks on a small number of machines, the drawback to it is that it is executed linearly, so the total time the job will take is as long as the task takes to finish multiplied by the number of machines you are executing it on, which means it could take a while, so you'd better get a Snickers.

Fortunately, people have recognized this problem and have developed a number of tools have been developed to solve this, by running your SSH commends in parallel.

These include:

• pssh
• sshpt
• mussh
• clusterssh

I'll illustrate how these work with a few examples.

First, here is a basic example of pssh (on Ubuntu the package is 'pssh,' but the command is 'parallel-ssh'):

# cat hosts-file
p1
p2

# pssh -h hosts-file -l ben date
[1] 21:12:55 [SUCCESS] p2 22
[2] 21:12:55 [SUCCESS] p1 22

# pssh -h hosts-file -l ben -P date
p2: Thu Oct 16 21:14:02 EST 2008
p2: [1] 21:13:00 [SUCCESS] p2 22
p1: Thu Sep 25 15:44:36 EST 2008
p1: [2] 21:13:00 [SUCCESS] p1 22

Second, here is an example of using sshpt:

./sshpt -f ../testhosts.txt "echo foo" "echo bar"
Username: myuser
Password:
"devhost","SUCCESS","2009-02-20 16:20:10.997818","0: echo foo
1: echo bar","0: foo
1: bar"
"prodhost","SUCCESS","2009-02-20 16:20:11.990142","0: echo foo
1: echo bar","0: foo
1: bar"

As you can see, these tools simplify and parallelize your SSH commands, decreasing the execution time that your tasks take and improving your efficiency.

Some further reading on this includes:

• What is a good modern parallel SSH tool?
• Linux - Running The Same Command on Many Machines at Once
• Effective adhoc commands in clusters

Smarter tools for multiple machines

Once you've adopted the mindset your tasks can be done in parallel and you've started using one of the parallel ssh tools for executing ad-hoc commands in a parallel fashion, you may find yourself thinking that you'd like to be able to execute tasks in parallel, but in a more repeatable, extensible, and organized fashion.

If you were thinking this, you are in a luck. There is a class of tools commonly classified as Command and Control or Orchestration tools. These tools include:

• marionette collective aka mcollective
• func
• fabric, aka fab
• capistrano, aka cap

These tools are built to be frameworks within which you can build repeatable systems automation. Mcollective and capistrano are written in Ruby, and Func and Fabric are written in Python. This gives you options for whichever language you prefer. Each has strengths and weaknesses. I'm a big fan of Mcollective in particular, because it has the strength of being built on Puppet and its primary author, R.I. Pienaar has a vision for it to become an extremely versatile tool for the kinds of needs that fall within the realm of Command and Control or Orchestration.

As it's always easiest to grasp a tool by seeing it in action, here are basic examples of using each tool:

mcollective

% mc-package install zsh

 * [ ============================================================> ] 3 / 3

web2.my.net                      version = zsh-4.2.6-3.el5
web3.my.net                      version = zsh-4.2.6-3.el5
web1.my.net                      version = zsh-4.2.6-3.el5

---- package agent summary ----
           Nodes: 3 / 3
        Versions: 3 * 4.2.6-3.el5
    Elapsed Time: 16.33 s

func

% func client15.example.com call hardware info
{'client15.example.com': {'bogomips': '7187.63',
                          'cpuModel': 'Intel(R) Pentium(R) 4 CPU 3.60GHz',
                          'cpuSpeed': '3590',
                          'cpuVendor': 'GenuineIntel',
                          'defaultRunlevel': '3',
...
                          'systemSwap': '8191',
                          'systemVendor': 'Dell Inc.'}}

fabric

% fab -H localhost,linuxbox host_type
[localhost] run: uname -s
[localhost] out: Darwin
[linuxbox] run: uname -s
[linuxbox] out: Linux

Done.
Disconnecting from localhost... done.
Disconnecting from linuxbox... done.

capistrano

# cap invoke COMMAND="yum -y install zsh"
  * executing `invoke'
  * executing "yum -y install zsh"
    servers: ["web1", "web2", "web3"]
    [web2] executing command
    [web1] executing command
    [web3] executing command
    [out :: web3] Nothing to do
    [out :: web2] Nothing to do
    [out :: web1] Complete!
    command finished

As you can see from these brief examples, each of these tools accomplishes similar things, each one has a unique ecosystem, plugins available, and strengths and weaknesses, a description of which, is beyond the scope of this post.

Taking your own script(s) multithreaded

The kernel of this article was an article I recently wrote for my employer's blog, Taking your script multithreaded, in which I detailed how I wrote a Python script to make an rsync job multithreaded and cut the execution time of a task from approximately 6 hours, down to 45 minutes.

I've created a git repo out of the script, so you can take my code and poke at it. If you end up using the script and make improvements, feel free to send me patches!

With the help of David Grieser, there is also a Ruby port of the script up on Github.

These are two good examples of how you can easily implement a multithreaded version of your own scripts to help parallelize your tasks.

Conclusion

There are clearly many steps you can take along the path to going parallel. I've tried to highlight how you can begin with using tools to execute commands in a more parallel fashion, progress to tools which help you execute ad-hoc and then repeatable tasks across many hosts, and finally, given some examples on how to make your own scripts more parallel.