Written by: Tray Torrance (@torrancew)
Edited by: Tom Purl (@tompurl)
TLS Edition
Preface
A previous job gave me the unique experience of building out the infrastructure for a security-oriented startup, beginning completely from scratch. In addition to being generally novel, this gave me the experience to learn a tremendous amount about security best practices, and TLS in particular, during the leaks of documents exposing various mass surveillance programs and cutely-named vulnerabilities such as "Heartbleed". Among our lofty goals was a very strict expectation around what protocols, ciphers and key sizes were acceptable for SSL/TLS connections (for the rest of this article, I will simply refer to this as "TLS"). This story is based on my experience implementing those standards internally.
Disclaimer
TLS is a heavy subject, and it is very easy to feel overwhelmed when approaching it, especially for a newcomer. This is NOT a guide to simplify that problem. For that type of help, see the resources section at the bottom of this article. To keep the tone lighter (and ideally combat the fact that TLS cipher recommendations can change at the drop of a single exploit), and hopefully more approachable for folks with less hands-on experience with these problems, I will use a single cipher, referred to by OpenSSL as ECDHE-RSA-AES256-GCM-SHA384 when demonstrating how a certain library represents its ciphers, along with amusing placeholders in my examples.
This will also (hopefully) help combat the fact that while this post may live for many years, TLS cipher recommendations can change. For the uninitiated, the above cipher string means:
Key Exchange: ECDHE with RSA keys
Encryption Algorithm: 256-bit AES in GCM mode
Signature: SHA384
These are the three components you need for a TLS-friendly cipher.
The Goal
With regards to TLS, our main objectives, both for ourselves and our customers, were:
Secure all internal and external communications with TLS
Enforce a mandatory key size
Enforce a consistent, strong set of ciphers across all TLS connections
For cipher selection, we effectively chose a subset of the
Mozilla-recommended "modern" ciphers for reasons that are out of scope for this post.
The Solution
With a modern configuration management solution (we used Puppet), this seems like a pretty trivial problem to solve. Indeed, we quickly threw together an NGINX template that enforced our preferences, and life was good.
...
ssl_ciphers ACIPHER_GOES_HERE:ANOTHER_GOES_HERE;
...
The Problem: EINCONSISTENT
So, that's it, right? Well, no. Anyone who's had to deal with these types of
things at this level can tell you that sooner or later, something will come
along that doesn't interoperate. In our case, eventually we added some software
that more or less required the use of Apache instead of NGINX, and now we had
a new config file in which to manage TLS ciphers.
The (Modified) Solution
Frustrated, we did what any good DevOps engineer does, and we abstracted the
problem away. We drew a puppet variable in at "top scope", which let us
access it from any other part of our codebase, and then referenced it from both
our Apache and NGINX templates:
# manifests/site.pp
$ssl_ciphers = 'YOU_GET_ACIPHER:EVERYBODY_GETS_ACIPHER'
# modules/nginx/templates/vhost.conf.erb
...
ssl_ciphers <%= @ssl_ciphers %>
...
# modules/apache/templates/vhost.conf.erb
...
SSLCiphers <%= @ssl_ciphers %>
...
The (Second) Problem: EUNSUPPORTED
As these things tend to go, after a few weeks (or maybe months, if we were
lucky - I don't recall) of smooth sailing, another compatibility issue was
introduced into our lives. This time, it was JRuby. For reasons that will be elaborated upon below, JRuby cannot use the OpenSSL library to provide its TLS support in the way that "normal" Ruby does. Instead, JRuby maintains a jopenssl library, whose purpose is to provide API-compatibility with Ruby's OpenSSL wrapper. Unfortunately, the library JRuby does use has a different notation for expressing TLS ciphers than OpenSSL, so jopenssl maintains a static mapping. Some of you may be groaning right about now, but wait - there's more!
In addition to not supporting some of the more modern ciphers we wanted to use (though it happily ignored them when specified, which was in this case helpful), feeding it malformed versions of the "magic" (aka ALL, LOW, ECDHE+RSA, etc) names supported by OpenSSL seemed to cause it to support any cipher that it understood - several of which are no longer secure enough for serious use.
This is why we can't have nice things.
The (Second) Solution
We had some pretty intelligent and talented folks attempt to patch this, but
they were unsuccessful at unwinding the rather complicated build process by
which JRuby tests and releases jopenssl. Ultimately, we decided that since
the JRuby application was internal only, that we could extend our policy for
internal services to include the strongest two ciphers JRuby supported at the
time. This meant adding another top scope puppet variable for use there:
# manifests/site.pp
...
$legacy_ssl_ciphers = "${ssl_ciphers}:JRUBY_WEAK_SAUCE"
...
And then, once more, referencing it in the proper template.
The (Third) Problem: ENOMENCLATURE
After another brief reprieve, along came a "proper" Java application. You may recall that I mentioned JRuby cannot use OpenSSL - well, this is because the JVM, being a cross-platform runtime, provides its own implementation of the TLS stack via a set of libraries referred to as JSSE (Java Secure Socket Extension). Now, for a brief digression.
OpenSSL cipher names are, you see, only barely based in reality. TLS cipher names are defined in RFCs, and our scapegoat cipher's official name, for example, is TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384. In my experience, OpenSSL almost always (annoyingly) ignores these names in favor of one they themselves make up. JSSE on the other hand, (correctly, though I rarely use that word in the context of Java) uses the RFC names.
Still with me? Great. As you may have put together, adding a Java app meant that our existing variable was not going to be able to do what we needed on its own. Attempts to cobble together a programmatic mapping via files describing RFC names and tricks with OpenSSL syntax were fairly successful, but unidirectional, relatively brittle and prone to needing manual updates of the RFC name list in the future.
The (Third) Solution
As you may have guessed, it's certainly simple enough to do the following, using Java-compatible cipher names:
# manifests/site.pp
...
$java_ssl_ciphers = 'IDK:WHO:I:EVEN:AM:ANYMORE'
...
And then use that variable as needed in the various Java templates, which are almost always XML.
The (Fourth) Problem: EERLANG
As I'm sure you've guessed by now, after a bit of smooth sailing, something
else came along. This time, it was RabbitMQ, which is written in Erlang.
Rabbit (and possibly other Erlang tools) support SSL cipher configuration via
an array of 3-element tuples. In RabbitMQ, our example cipher would be
expressed as: {ecdhe_rsa,aes_256_gcm,sha384}. Now, let me first say that, academically, this is very clever. Practically, though, I want to start a pig farm.
The (Fourth) Solution
At this point, a hash, or really any data structure, was starting to look more appealing than a never-ending string of arbitrarily named global variables, so a refactor takes us to:
# manifests/site.pp
$ssl_ciphers = {
'openssl' => 'MY:HANDS:ARE:TYPING:WORDS',
'jruby' => 'OH:MY:DEAR:WORD:WHY?!?',
'java' => 'PLEASE:MAKE:IT:STOP',
'erlang' => "[{screw,this,I},{am,going,home}]"
}
The Conclusion
Well, by now, you have possibly realized that everything is terrible and I need a drink (and perhaps a bit of therapy). While that is almost certainly true, it's worth noting a few things:
TLS, while important, can be very tedious/error-prone to deploy - this is why best practices are important!
A good config management solution is worth a thousand rants - our use of config management reduced this problem to figuring out the proper syntax for each new technology rather than reinventing the entire approach each time. This is more valuable than I can possibly express, and can be achieved with basically any config management framework available.
While we can't currently have nice things (and TLS is not the only reason), tools like Puppet (and its many alternatives, such as Chef, Ansible or Salt) can let us start thinking about a world where we can. With enough traction, maybe we'll get there one day. For now, I'll be off to find that drink.
References
Mozilla Server-Side TLS Guide - Use this if you need help configuring TLS for your services
Qualys SSL Labs - Use this to test your TLS configuration
Acknowledgements
I'd like to thank the following people for their help with this, regardless of whether they realized or volunteered to do so.
- My wonderful partner Mandy, for her endless support (and proofing early drafts of this)
- My SysAdvent editor, Tom, for being flexible and thorough
- My former colleague, William, who inadvertantly mentored much of my TLS education
Posts
