December 23, 2015

Day 23 - This Is Why We Can't Have Nice Things

Written by: Tray Torrance (@torrancew)
Edited by: Tom Purl (@tompurl)

TLS Edition


A previous job gave me the unique experience of building out the infrastructure for a security-oriented startup, beginning completely from scratch. In addition to being generally novel, this gave me the experience to learn a tremendous amount about security best practices, and TLS in particular, during the leaks of documents exposing various mass surveillance programs and cutely-named vulnerabilities such as "Heartbleed". Among our lofty goals was a very strict expectation around what protocols, ciphers and key sizes were acceptable for SSL/TLS connections (for the rest of this article, I will simply refer to this as "TLS"). This story is based on my experience implementing those standards internally.


TLS is a heavy subject, and it is very easy to feel overwhelmed when approaching it, especially for a newcomer. This is NOT a guide to simplify that problem. For that type of help, see the resources section at the bottom of this article. To keep the tone lighter (and ideally combat the fact that TLS cipher recommendations can change at the drop of a single exploit), and hopefully more approachable for folks with less hands-on experience with these problems, I will use a single cipher, referred to by OpenSSL as ECDHE-RSA-AES256-GCM-SHA384 when demonstrating how a certain library represents its ciphers, along with amusing placeholders in my examples.

This will also (hopefully) help combat the fact that while this post may live for many years, TLS cipher recommendations can change. For the uninitiated, the above cipher string means:

  • Key Exchange: ECDHE with RSA keys

  • Encryption Algorithm: 256-bit AES in GCM mode

  • Signature: SHA384

These are the three components you need for a TLS-friendly cipher.

The Goal

With regards to TLS, our main objectives, both for ourselves and our customers, were:

  • Secure all internal and external communications with TLS

  • Enforce a mandatory key size

  • Enforce a consistent, strong set of ciphers across all TLS connections

For cipher selection, we effectively chose a subset of the

Mozilla-recommended "modern" ciphers for reasons that are out of scope for this post.

The Solution

With a modern configuration management solution (we used Puppet), this seems like a pretty trivial problem to solve. Indeed, we quickly threw together an NGINX template that enforced our preferences, and life was good.





So, that's it, right? Well, no. Anyone who's had to deal with these types of

things at this level can tell you that sooner or later, something will come

along that doesn't interoperate. In our case, eventually we added some software

that more or less required the use of Apache instead of NGINX, and now we had

a new config file in which to manage TLS ciphers.

The (Modified) Solution

Frustrated, we did what any good DevOps engineer does, and we abstracted the

problem away. We drew a puppet variable in at "top scope", which let us

access it from any other part of our codebase, and then referenced it from both

our Apache and NGINX templates:

# manifests/site.pp


# modules/nginx/templates/vhost.conf.erb


ssl_ciphers <%= @ssl_ciphers %>


# modules/apache/templates/vhost.conf.erb


SSLCiphers <%= @ssl_ciphers %>


The (Second) Problem: EUNSUPPORTED

As these things tend to go, after a few weeks (or maybe months, if we were

lucky - I don't recall) of smooth sailing, another compatibility issue was

introduced into our lives. This time, it was JRuby. For reasons that will be elaborated upon below, JRuby cannot use the OpenSSL library to provide its TLS support in the way that "normal" Ruby does. Instead, JRuby maintains a jopenssl library, whose purpose is to provide API-compatibility with Ruby's OpenSSL wrapper. Unfortunately, the library JRuby does use has a different notation for expressing TLS ciphers than OpenSSL, so jopenssl maintains a static mapping. Some of you may be groaning right about now, but wait - there's more!

In addition to not supporting some of the more modern ciphers we wanted to use (though it happily ignored them when specified, which was in this case helpful), feeding it malformed versions of the "magic" (aka ALL, LOW, ECDHE+RSA, etc) names supported by OpenSSL seemed to cause it to support any cipher that it understood - several of which are no longer secure enough for serious use.

This is why we can't have nice things.

The (Second) Solution

We had some pretty intelligent and talented folks attempt to patch this, but

they were unsuccessful at unwinding the rather complicated build process by

which JRuby tests and releases jopenssl. Ultimately, we decided that since

the JRuby application was internal only, that we could extend our policy for

internal services to include the strongest two ciphers JRuby supported at the

time. This meant adding another top scope puppet variable for use there:

# manifests/site.pp


$legacy_ssl_ciphers = "${ssl_ciphers}:JRUBY_WEAK_SAUCE"


And then, once more, referencing it in the proper template.

The (Third) Problem: ENOMENCLATURE

After another brief reprieve, along came a "proper" Java application. You may recall that I mentioned JRuby cannot use OpenSSL - well, this is because the JVM, being a cross-platform runtime, provides its own implementation of the TLS stack via a set of libraries referred to as JSSE (Java Secure Socket Extension). Now, for a brief digression.

OpenSSL cipher names are, you see, only barely based in reality. TLS cipher names are defined in RFCs, and our scapegoat cipher's official name, for example, is TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384. In my experience, OpenSSL almost always (annoyingly) ignores these names in favor of one they themselves make up. JSSE on the other hand, (correctly, though I rarely use that word in the context of Java) uses the RFC names.

Still with me? Great. As you may have put together, adding a Java app meant that our existing variable was not going to be able to do what we needed on its own. Attempts to cobble together a programmatic mapping via files describing RFC names and tricks with OpenSSL syntax were fairly successful, but unidirectional, relatively brittle and prone to needing manual updates of the RFC name list in the future.

The (Third) Solution

As you may have guessed, it's certainly simple enough to do the following, using Java-compatible cipher names:

# manifests/site.pp


$java_ssl_ciphers  = 'IDK:WHO:I:EVEN:AM:ANYMORE'


And then use that variable as needed in the various Java templates, which are almost always XML.

The (Fourth) Problem: EERLANG

As I'm sure you've guessed by now, after a bit of smooth sailing, something

else came along. This time, it was RabbitMQ, which is written in Erlang.

Rabbit (and possibly other Erlang tools) support SSL cipher configuration via

an array of 3-element tuples. In RabbitMQ, our example cipher would be

expressed as: {ecdhe_rsa,aes_256_gcm,sha384}. Now, let me first say that, academically, this is very clever. Practically, though, I want to start a pig farm.

The (Fourth) Solution

At this point, a hash, or really any data structure, was starting to look more appealing than a never-ending string of arbitrarily named global variables, so a refactor takes us to:

# manifests/site.pp

$ssl_ciphers = {

  'openssl' => 'MY:HANDS:ARE:TYPING:WORDS',

  'jruby'   => 'OH:MY:DEAR:WORD:WHY?!?',

  'java'    => 'PLEASE:MAKE:IT:STOP',

  'erlang'  => "[{screw,this,I},{am,going,home}]"


The Conclusion

Well, by now, you have possibly realized that everything is terrible and I need a drink (and perhaps a bit of therapy). While that is almost certainly true, it's worth noting a few things:

  • TLS, while important, can be very tedious/error-prone to deploy - this is why best practices are important!

  • A good config management solution is worth a thousand rants - our use of config management reduced this problem to figuring out the proper syntax for each new technology rather than reinventing the entire approach each time. This is more valuable than I can possibly express, and can be achieved with basically any config management framework available.

While we can't currently have nice things (and TLS is not the only reason), tools like Puppet (and its many alternatives, such as Chef, Ansible or Salt) can let us start thinking about a world where we can. With enough traction, maybe we'll get there one day. For now, I'll be off to find that drink.



I'd like to thank the following people for their help with this, regardless of whether they realized or volunteered to do so.

  • My wonderful partner Mandy, for her endless support (and proofing early drafts of this)
  • My SysAdvent editor, Tom, for being flexible and thorough
  • My former colleague, William, who inadvertantly mentored much of my TLS education

No comments :