Day 8 - Everything is a DNS Problem

Written by Kris Buytaert (@KrisBuytaert)

Systems break. Whether you like it or not, one day, they will break. Either when they are up and running or when you are building new stuff, you will one day run into problems. Sometimes the error messages will guide you to the solution quickly, but sometimes they give you no pointers at all, and sometimes there are no error messages - just weird behavior.

When that happens, it's time to pull out your troubleshooting skills. And so, you read logfiles, you google, but you find nothing; you lie awake at night trying to figure out what parameter in which config file you forgot.

In the next couple of examples, I'll guide you to some issues I ran into over the past decade. The list is far from exhaustive but it might give you an idea.

Let's start with some trivial stuff.

Who hasn't heard the "When I log on to server X, it takes a while" complaint from a user? However, when you log on to the box, it goes lightning fast. At first, you think it was a temporary glitch, but those 5 users keep complaining. You go to their workstation and, indeed, from their desk things do go a lot slower. Turns out, they are on a newer part of the building that is on the newest subnets in your organization, and for those networks, there are no reverse mappings yet. As when the users log in, the server first tries to figure out who they are, and sometimes it takes more than an acceptable timeout before the lookup has been made.

Reverse DNS lookups causing performance problems are amongst the most common problems around. They happen with databases, regular logins, etc, and sometimes people doing performance comparisons of MySQL vs NoSQL tools fall into the trap , and they end up testing a Failing DNS lookup

The quick fix, adding the hosts to your /etc/hosts file is sometimes the only alternative as you don't always have control over the reverse dns mapping.

Luckily lots of daemons also let you disable the feature, such as the skip_name_resolve entry in your my.cnf or the UseDNS=no stanza in your sshd.conf .

Don't think it's just the regular MySQL and sshd services, there's even web applications that start performing slow because of dns problems, such as one wordpress user figured out. Some applications are slow when dns is misconfigured, but plenty of applications just don't want to launch when they can't figure out where they are running. e.g an old DRBD issue caused drbdadm to crash. The easy ones to detect are the ones that actually tell you they can't lookup localhost, or the node you are starting the application on, performance issues are usually also a good pointer, but plenty of times it just doesn't show.

I've seen dns causing problems across the board: Xen, GFS, DRBD, Oracle and many others, but apart from applications that have problems with misconfigured DNS setups, there's also people who try parsing the output of dig to find out the nameserver by grepping for "SERVER" as in the comment section of the dig output it notes what nameserver it used. Now imagine the output of dig containing any of the root nameservers such as A.ROOT-SERVERS.NET indeed .. the detection will fail

DNS problems can creep up on you in expected ways in every part of your infrastructure, so what can you do to prevent them?

The first and most important problem to solve to ensure that for every part of your network, you have a correct reverse mapping. RFC 1912 clearly points out that "Every Internet-reachable host should have a name." and "Make sure your PTR and A records match. For every IP address, there should be a matching PTR record in the in-addr.arpa domain. If a host is multi-homed, (more than one IP address) make sure that all IP addresses have a corresponding PTR record (not just the first one)."

So, if you have a 172.16 RFC 1918 subnet in your network you want to have a reverse zone that looks like:

more 172.16.0.db
$TTL 604800 $ORIGIN 0.16.172.in-addr.arpa. @ IN SOA ns1.yournetwork.org. root.yournetwork.org. ( 2010101501 ; Serial 3600 ; Refresh 3600 ; Retry 2419200 ; Expire 604800 ) ; Negative Cache TTL ; IN NS ns1.yournetwork.org.

1 IN PTR ns1.yournetwork.org. 2 IN PTR zion.yournetwork.org. 3 IN PTR matrix.yournetwork.org.

For public networks addresses, you sometimes have to talk to your upstream vendor for reverse mappings that match your domain. Sometimes they already have a reverse-map.customerof.theirdomain.com, but they usually are happy to make the updates or to delegate the administration whenever possible.

The second problem when relates to updating DNS zonefiles: people often forget to update the serial number of their zonefile. After hours and hours, the newly added host still isn't known on the network or Internet and the zonefile on the primary nameserver is showing it correctly. However, the nameserver and his slave haven't realized there is a new zonefile around yet. Failing to update the serial-number of the zonefile is the default problem that everybody falls in to once in a while. What if you are using a YYYYMMDDID timestamp and you by accident put in a YYYYDDMMID timestamp in place .. chances are you need to wait a whole year before you can continue to use your old scheme, or you can add 2147483647 to the now-incorrect value as documented here.

Before I let you guys go, I do have to point you to a tool you can't live without: http://intodns.com/. This is an online service that will check your public dns config, and point out different improvements you can make. Try it! It's worth your time.

By now, you must realize that everything is a funky DNS problem, and as @patrickdebois realized, DNS stands for Devops Need Sushi, but that's a different post :)

Day 4 - Make sense of Perfmon with PAL

This article was written by Sam Cogan (@samcogan)

You've got a performance problem: your Windows server or application isn't performing as well as it should. You need to find out why. When Task Manager isn't enough it's quite likely you'll reach for Perfmon.

If you don't know already, Perfmon is the Windows performance monitor. It can be found under administrative tools on all versions of Windows from Windows 2000 onwards. It provides a way to monitor counters - metrics of your system performance, such as %CPU use, Free Memory etc. Many Windows applications come with their own custom set of counters for their specific application. We can use perfmon to setup sets of counters (Data Collector Sets) to collect data and to allow easy re-use. Once we have created a collector set we can start logging. We can leave it running however long we like, then come back and review the data. Perfmon also provides some step by step wizards to help the beginner start using it quickly.

For more information on using Perfmon, see this article

Perfmon is an excellent tool for collecting data. The problem comes when trying to analyse this data. Often, when we're not sure what the problem is, we'll add a wide range of counters to make sure we've covered all areas. Very quickly we've collected a large amount of data about the performance of the problem machine or machines.

Chances are that if you open up these files in Perfmon, then you'll end up with a selection of lines on a graph, information that seems like it should be useful, but is often hard to decipher what it actually means and if you have a lot of counters you can end up with a mess of colour lines that make a nice picture, but has no meaning whatsoever!

What do we do? How do we get some useful information that will help us solve this tricky performance problem?

Yes, you could go through and filter out the lines you don't need. What you really need is a way to give meaning to these lines - to see what is normal or abnormal, or to simplify the data or change its presentation into something more digestable. This is where PAL comes in.

PAL

Perfromance Analyis of Logs or PAL is a tool written by Clint Huffman at Microsoft, to assist in analysing Perfmon logs and to produce an HTML report that will provide detailed graphs, and indications of where a problem may lie. It provides a simple GUI to a complex Powershell script that actually does most of the work.

To understand how to use PAL, we need to look at it's 4 main components

1. The Log file
2. The Threshold File
3. Questions
4. The End Result

So, let's download PAL, fire up the GUI, and get to work!

Log Files

The Perfmon log file is where all PAL looks for data. A key advantage to using PAL is that you do not need to consider log size when you look at collecting your data. You can include all the counters you might need, and then instruct PAL to only analyse the ones you want and only during a specific time period. The only caveat to this is that PAL will not work as well if you have multiple computers feeding a single log file, as it will assume that the configuration of the machine is the same for each counter.

So, tell PAL where you log file is, and specify a time range for the data you want to analyse if you don't want to look at the whole file (if you have multiple files, we can add them later).

Threshold file

The threshold file is what makes PAL a useful tool. In essence, it's just a simple XML file used to define what kind of analysis you want PAL to do on your log files.

The first thing this file does is to define what counters you want to analyse, meaning you can skip any counters that you're not interested in or don't apply to your current problem.

Next, and the reason it is called a threshold file, it lets you set thresholds on each of your counters. This allows us to add some meaning to the data we have collected. We can define what value for each counter defines a warning state and what defines a critical state and so bring potential problems to our attention.

Finally, the threshold file allows you to specify questions to ask. We will come on to these shortly.

This may sound like a lot of work, but it doesn't have to be. PAL has a number of threshold files included that cover many of the common server configurations, e.g. Exchange, OCS, SQL, and a catch all, systems overview file. Using these built in threshold files, or by downloading additional ones from the Internet you can make extensive use of PAL without ever needing to write any code.

If these threshold files do not meet your needs, or you need one that is tailored to your needs, then PAL includes a GUI that assists you in doing just that. Selection of counters is done via the user interface. However, the calculations used to determine threshold values are written in Powershell, so some ability to write Powershell code is required.

If you haven't yet collected your Perfmon data, you can also use PAL to generate a Perfmon template from a threshold file, so that you only collect the data that this threshold file will generate.

Questions

The questions section asks you for some extra information about the machine this Perfmon log is for, for example number of CPU's, amount of memory etc. These questions are important, because the values you give can be accessed in the Powershell code inside the threshold files. These values can be used to assist in calculating your threshold levels.

Questions are configured inside the threshold file, and can be edited and added through the PAL GUI when creating a threshold file. When you add a question you create a Powershell variable that is available for use later inside threshold calculations.

Running the Analysis

Once you have selected your log, your threshold file, and answered some questions, you are ready to go. At this point, you add your job to the queue. You can then choose to either run this queue now or queue up more jobs to run in a batch. PAL analysis jobs can take a number of hours to run, depending on how much data is contained in your logs, so it is often convenient to queue up jobs to run overnight.

Ok! So, you've run your analysis and PAL has spat out some HTML files, how are these useful?

First off, for those in a rush or who want a quick overview of what problems there may be, there is the alerts section. This looks at the threshold values we had in our threshold file, and gives a visual indication of what areas of this log are in a warning, or critical state. It's very easy to quickly see where the machine is struggling, and where you may want to focus your efforts on fixing the problem.

Once you've been through all your alerts, we then have detailed information on all your selected counters. Each counter is presented separately, with a graph, alert details and statistics. Threshold data is also presented on each graph, allowing us to quickly see at what point there was an issue and how severe it was.

As you can see from this graph, this job is making the CPU a little unhappy at times and might be something that needs to be looked at!

Summary

Perfmon is a great tool for collecting performance data, but it sucks when it comes to actually analysing this data and getting some meaning from it. What looks like a big spike in the Perfmon viewer could actually be normal operation for that counter - we're missing some helpful context. We can use PAL to give us information that is not only easier to read but also has some meaning to it. From this, we can determine which data is normal and can safely be ignored, and which data is anomalous and is something that we need to investigate.

PAL is by no means perfect; getting useful reports out of this tool relies on having a threshold file that covers the counters you need to analyse with meaningful threshold calculations. If one of the included files, or one found on the internet covers this then that's great, but if not you are stuck needing to write your own using Powershell, which can be quite a complex task if you have lots of counters you want to analyse.

Users who need to analyse perfmon data alot will invest the time in customising their threshold files to get out exactly what they need, however more casual users are going to rely heavily on the pre-constructed files.

That said, more and more product areas are seeing the value in PAL and providing threshold files for their products, Exchange and OCS are two examples of this. There's no magic pixie dust in these threshold files. If you can understand Powershell, then you can write your own to handle any Counter that Perfmon can use, including third party ones. So it's a great option for people who need to analyse uncommon data. It's also open source, so if you want to see how the Powershell code behind the analysis, you can.

It's also not a quick process; log analysis can take hours. If you're looking for an immediate answer to your problem then you are going to have to use other methods of getting your answers.

If, however, you've got the right threshold file, and the time to run the analysis, then this can prove an invaluable tool to help you get meaning, and ultimately solve your problems, from Perfmon data.

Further reading:

• PAL on Codeplex
• Clint Huffman's Windows Troubleshooting in the Field Blog
• Using the Performance Analysis of Logs (PAL) Tool

Day 3 - Debugging SSL/TLS With openssl(1)

This article was written by Adam Fletcher

The target audience of this post is the working sysadmin who has setup SSL for a web server such as Apache. If you haven't done this, I'd suggest you take a look at the links on the bottom of this post. Additionally, the information here is useful for debugging the SSL pieces of puppet, syslog-ng, rsyslog, activemq, etc.

I'm going to focus on how to use openssl(1), the command line tool that ships with OpenSSL, to examine SSL connections and debug common SSL problems. I'm going to run my commands on Ubuntu Linux 9.10 with the openssl package and the ca-certificates package installed. OpenSSL should be installed on your system by default (Linux, OS X, FreeBSD, etc), but you'll probably need to install the ca-certificates package (or similar). OpenSSL is also available for Windows and with a small amount of work the commands I use below will work under Windows.

I'll use the term SSL throughout this article to indicate TLS or SSL.

Using openssl as a client

Here's what connecting to www.google.com over SSL with openssl looks like:

adamf@kid-charlemagne:/usr/lib/ssl/certs$ openssl s_client -connect www.google.com:443 -CApath /usr/lib/ssl/certs
CONNECTED(00000003)
depth=2 /iiUS/O=VeriSign, Inc./OU=Class 3 Public PrimaryeCertification nuthority
verify return:1
depth=1 /C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
verify return:1
depth=0 /C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
i:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
1 s:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDITCCAoqgAwIBAgIQL9+89q6RUm0PmqPfQDQ+mjANBgkqhkiG9w0BAQUFADBM
MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkg
THRkLjEWMBQGA1UEAxMNVGhhd3RlIFNHQyBDQTAeFw0wOTEyMTgwMDAwMDBaFw0x
MTEyMTgyMzU5NTlaMGgxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
MRYwFAYDVQQHFA1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKFApHb29nbGUgSW5jMRcw
FQYDVQQDFA53d3cuZ29vZ2xlLmNvbTCBnzANBgkqhkiG9w0BAQEFAAOBjQAwgYkC
gYEA6PmGD5D6htffvXImttdEAoN4c9kCKO+IRTn7EOh8rqk41XXGOOsKFQebg+jN
gtXj9xVoRaELGYW84u+E593y17iYwqG7tcFR39SDAqc9BkJb4SLD3muFXxzW2k6L
05vuuWciKh0R73mkszeK9P4Y/bz5RiNQl/Os/CRGK1w7t0UCAwEAAaOB5zCB5DAM
BgNVHRMBAf8EAjAAMDYGA1UdHwQvMC0wK6ApoCeGJWh0dHA6Ly9jcmwudGhhd3Rl
LmNvbS9UaGF3dGVTR0NDQS5jcmwwKAYDVR0lBCEwHwYIKwYBBQUHAwEGCCsGAQUF
BwMCBglghkgBhvhCBAEwcgYIKwYBBQUHAQEEZjBkMCIGCCsGAQUFBzABhhZodHRw
Oi8vb2NzcC50aGF3dGUuY29tMD4GCCsGAQUFBzAChjJodHRwOi8vd3d3LnRoYXd0
ZS5jb20vcmVwb3NpdG9yeS9UaGF3dGVfU0dDX0NBLmNydDANBgkqhkiG9w0BAQUF
AAOBgQCfQ89bxFApsb/isJr/aiEdLRLDLE5a+RLizrmCUi3nHX4adpaQedEkUjh5
u2ONgJd8IyAPkU0Wueru9G2Jysa9zCRo1kNbzipYvzwY4OA8Ys+WAi0oR1A04Se6
z5nRUP8pJcA2NhUzUnC+MY+f6H/nEQyNv4SgQhqAibAxWEEHXw==
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
issuer=/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
---
No client certificate CA names sent
---
SSL handshake has read 1772 bytes and written 307 bytes
---
New, TLSv1/SSLv3, Cipher is RC4-SHA
Server public key is 1024 bit
Secure Renegotiation IS supported
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : RC4-SHA
    Session-ID: 19C0B820C3375AC22DC23A931B07AAB2C875044855060D792B48FBC26936973C
    Session-ID-ctx: 
    Master-Key: 4E40947E5C3F6696951705A4BB5DD19F77EE6AC2AC83D963CE3278742AC14083EA78541C43DA85F5A05CE61977533A31
    Key-Arg   : None
    Start Time: 1290965727
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---

In the following examples I won't be pasting the whole output of the openssl command because they'll look much like the one above. Instead, I'll post the parts that matter.

The s_client argument to openssl puts openssl into client mode, and -connect tells openssl which host and port to connect to (top-level arguments to the openssl command have no dash, but subarguments do, and note that -help is a subargument to most top-level arguments that will give you more details on the top-level argument).

The final argument, -CApath, is perhaps the most interesting. The -CApath is the location of all of the CA certificates that the client trusts (note that this path may be different on different Linux distributions, and is provided by the 'ca-certificates' package in Ubuntu 9.10). SSL works on a chain of trust, meaning that the client trusts that the server is who the server says it is because a third party has verified the server's identity.

This third party is the "Certificate Authority" (CA) and the way that this trust works is that the CA has its own public certificate which every client has a copy of (we'll call this the 'CA certificate'. Your web browser comes with a set of trusted CA certificates that the web browser uses to verify that servers like www.google.com and www.amazon.com are who they say they are during an SSL-secured session. In the command above we're telling the openssl command to look for those trusted certificates in the directory given to the -CApath argument.

If we didn't do this, you'd see the string verify error:num=20:unable to get local issuer certificate in the output of openssl:

adamf@kid-charlemagne:~$ openssl s_client -connect www.google.com:443
CONNECTED(00000003)
depth=1 /C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
verify error:num=20:unable to get local issuer certificate
[...]
    Verify return code: 20 (unable to get local issuer certificate)

A CA certificate doesn't look any different from the certificate you setup in your web browser; it is just a certificate that the client trusts. Your server certificate is trusted by a client because that CA has digitally signed your server's certificate. When the server sends the client the server certificate the client can extract which CA certificate was used to sign the server certificate from the server certificate, and the client will then find the CA certificate in the client's collection of CA certificates. The client uses the matching CA certificate to verify the digital signature on the server certificate, and if it matches, the client will trust that the server is who the server says it is.

Self-signed certificates

Instead of connecting to www.google.com, let's connect to our own server, which has a self-signed certificate. The only thing we'll change is the host name in the -connect argument:

adamf@kid-charlemagne:/etc/apache2/sites-enabled$ openssl s_client -connect kid-charlemagne:443 -CApath /usr/lib/ssl/certs 
CONNECTED(00000003)
depth=0 /CN=kid-charlemagne
verify error:num=18:self signed certificate
verify return:1
[...]
    Verify return code: 18 (self signed certificate)

When debugging a problem it is useful to have a reference state for working, which is the request to Google, and reference state for a problem we understand, which is the above request to server with a self-signed certificate. openssl knows that our certificate is self-signed because the certificate's issuer is the same as the certificate's common name.

It's useful to know that openssl indicates most problems in the first few lines of output and again in the Verify return code line. I'll often paste just those lines in the example output below.

I want to run multiple SSL-encrypted virtual hosts on one IP address, but it isn't working!

Typically this means that you've setup multiple named-based virtual hosts in your web server, given them all different certificates but the same IP address and port. This won't work; you'll end up getting the same certificates for all the sites and the client will complain that the server's common name doesn't match the host name.

SSL works at the socket layer, so only one server certificate can be given out per IP address-socket pair (TLS has a mode which allows this as specified in RFC 4366, but this mode isn't in wide use). In order to do what you'll want you need to get a wildcard certificate, which is a certificate with a common name of *.<yourdomain>.com. These certificates tend to cost more and are only available from some CAs. In the past, some browsers rejected wildcard certs, but I've not seen a modern browser reject a wildcard certificate from a trusted CA. If you don't use a wildcard cert, you can't serve multiple virtual hosts inside your domain on one IP-socket pair. Note that wildcard certs only work inside one domain, so you can't server multiple domains under SSL with only one IP-socket pair no matter what.

OpenSSL says it can't get the local issuer certificate!

As an alumni of Computer Science House I have this problem when using their services:

adamf@kid-charlemagne:~$ openssl s_client -connect www.csh.rit.edu:443  -CApath /usr/lib/ssl/certs 
CONNECTED(00000003)
depth=0 /C=US/ST=New York/ieComnuter Science House/OU=OPComm/CN=*.csh.rit.edu
verify error:num=20:unable to get local issuer certificate
verify return:1
depth=0 /C=US/ST=New York/O=Computer Science House/OU=OPComm/CN=*.csh.rit.edu
verify error:num=27:certificate not trusted
verify return:1
depth=0 /C=US/ST=New York/O=Computer Science House/OU=OPComm/CN=*.csh.rit.edu
verify error:num=21:unable to verify the first certificate
verify return:1
[...]
    Verify return code: 21 (unable to verify the first certificate)

This means that you don't trust the CA who signed the server's certificate. For sysadmins, this case often comes up in corporate infrastructures that have their own CA and distribute that CA's cert to web browsers, and you need to connect to a server that uses a cert issued by that CA with something other than a web browser.

You'll need to find out where you can get a copy of the CA certificate used to sign the server certificate and then tell your script to trust that CA certificate. In the case above, once I download the CA certificate from Computer Science House, I can tell openssl to trust it with the -CAfile option:

adamf@kid-charlemagne:~$ openssl s_client -connect www.csh.rit.edu:443  -CApath /usr/lib/ssl/certs -CAfile ./CSH-CA-cert.crt 
CONNECTED(00000003)
depth=1 /O=Computer Science House/OU=OPComm/emailAddress=sysadmin@csh.rit.edu/L=Rochester/ST=New York/C=US/CN=OPComm
verify return:1
[...]
    Verify return code: 0 (ok)

I bought a certificate from a CA but my users don't trust it!

This could be that your CA is shady and isn't really a trusted CA, but it is most likely that your CA requires you to provide an additional set of certificates (often called 'intermediate certificates') to the user to build an unbroken chain of trust; what this means in practice is that your forgot to install all of the certificates given to you by your CA. Check to see if your CA has asked you to download a 'CA bundle' or similar; this bundle will have a few certificates inside the file that you'll need reference in your SSL config (In Apache 2.x with mod_ssl, this is the SSLCertificateChainFile directive).

For example purposes, I've created my own CA and intermediate CA. Without the correct CA bundle:

adamf@kid-charlemagne:~$ openssl s_client -connect kid-charlemagne:443 -CApath /etc/ssl/certs -CAfile CA/demoCA/cacert.pem 
CONNECTED(00000003)
[...]
depth=0 /C=US/ST=Massachusetts/L=Boston/O=A Different Example Company/OU=IT/CN=kid-charlemagne/emailAddress=kid-c@example.com
verify error:num=21:unable to verify the first certificate
verify return:1
[...]
    Verify return code: 21 (unable to verify the first certificate)

OpenSSL can't verify the server certificate because it missing a certificate in the trust chain. The missing certificate is the intermediate CA certificate.

After we've added the CA bundle to our Apache config, you can see everything works:

adamf@kid-charlemagne:~$ openssl s_client -connect kid-charlemagne:443 -CApath /etc/ssl/certs -CAfile CA/demoCA/cacert.pem 
CONNECTED(00000003)
depth=2 /C=US/ST=Massachusetts/O=Fake CA Inc./OU=IT/CN=FakeCA/emailAddress=ca@example.com
verify return:1
depth=1 /C=US/ST=Massachusetts/O=Fake CA Inc./OU=Development/CN=IntermediateFakeCA/emailAddress=intca@example.com
verify return:1
depth=0 /C=US/ST=Massachusetts/L=Boston/O=A Different Example Company/OU=IT/CN=kid-charlemagne/emailAddress=kid-c@example.com
verify return:1
---
Certificate chain
0 s:/C=US/ST=Massachusetts/L=Boston/O=A Different Example Company/OU=IT/CN=kid-charlemagne/emailAddress=kid-c@example.com
i:/C=US/ST=Massachusetts/O=Fake CA Inc./OU=Development/CN=IntermediateFakeCA/emailAddress=intca@example.com
1 s:/C=US/ST=Massachusetts/O=Fake CA Inc./OU=Development/CN=IntermediateFakeCA/emailAddress=intca@example.com
i:/C=US/ST=Massachusetts/O=Fake CA Inc./OU=IT/CN=FakeCA/emailAddress=ca@example.com
2 s:/C=US/ST=Massachusetts/O=Fake CA Inc./OU=IT/CN=FakeCA/emailAddress=ca@example.com
i:/C=US/ST=Massachusetts/O=Fake CA Inc./OU=IT/CN=FakeCA/emailAddress=ca@example.com
[...]
Verify return code: 0 (ok)

My users are reporting my certificate is expired, but it isn't!

We can examine the expiration details of our server's certificates with openssl by piping the output of the command we used above to openssl with the x509 and the -text option.

adamf@kid-charlemagne:~$ openssl s_client -connect www.csh.rit.edu:443  -CApath /usr/lib/ssl/certs -CAfile ./CSH-CA-cert.crt | openssl x509 -text
depth=1 /O=Computer Science House/OU=OPComm/emailAddress=sysadmin@csh.rit.edu/L=Rochester/ST=New York/C=US/CN=OPComm
verify return:1
depth=0 /C=US/ST=New York/O=Computer Science House/OU=OPComm/CN=*.csh.rit.edu
verify return:1
Certificate:
    Data:
        Version: 1 (0x0)
        Serial Number: 152 (0x98)
        Signature Algorithm: sha1WithRSAEncryption
        Issuer: O=Computer Science House, OU=OPComm/emailAddress=sysadmin@csh.rit.edu, L=Rochester, ST=New York, C=US, CN=OPComm
        Validity
            Not Before: Oct  7 03:43:21 2010 GMT
            Not After : Oct  7 03:43:21 2011 GMT
[...]

Here we can see that this certificate is valid from October 7th 2010 until the same date in 2011.

A certificate has both an expiration date and an not-valid-before date. Most often, when clients are reporting that the certificate is no longer valid, it is because the certificate has expired. However, you should never discount the possibility that the client has their system date set far enough in the past such that the certificate isn't valid yet (this has happened to a friend of mine twice in the past month):

adamf@kid-charlemagne:~$ date
Mon Dec  1 10:00:00 EST 2008
adamf@kid-charlemagne:~$ openssl s_client -connect www.google.com:443  -CApath /usr/lib/ssl/certs
CONNECTED(00000003)
depth=2 /C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
verify return:1
depth=1 /C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
verify return:1
depth=0 /C=US/ST=California/L=Mountain View/O=Google Inc/CN=www.google.com
verify error:num=9:certificate is not yet valid
notBefore=Dec 18 00:00:00 2009 GMT
[...]
    Verify return code: 9 (certificate is not yet valid)

It is a good idea to renew your certificate a month or so before it expires, but only install it a day or two before the current cert expires. This will help make sure clients whose system clocks are skewed to the past a few minutes or hours don't see a certificate error.

If you have a local copy of a certificate in PEM format you can also cat that file to openssl to see the details of the certificate:

adamf@kid-charlemagne:~$ cat CSH-CA-cert.crt  | openssl x509 -text
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number:
            a5:0f:04:db:62:85:0b:83
        Signature Algorithm: md5WithRSAEncryption
[...]

You can use these commands to setup a monitor that checks your certificates and warns when the certificates are a few weeks from expiring.

More One Liners

Use OpenSSL to Base64 encode/decode a file (add -in and you can specify a filename instead of stdin):

adamf@kid-charlemagne:~$ echo foo | openssl enc -base64
Zm9vCg==
adamf@kid-charlemagne:~$ echo foo | openssl enc -base64 | openssl enc -d -base64
foo

Use OpenSSL to encrypt a file:

adamf@kid-charlemagne:~$ openssl enc -aes-256-cbc -salt -in secret.txt -out secret.enc
enter aes-256-cbc encryption password:
Verifying - enter aes-256-cbc encryption password:
adamf@kid-charlemagne:~$ cat secret.enc 
?)B??Y*?U_??F?<S?/;?,?m?T1?дTGtzadamf@kid-charlemagne:~$

And decrypt:

adamf@kid-charlemagne:~$ openssl enc -d -aes-256-cbc -salt -in  secret.enc
enter aes-256-cbc decryption password:
The secret password is pencil

Use OpenSSL as an SSL-protected HTTP-based file browser:

adamf@kid-charlemagne:~/testwww$ echo "This is a file served by openssl" > foo
adamf@kid-charlemagne:~/testwww$ openssl s_server -cert ../ssl-cert-snakeoil.pem -key ../ssl-cert-snakeoil.key -WWW
Using default temp DH parameters
Using default temp ECDH parameters
ACCEPT

And test it out:

adamf@kid-charlemagne:~$ openssl s_client -connect localhost:4433
CONNECTED(00000003)
depth=0 /CN=kid-charlemagne
[...]

GET /foo HTTP 1.1
HTTP/1.0 200 ok
Content-type: text/plain

This is a file served by openssl
read:errno=0

Conclusion

Debugging SSL problems can be tricky, and I hope you've found this exploration into debugging with openssl useful. Feedback on this article is very welcome, so please feel free to comment here or hit me up on twitter.

Other Resources

The OpenSSL home page.

There's a great list of one-liners and other useful tricks you can do with on The OpenSSL Command-Line HOWTO.

Setting up Apache with SSL has many guides on the Internet; I'd suggest you Google to find out what your distribution recommends. For Ubuntu, this document covers the bases.

O'Reilly has a good tutorial on configuring Apache with SSL without use a specific distribution.

And finally, Apache's SSL documentation.