How Chrome 57 and StartSSL broke my website and how I fixed it

Or: How I made my website so secure, nobody could see it.

Summary

A few weeks ago I updated Chrome to the latest version (57) and found I couldn't browse to https://tomssl.com. After indulging in a few moments of abject panic, a brief investigation led me to discover that, since version 57, Chrome doesn't trust WoSign or StartCom certificates.

Due to my being extra security-conscious, resolving this problem was slightly more difficult than it might have been. However, all's well that ends well and here is an explanation of how I fixed it and why it wasn't completely straightforward. And why that's a good thing.

Background

A while ago, I wrote about how to fix your http response headers on Azure Web Apps to get an A+ on Security Headers.io, a very useful free service from Scott Helme. [Bear in mind that this gamification^[1] has the serious aim of making the web more secure. I like it.]

As part of this header-fixing effort, I decided to to deploy HPKP - HTTP Public Key Pinning. That's the Wikipedia link, but if you want to know more about it, you should read Scott Helme's article on HPKP which provides a lot more information.

In brief, this meant that I published a set of public keys and said, "only trust SSL/TLS certificates with one of these public keys". You can easily see why this might be desirable; if somebody were to hijack my site and divert traffic away to their own site, if the public keys of their certificates didn't match mine then, if a legitimate user had visited my site before, their browser would know something was amiss and wouldn't let them visit the site. At all.

Incidentally, when I glibly refer to public keys above, what I actually mean is the Base64 encoded SHA256 hash of the SPKI Fingerprint, but it amounts to the same thing. One of the nice things about this is that you can generate a pin from your Certificate Signing Request (CSR), so you don't need to buy any spare certificates in advance. From Section 2.4. Semantics of Pins of RFC7469:

An SPKI Fingerprint is defined as the output of a known cryptographic
hash algorithm whose input is the DER-encoded ASN.1 representation of
the Subject Public Key Info (SPKI) of an X.509 certificate. A Pin is
defined as the combination of the known algorithm identifier and the
SPKI Fingerprint computed using that algorithm.

You can pin the key of any of your certificates, from the leaf certificate (the specific certificate for your site) right up the chain to the root certificate, with the pinning becoming less exclusive the further up the chain you go. Clearly, if you pin a root certificate, then any certificate sharing that root certificate will be deemed to be okay. If you pin only your leaf certificate then, once that certificate expires (or is revoked), if you are unable to generate another certificate using the same public key, you are in trouble.

At this point you can probably guess where this is going. But first a bit more background.

HPKP is a TOFU (Trust On First Use) protocol. That means that the following could conceivably happen:

• Somebody visits a security-conscious website where HPKP is deployed;
• Their browser grabs the HPKP header which instructs it to trust only certificates with certain public keys and to remember those keys for a period of time (perhaps a month or two, but possibly longer);
• Each time the user visits the website, their browser generates the SHA256 hashes of the SPKIs of the presented certificates and checks that they match the keys it has already cached. If they do, it resets the amount of time for which it will remember them;
• The certificate changes to one with a different public key, which is not pinned (either by the site being hijacked or due to a legitimate certificate change being organised badly);
• Hilarity ensues Bad stuff happens as new users can browse to the site but previous users can't and, worse still, they get a scary-looking security error;
• Worse still, even if it was a legitimate change and you'd changed the pins to reflect this, bad luck, some users have cached the old pins.

In other words, if you foolishly told all browsers to remember your keys for two years and they all agreed (which they wouldn't; some browsers may set their own limit a lot lower than this, in accordance with Section 4.1 of RFC7469) and then something bad happened and you were unable to supply a certificate with the correct public key, people who had previously visited your website would be unable to do so for two years. Good luck if that happens to your business. No wonder some people suggest that you shouldn't even consider using HPKP.

What happened to me

When Chrome 57 was released in March, I duly updated my Chrome installation and this website suddenly stopped working in the manner alluded to above. As in, this happened:

Observe that there's no Proceed to ... (unsafe) link, courtesy of Chrome deciding that it really doesn't like StartCom certificates (unlike in this screenshot from another article of mine, where there is). Also remember that I have enabled HSTS preloading, including subdomains, which means that browsers like Chrome know that they mustn't visit tomssl.com (or any subdomains thereof) over plain HTTP, even if you haven't been there before. Oh dear.

And that was that. There was no way of navigating to https://tomssl.com using the current version of Chrome.

It's possible that I could have known this was going to happen^[2] but, for some reason, when I read that certificates issued later than 21st October 2016 were not going to be trusted, I didn't worry as mine had been issued in June 2015. Having taken another look at https://security.googleblog.com/2016/10/distrusting-wosign-and-startcom.html, it's still not clear to me that my certificate should suddenly have become libellus non grata^[3]. And yet, here we are (or, rather, there we were - I fixed it pretty quickly).

How I fixed it

Ordinarily, this sort of thing wouldn't present much of a problem. I'd just get a new certificate (almost certainly a free one from Let's Encrypt), install it and everything would be fine in a matter of minutes.

Unfortunately, because I'm being extra security-conscious and am using HPKP, it wasn't going to be quite so easy. I would need to get a new certificate with the correct public key.

That still sounds easy and, as long as I wasn't an idiot and had got some suitable backup pins, everything would be fine...

It didn't take long to check my pinned public keys by using Scott Helme's free tool at https://report-uri.io/home/pkp_analyse.

Doing this reminded me that the keys I needed to use to generate replacement certificates were securely stored on a machine which was (a) not with me and (b) not connected to the internet. Nice and secure, but rather inaccessible.

As luck would have it, I'd also pinned the key for COMODO's ECC Root Certificate. Presumably I did this because I thought I might use CloudFlare to provide free SSL and I knew that their provided certificates can have a short lifetime and that different subdomains might end up being SANs on different certificates. In other words, pinning the leaf certificates would be absurd as they might change without my knowledge at any moment.

Now all I needed to do was to get an ECC certificate from COMODO (with the requisite root certificate in its chain). I could do this before getting home and regaining access to my super-secure (and presently inaccessible) machine.

In the interest of preventing this article from being much longer, suffice it to say that this was also slightly more complicated than it might have been. In the end, I followed COMODO's Microsoft Servers: Create ECC CSR and Install ECC SSL Certificate instructions. This meant that I had to create the CSR, buy a certificate (from a COMODO reseller), install it, then export it (with its public key) and install the resultant .pfx file on my Azure Web App. So that's what I did and it worked.

Finally I had managed to change this

into this

Conclusion

The takeaway message from all of this is that, if you're going to use HPKP, you need to be very careful and make sure you have suitable backup pins.

If you found this article interesting or useful (or neither), you can comment below, subscribe for free Azure and SQL ebooks (I daresay you've just seen a pop-up of some kind suggesting you might like to do so. Click here to see it again. I promise not to pester you and you might even win something) or follow me on Twitter (I'll probably follow you back). Follow @TomChantler