Glassfish and Intermediate SSL Certificates

About a week ago, EngSoc bought a wildcard SSL certificate for our online properties. We've been running with self-signed certificates for a few years now and our users have had to put up with those security warnings.

One of the properties we wanted to secure with this certificate was a Glassfish installation that the engineering society uses for elections. I installed the certificate and confirmed it worked in Internet Explorer and Chrome -- at the time I didn't have Firefox installed on my computer so I didn't give it much thought.. but when I started getting reports that the warning was still popping up in Firefox, I had to investigate.

There are a couple of aggravating circumstances that caused this to happen -- Murphy must have been having a field day.

1. Internet Explorer and Chrome access the same set of root CA certificates, as managed by the Windows Crypto API. Firefox uses its own certificate store.
2. The server cert we bought is issued by an intermediate CA. This means that the server cert is not directly certified by the most commonly distributed root CA certs. Any server using this cert must send the intermediate cert along with the server cert, or else the browser cannot build a trust path from the server CA to one of the root CAs.
3. For some reason, the Windows CA store happens to have the intermediate CA cert, while the Firefox store does not. For example, Windows could simply have a newer set of certs, but the reason is immaterial for this discussion. The consequence is that IE/Chrome do not show a warning, while FF does:
   
   
   
   
4. Apache's mod_ssl can be configured to send this intermediate CA using the SSLCertificateChainFile directive. Glassfish does not have documentation for this, though it is possible. Keep reading.

First, we start off with a keystore.jks that contains your private key and CA-issued cert. Presumably you got to this point by generating a key pair, sending a certificate request and importing the certificate into your store.

# keytool -keystore keystore.jks -storepass changeit -list -v

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 1 entry

Alias name: https

Creation date: Jan 30, 2009

Entry type: PrivateKeyEntry

Certificate chain length: 1

Certificate[1]:

Owner: CN=*.engsoc.org, OU=EngSoc Project, O=Carleton University, L=Ottawa, ST=Ontario, C=CA

Issuer: CN=DigiCert Global CA, OU=www.digicert.com, O=DigiCert Inc, C=US

Serial number: 780a3e6ee2948ad9a36c504fcf2c4c1

Valid from: Wed Jan 28 19:00:00 EST 2009 until: Tue Feb 02 18:59:59 EST 2010

Certificate fingerprints:

         MD5:  23:35:C5:7F:E8:77:55:4E:CE:47:FA:8E:18:E8:F0:9C

         SHA1: 9E:8E:E1:44:C8:02:4A:07:2B:6E:E9:59:34:B9:46:7B:56:AC:CE:E3

         Signature algorithm name: SHA1withRSA

         Version: 3

[...]

Note "Certificate chain length: 1". This is the server cert only and is not sufficient for Firefox to generate a trust path to one of its root CA certs. We can see that Glassfish indeed sends only this cert during the SSL handshake:

Our goal is to add the intermediate CA to this key entry and make the chain length be 2: the server cert and the intermediate CA cert.

We first need to create an X.509 file that contains both certs. Luckily, this is as simple as concatenating the two certs:

# cat certs/DigiCertCA.crt certs/star_engsoc_org.crt > certs/DigiCertCA+star_engsoc_org.crt

And now we import them both, overwriting the cert associated with the key entry:

# keytool -keystore keystore.jks -storepass changeit -importcert -alias https -file certs/DigiCertCA+star_engsoc_org.crt -noprompt

Certificate reply was installed in keystore

Now we look at the key entry again:

# keytool -keystore keystore.jks -storepass changeit -list -v

Keystore type: JKS

Keystore provider: SUN

Your keystore contains 1 entry

Alias name: https

Creation date: Feb 15, 2009

Entry type: PrivateKeyEntry

Certificate chain length: 2

Certificate[1]:

Owner: CN=*.engsoc.org, OU=EngSoc Project, O=Carleton University, L=Ottawa, ST=Ontario, C=CA

Issuer: CN=DigiCert Global CA, OU=www.digicert.com, O=DigiCert Inc, C=US

Serial number: 780a3e6ee2948ad9a36c504fcf2c4c1

Valid from: Wed Jan 28 19:00:00 EST 2009 until: Tue Feb 02 18:59:59 EST 2010

Certificate fingerprints:

         MD5:  23:35:C5:7F:E8:77:55:4E:CE:47:FA:8E:18:E8:F0:9C

         SHA1: 9E:8E:E1:44:C8:02:4A:07:2B:6E:E9:59:34:B9:46:7B:56:AC:CE:E3

         Signature algorithm name: SHA1withRSA

         Version: 3

[...]

Certificate[2]:

Owner: CN=DigiCert Global CA, OU=www.digicert.com, O=DigiCert Inc, C=US

Issuer: CN=Entrust.net Secure Server Certification Authority, OU=(c) 1999 Entrust.net Limited, OU=www.entrust.net/CPS incorp. by ref. (limits liab.), O=Entrust.net, C=US

Serial number: 4286aba0

Valid from: Fri Jul 14 13:10:28 EDT 2006 until: Mon Jul 14 13:40:28 EDT 2014

Certificate fingerprints:

         MD5:  FB:14:1E:91:00:CA:CB:77:D8:01:62:D8:8C:B8:84:48

         SHA1: 25:B7:8E:B9:36:A4:00:CE:34:13:1D:9A:6D:E8:BE:A0:4B:34:76:07

         Signature algorithm name: SHA1withRSA

         Version: 3

[...]

Indeed, there are now two certs for this key and Glassfish will now send them both during the SSL handshake:

Note also that, after connecting to the server once, Firefox added the intermediate CA cert (DigiCert Global CA) to its cert store:

Things get even more interesting when you have multiple servers using this cert. Our Apache server was properly configured to send both certs, so when a Firefox user went to an Apache site, Firefox would add the intermediate CA cert to its store. Subsequent visits to the Glassfish server would show no warnings. However, if someone visited the Glassfish server first, they would get the warning. This behaviour is reproducible by simply deleting the "DigiCert Global CA" cert from Firefox's store (it's under "Entrust.net").

Well, I hope this helped. It definitely solved our problem.

-Cat