This past weekend I got to debug a super fun issue! One of my Kubernetes clusters was seeing a slew of ErrImagePull errors. When I logged into one of the Kubernetes workers, the dockerd debug logs showed it had an issue pulling an image, but it didn’t log WHY it couldn’t pull it. Fortunately I use a private container registry, so I figured I could print the registry communications with ssldump. When I ran ssldump with the registries private key, I saw numerous application_data lines, but no data.
It turns out newer versions of TLS use forward secrecy. When forward secrecy is used, the server’s private key is used to encrypt the initial communications, and to negotiate a session key which is used for the remainder of the TLs session. Ssldump doesn’t currently support decrypting communications with the session key, so I needed to come up with plan B to get the HTTP headers. It turns out the sslsplit utility is an ideal solution for this! If you haven’t used it, SSLsplit is a MITM proxy which can be configured to decrypt TLS communications from one or more clients.
SSLsplit is easy to use, but needs a few things in place before it can start decoding TLS record layer messages. First, we need to generate a RootCA certificate and the associated private key. Mkcert makes this super easy:
$ mkcert -install
The new RootCA is used to mint the certificate that sslsplit will present to the client (dockerd in this case). Next, we need to tell the OS (CentOS 7 in this case) to trust the new CA certificate:
$ cd /etc/pki/ca-trust/source/anchors
$ cp /home/vagrant/.local/share/mkcert/rootCA.pem .
$ update-ca-trust
If you skip this step, docker will complain about an unknown certificate authority:
$ docker pull nginx
Error response from daemon: Get https://harbor/v2/: x509: certificate signed by unknown authority
Next we need to restart dockerd to pick up the new RootCA certificate:
$ systemctl restart docker
Now that the foundation is built, we can fire up sslsplit:
$ sudo /usr/local/bin/sslsplit -k /home/vagrant/.local/share/mkcert/rootCA-key.pem -c /home/vagrant/.local/share/mkcert/rootCA.pem -P ssl 0.0.0.0 443 10.10.10.250 443 -D -X /home/vagrant/ssl.pkt -L /home/vagrant/ssl.log
In the example above, I started sslsplit with the “-k” (CA certificate private key), “-c” (CA certificate), “-D” (don’t detach from the terminal and turn on debugging), -X (log packets to file), -L (log content to a text file), and the “-P” option. The “-P” option contains the proxy specification, which takes the following form:
PROTOCOL LISTEN_ADDRESS LISTEN_PORT HOST_TO_FORWARD_TO PORT_TO_FORWARD_TO
And that’s it! Now if you add an entry to /etc/hosts with your container registry name and the local IP, all communications will flow through SSLsplit. Once you gather the information you need, you can remove the host file entry and stop sslsplit. Then you can review the log or feed the packet capture to your favorite decoder to see what’s going on. My issue turned out to be a Harbor configuration issue, which was easily fixed once I reviewed the HTTP requests and responses.