How the docker pull command works under the covers (with HTTP headers to illustrate the process)
I talked previously about needing to decode docker HTTP headers to debug a registry issue. That debugging session was super fun, but I had a few questions about how that interaction actually works. So I started to decode all of the HTTP requests and responses from a $(docker pull), which truly helped me solidify how the docker daemon (dockerd) talks to a container registry. I figured I would share my notes here so I (as well as anyone else on the 'net) can reference them in the future…
$ read more →Using the sslsplit MITM proxy to capture Docker registry communications
This past weekend I got to debug a super fun issue! One of my Kubernetes clusters was seeing a slew of ErrImagePull errors. When I logged into one of the Kubernetes workers, the dockerd debug logs showed it had an issue pulling an image, but it didn't log WHY it couldn't pull it. Fortunately I use a private container registry, so I figured I could print the registry communications with ssldump…
$ read more →Using Kubernetes server side validation to validate your deployment manifests
Kubernetes server side validation recently landed, and it's a super useful feature. Prior to server side validation, you could use the kubectl dry-run feature to validate your deployment manifests: When this command runs, the validation occurs on the machine that hosts the kubectl binary. While useful, there are a few use cases were your manifest would validate locally, but wouldn't apply when you sent it to the API server. One example is if your kubectl binary was older than 1.16, and you tried to send a JSON payload with deprecated APIs to a 1.16+ API server…
$ read more →Using node local caching on your Kubernetes nodes to reduce CoreDNS traffic
Kubernetes 1.18 was recently released, and with it came a slew of super useful features! One feature that hit GA is node local caching. This allows each node in your cluster to cache DNS queries, reducing load on your primary in-cluster CoreDNS servers. Now that this feature is GA, I wanted to take it for a spin…
$ read more →Managing multiple Kubernetes resources by label
Kubernetes labels are super useful. If you aren't familiar with them, a label is a key/value pair assigned in the metadata section (either metadata.labels, or spec.template.metadata.labels) of a deployment manifest. The following example assigns three key/value labels to a deployment: Labels get super useful when you need to apply an action to multiple resources. Actions can include get: Which will get all pods with the label "app=nginx"…
$ read more →