Archive
Posts in Kubernetes
Using the sslsplit MITM proxy to capture Docker registry communications
This past weekend I got to debug a super fun issue! One of my Kubernetes clusters was seeing a slew of ErrImagePull errors. When I logged into one of the Kubernetes workers, the dockerd debug logs showed it had an issue pulling an image, but it didn't log WHY it couldn't pull it. Fortunately I use a private container registry, so I figured I could print the registry communications with ssldump…
$ read more →Using Kubernetes server side validation to validate your deployment manifests
Kubernetes server side validation recently landed, and it's a super useful feature. Prior to server side validation, you could use the kubectl dry-run feature to validate your deployment manifests: When this command runs, the validation occurs on the machine that hosts the kubectl binary. While useful, there are a few use cases were your manifest would validate locally, but wouldn't apply when you sent it to the API server. One example is if your kubectl binary was older than 1.16, and you tried to send a JSON payload with deprecated APIs to a 1.16+ API server…
$ read more →Using node local caching on your Kubernetes nodes to reduce CoreDNS traffic
Kubernetes 1.18 was recently released, and with it came a slew of super useful features! One feature that hit GA is node local caching. This allows each node in your cluster to cache DNS queries, reducing load on your primary in-cluster CoreDNS servers. Now that this feature is GA, I wanted to take it for a spin…
$ read more →Managing multiple Kubernetes resources by label
Kubernetes labels are super useful. If you aren't familiar with them, a label is a key/value pair assigned in the metadata section (either metadata.labels, or spec.template.metadata.labels) of a deployment manifest. The following example assigns three key/value labels to a deployment: Labels get super useful when you need to apply an action to multiple resources. Actions can include get: Which will get all pods with the label "app=nginx"…
$ read more →Finding Kubernetes issues with Popeye
Kubernetes is an incredible platform, but there are a lot of things that can go wrong. This is especially the case when you are new to K8S, and are overwhelmed with configuration options, deployment manifests, networking, and how containers work. Fortunately Kubernetes has matured quickly, and there are tons of opensource tools to troubleshoot and monitor your clusters. One of these tools, Popeye, is a must for any Kubernetes operator…
$ read more →