Archive
Posts in Kubernetes
Ways to debug Kubernetes pods without shells
Debugging production issues can sometimes be a challenge in Kubernetes environments. One specific challenge is debugging containers that don't contain a shell. You may have seen the following when troubleshooting an issue: Not including a shell in your base image is a best practice, and projects like distroless make it super easy to package your applications with a small shell-less footprint. But when apps go rogue, what options do we have to debug them if the container doesn't include a shell…
$ read more →Using the Kubernetes K14S kapp utility to view deployment manifest changes prior to applying them
If you've worked with Kubernetes for any length of time, you are probably intimately familiar with deployment manifests. If this concept is new to you, deployment manifests are used to add resources to a cluster in a declarative manor. Some of the larger projects (cert-manager, Istio, CNI plug-ins, etc.) in the Kubernetes ecosystem provide manifests to deploy the resources that make their application work. These can often be 1000s of lines, and if you are security conscious you don't want to deploy anything to a cluster without validating what it is…
$ read more →Using Kubernetes affinity rules to control where your pods are scheduled
Kubernetes has truly revolutioned distributed computing. While it solves a number of super hard problems, it also adds a number of new challenges. One of these challenges is ensuring your Kubernetes clusters are designed with failure domains in mind. Designing around failure domains includes things like provisioning infrastructure across availability zones, ensuring your physical servers are in different racks, or making sure the pods that support your application don't wind up on the same physical Kubernetes worker…
$ read more →Debugging Kubernetes network issues with nsenter, dig and tcpdump
As a Kubernetes administrator I frequently find myself needing to debug application and system issues. Most of the issues I encounter can be solved with Grafana dashboards and Prometheus metrics, or by running one or more Elasticsearch queries to examine logs. But there are times when I need to go deeper and actually inspect activity inside a running pod. A lot of debugging guides use the kubectl exec command to run one or more commands inside a container: But what happens if you don't have a shell installed in the container…
$ read more →How the docker pull command works under the covers (with HTTP headers to illustrate the process)
I talked previously about needing to decode docker HTTP headers to debug a registry issue. That debugging session was super fun, but I had a few questions about how that interaction actually works. So I started to decode all of the HTTP requests and responses from a $(docker pull), which truly helped me solidify how the docker daemon (dockerd) talks to a container registry. I figured I would share my notes here so I (as well as anyone else on the 'net) can reference them in the future…
$ read more →