Ways to debug Kubernetes pods without shells
Debugging production issues can sometimes be a challenge in Kubernetes environments. One specific challenge is debugging containers that don't contain a shell. You may have seen the following when troubleshooting an issue: Not including a shell in your base image is a best practice, and projects like distroless make it super easy to package your applications with a small shell-less footprint. But when apps go rogue, what options do we have to debug them if the container doesn't include a shell…
$ read more →The importance of the C asm volatile statement
Last month I started a course that teaches you how to write your own Operating System. Working at the intersection of hardware and software (X86 Assembly and C) has been incredibly rewarding. I've learned a TON! One interesting thing I came across in the Linux kernel's bootloader code is the use of "asm volatile"…
$ read more →Diving into container images
Container images are one of the items that makes up a "container." In most cases container images use a base image (e.g., Alpine, Ubuntu, etc.), and then one or more application-specific layers are added on top of that. There are numerous documented best practices for optimizing container images, and these best practices result in smaller images, less network traffic, and a reduction in container creation time. Unfortunately in practice, I've seen numerous cases were these best practices weren't followed. I've come across Dockerfiles that used dozens of RUN commands, didn't take advantage of multistaged builds, didn't optimize for image layer re-use, etc…
$ read more →Why df fails to show one or more file systems when run as an unprivileged user
One of my friends recently reached out with a fun problem. His monitoring system was periodically not firing when file systems grew past the thresholds he defined. When we hopped on one of his EC2 instances to debug the issue, I noticed that we were getting a permission denied (EACCES) errno when running df as their monitoring user: When we ran the same command as trusty UID 0, everything worked as expected: A quick check with strace verified this as well: If you aren't familiar with statfs(2), it returns information about a mounted file system in a statfs structure. Here is a blurb from the manual page describing which information is returned: I thought that df was setuid root like the mount uility, so when I initially saw the permission denied error I thought it was something unrelated to permissions…
$ read more →Using the Kubernetes K14S kapp utility to view deployment manifest changes prior to applying them
If you've worked with Kubernetes for any length of time, you are probably intimately familiar with deployment manifests. If this concept is new to you, deployment manifests are used to add resources to a cluster in a declarative manor. Some of the larger projects (cert-manager, Istio, CNI plug-ins, etc.) in the Kubernetes ecosystem provide manifests to deploy the resources that make their application work. These can often be 1000s of lines, and if you are security conscious you don't want to deploy anything to a cluster without validating what it is…
$ read more →