https://prefetch.net/blog/ Debugging writeups, infrastructure notes, references, code, and technical articles for people who work close to systems. en-us Tue, 02 Jun 2026 22:18:48 +0000 https://prefetch.net/blog/2026/06/01/the-new-ai-frontier/ It has been a while since I last posted in 2022. A lot has changed since then: the pandemic reshaped how we work, software and infrastructure practices continue to evolve, and AI has become one of the most significant technology shifts I have ever seen. I have always loved learning, and AI has accelerated that process in a meaningful way. It makes it easier to ask focused questions, understand unfamiliar systems, debug problems, and create documentation that reflects what was actually built… Mon, 01 Jun 2026 10:17:15 -0400 https://prefetch.net/blog/2026/06/01/the-new-ai-frontier/ https://prefetch.net/blog/2022/05/08/using-terrascan-to-detect-compliance-and-security-violations/ Over the past several years I've read numerous horror stories about cloud deployments gone wrong. S3 buckets with PCI data left open to the raw Internet, EC2 instance profiles that weren't scoped properly, misconfigured NSGs, etc. It takes a LOT of time to truly understand all the ins and outs of running workloads in the cloud, and making sure you get it "right". This is one reason I'm always on the lookout for tools that can add additional guard rails to the infrastructure provisioning process… Sun, 08 May 2022 00:00:00 -0500 https://prefetch.net/blog/2022/05/08/using-terrascan-to-detect-compliance-and-security-violations/ https://prefetch.net/blog/2022/05/01/understanding-cloud-spend-in-your-terraform-workflows/ Having worked in the "cloud" for several years, one thing that I'm super conscious about is our cloud bill. There are tons of subtleties associated with billing, such as AZ-to-AZ traffic costs or how VPC endpoints can reduce egress charges. If you utilize Terraform for infrastructure provisioning, you may want to look at infracost. Infracost can help you understand cloud spend for a green field deployment, or what it will cost to expand your existing infrastructure… Sun, 01 May 2022 00:00:00 -0500 https://prefetch.net/blog/2022/05/01/understanding-cloud-spend-in-your-terraform-workflows/ https://prefetch.net/blog/2022/04/13/using-tfswitch-to-manage-terraform-versions/ The growth of the Terraform community is absolutely astounding. New providers are constantly popping up, providers are being upgraded at a feverish pace, and amazing new features are constantly being added. With all of this change, deprecations and breaking changes periodically surface. One way to protect yourself from breaking changes is to pin providers and modules to specific versions… Wed, 13 Apr 2022 00:00:00 -0500 https://prefetch.net/blog/2022/04/13/using-tfswitch-to-manage-terraform-versions/ https://prefetch.net/blog/2022/04/08/using-the-kubernetes-can-i-subcommand-to-debug-authentication-issues/ When I was first getting started with Kubernetes, RBAC was one of the topics that took me the longest to grok. Not because the resources (Roles, ClusterRoles, etc) are hard to interpret, but learning how to scope your Roles to minimize access takes some practice. That and a lot of reading to understand the various API groups and what they contain. In a previous post I mentioned access-matrix, which is an incredible tool for visualizing the RBAC permissions an entity (User, SA, Group, etc.) has… Fri, 08 Apr 2022 01:00:00 -0500 https://prefetch.net/blog/2022/04/08/using-the-kubernetes-can-i-subcommand-to-debug-authentication-issues/ https://prefetch.net/blog/2022/04/08/ways-to-debug-kubernetes-pods-without-shells/ Debugging production issues can sometimes be a challenge in Kubernetes environments. One specific challenge is debugging containers that don't contain a shell. You may have seen the following when troubleshooting an issue: Not including a shell in your base image is a best practice, and projects like distroless make it super easy to package your applications with a small shell-less footprint. But when apps go rogue, what options do we have to debug them if the container doesn't include a shell… Fri, 08 Apr 2022 00:00:00 -0500 https://prefetch.net/blog/2022/04/08/ways-to-debug-kubernetes-pods-without-shells/ https://prefetch.net/blog/2022/04/07/the-importance-of-asm-volatile/ Last month I started a course that teaches you how to write your own Operating System. Working at the intersection of hardware and software (X86 Assembly and C) has been incredibly rewarding. I've learned a TON! One interesting thing I came across in the Linux kernel's bootloader code is the use of "asm volatile"… Thu, 07 Apr 2022 00:00:00 -0500 https://prefetch.net/blog/2022/04/07/the-importance-of-asm-volatile/ https://prefetch.net/blog/2022/03/02/diving-into-container-images/ Container images are one of the items that makes up a "container." In most cases container images use a base image (e.g., Alpine, Ubuntu, etc.), and then one or more application-specific layers are added on top of that. There are numerous documented best practices for optimizing container images, and these best practices result in smaller images, less network traffic, and a reduction in container creation time. Unfortunately in practice, I've seen numerous cases were these best practices weren't followed. I've come across Dockerfiles that used dozens of RUN commands, didn't take advantage of multistaged builds, didn't optimize for image layer re-use, etc… Wed, 02 Mar 2022 09:00:00 -0500 https://prefetch.net/blog/2022/03/02/diving-into-container-images/ https://prefetch.net/blog/2022/03/01/why-df-fails-to-show-one-or-more-file-systems-when-run-as-an-unprivileged-user/ One of my friends recently reached out with a fun problem. His monitoring system was periodically not firing when file systems grew past the thresholds he defined. When we hopped on one of his EC2 instances to debug the issue, I noticed that we were getting a permission denied (EACCES) errno when running df as their monitoring user: When we ran the same command as trusty UID 0, everything worked as expected: A quick check with strace verified this as well: If you aren't familiar with statfs(2), it returns information about a mounted file system in a statfs structure. Here is a blurb from the manual page describing which information is returned: I thought that df was setuid root like the mount uility, so when I initially saw the permission denied error I thought it was something unrelated to permissions… Tue, 01 Mar 2022 09:00:00 -0500 https://prefetch.net/blog/2022/03/01/why-df-fails-to-show-one-or-more-file-systems-when-run-as-an-unprivileged-user/ https://prefetch.net/blog/2020/08/14/using-the-kubernetes-k14s-kapp-utility-to-view-changes-prior-to-applying-them/ If you've worked with Kubernetes for any length of time, you are probably intimately familiar with deployment manifests. If this concept is new to you, deployment manifests are used to add resources to a cluster in a declarative manor. Some of the larger projects (cert-manager, Istio, CNI plug-ins, etc.) in the Kubernetes ecosystem provide manifests to deploy the resources that make their application work. These can often be 1000s of lines, and if you are security conscious you don't want to deploy anything to a cluster without validating what it is… Fri, 14 Aug 2020 01:00:00 -0500 https://prefetch.net/blog/2020/08/14/using-the-kubernetes-k14s-kapp-utility-to-view-changes-prior-to-applying-them/ https://prefetch.net/blog/2020/08/14/upgrading-an-rpm-to-a-specific-version-with-yum/ This past week I got to spend some time upgrading my CI/CD systems. The Gitlab upgrade process requires stepping to a specific version when you upgrade major versions, which can be a problem if the latest version isn't supported by the upgrade scripts . In these types of situations, you can tell yum to upgrade to a specific version. To list the versions of a package that are available, you can use the search commands "--showduplicates" option: Once you eye the version you want, you can pass it to yum install: This can also be useful if you want to stick to a minor version vs… Fri, 14 Aug 2020 00:00:00 -0500 https://prefetch.net/blog/2020/08/14/upgrading-an-rpm-to-a-specific-version-with-yum/ https://prefetch.net/blog/2020/08/03/using-kubernetes-affinity-rules-to-control-where-your-pods-are-scheduled/ Kubernetes has truly revolutioned distributed computing. While it solves a number of super hard problems, it also adds a number of new challenges. One of these challenges is ensuring your Kubernetes clusters are designed with failure domains in mind. Designing around failure domains includes things like provisioning infrastructure across availability zones, ensuring your physical servers are in different racks, or making sure the pods that support your application don't wind up on the same physical Kubernetes worker… Mon, 03 Aug 2020 02:00:00 -0500 https://prefetch.net/blog/2020/08/03/using-kubernetes-affinity-rules-to-control-where-your-pods-are-scheduled/ https://prefetch.net/blog/2020/08/03/using-the-ansible-uri-module-to-test-web-services-during-playbook-execution/ Ansible has amazing support for testing services during playbook execution. This is super useful for validating your services are working after a set of changes take place, and when combined with serial you can stop execution if a change negatively impacts one one or more servers in your fleet. Ansible has a number of modules that can be used to test services, including the uri module. The uri module allows Ansible to interact with a web endpoint, and provides numerous options to control its behavior… Mon, 03 Aug 2020 01:00:00 -0500 https://prefetch.net/blog/2020/08/03/using-the-ansible-uri-module-to-test-web-services-during-playbook-execution/ https://prefetch.net/blog/2020/08/03/debugging-kubernetes-network-issues-with-nsenter,-dig-and-tcpdump/ As a Kubernetes administrator I frequently find myself needing to debug application and system issues. Most of the issues I encounter can be solved with Grafana dashboards and Prometheus metrics, or by running one or more Elasticsearch queries to examine logs. But there are times when I need to go deeper and actually inspect activity inside a running pod. A lot of debugging guides use the kubectl exec command to run one or more commands inside a container: But what happens if you don't have a shell installed in the container… Mon, 03 Aug 2020 00:00:00 -0500 https://prefetch.net/blog/2020/08/03/debugging-kubernetes-network-issues-with-nsenter,-dig-and-tcpdump/ https://prefetch.net/blog/2020/07/29/controlling-your-inventory-order-when-running-an-ansible-playbook/ This week I was updating some Ansible application and OS update playbooks. By default, when you run ansible-playbook it will apply your desired configuration to hosts in the order they are listed in the inventory file (or in the order they are returned by a dynamic inventory script). But what if you want to process hosts in a random order? Or by their sorted or reverse sorted names… Wed, 29 Jul 2020 00:00:00 -0500 https://prefetch.net/blog/2020/07/29/controlling-your-inventory-order-when-running-an-ansible-playbook/ https://prefetch.net/blog/2020/07/22/how-the-docker-pull-command-works-under-the-covers-with-http-headers-to-illustrate-the-process/ I talked previously about needing to decode docker HTTP headers to debug a registry issue. That debugging session was super fun, but I had a few questions about how that interaction actually works. So I started to decode all of the HTTP requests and responses from a $(docker pull), which truly helped me solidify how the docker daemon (dockerd) talks to a container registry. I figured I would share my notes here so I (as well as anyone else on the 'net) can reference them in the future… Wed, 22 Jul 2020 00:00:00 -0500 https://prefetch.net/blog/2020/07/22/how-the-docker-pull-command-works-under-the-covers-with-http-headers-to-illustrate-the-process/ https://prefetch.net/blog/2020/07/16/using-the-sslsplit-mitm-proxy-to-capture-docker-registry-communications/ This past weekend I got to debug a super fun issue! One of my Kubernetes clusters was seeing a slew of ErrImagePull errors. When I logged into one of the Kubernetes workers, the dockerd debug logs showed it had an issue pulling an image, but it didn't log WHY it couldn't pull it. Fortunately I use a private container registry, so I figured I could print the registry communications with ssldump… Thu, 16 Jul 2020 00:00:53 -0500 https://prefetch.net/blog/2020/07/16/using-the-sslsplit-mitm-proxy-to-capture-docker-registry-communications/ https://prefetch.net/blog/2020/07/15/using-dockle-to-check-docker-containers-for-known-issues/ As an SRE, I'm always on the look out for tooling that can help me do my job better. The Kubernetes ecosystem is filled with amazing tools, especially ones that can validate that your clusters and container images are configured in a reliable and secure fashion. One such tool is dockle. If you haven't heard of it, dockle is a container scanning tool that can be used verify that your containers are adhering to best practices… Wed, 15 Jul 2020 00:00:00 -0500 https://prefetch.net/blog/2020/07/15/using-dockle-to-check-docker-containers-for-known-issues/ https://prefetch.net/blog/2020/07/14/decoding-json-web-tokens-(jwts)-from-the-linux-command-line/ Over the past few months I've been spending some of my spare time trying to understand OAUTH2 and OIDC. At the core of OAUTH2 is the concept of a bearer token. The most common form of bearer token is the JWT (JSON Web Token), which is a string with three hexadecimal components separated by periods (e.g., XXXXXX.YYYYYYYY.ZZZZZZZZ). There are plenty of online tools available to decode JWTs, but being a command line warrior I wanted something I could use from a bash prompt… Tue, 14 Jul 2020 00:00:00 -0500 https://prefetch.net/blog/2020/07/14/decoding-json-web-tokens-(jwts)-from-the-linux-command-line/ https://prefetch.net/blog/2020/07/07/collecting-nginx-metrics-with-the-prometheus-nginx_exporter/ Over the past year I've rolled out numerous Prometheus exporters to provide visibility into the infrastructure I manage. Exporters are server processes that interface with an application (HAProxy, MySQL, Redis, etc.), and make their operational metrics available through an HTTP endpoint. The nginx_exporter is an exporter for Nginx, and allows you to gather the stub_status metrics in a super easy way. To use this exporter, you will first need to download the nginx_exporter binary from the projects Github release page… Tue, 07 Jul 2020 01:00:00 -0500 https://prefetch.net/blog/2020/07/07/collecting-nginx-metrics-with-the-prometheus-nginx_exporter/