Archive
Posts from 2020
Controlling the inventory order when running an Ansible playbook
This week I was updating some Ansible application and OS update playbooks. By default, when you run ansible-playbook it will apply your desired configuration to hosts in the order they are listed in the inventory file (or in the order they are returned by a dynamic inventory script). But what if you want to process hosts in a random order? Or by their sorted or reverse sorted names…
$ read more →How the docker pull command works under the covers (with HTTP headers to illustrate the process)
I talked previously about needing to decode docker HTTP headers to debug a registry issue. That debugging session was super fun, but I had a few questions about how that interaction actually works. So I started to decode all of the HTTP requests and responses from a $(docker pull), which truly helped me solidify how the docker daemon (dockerd) talks to a container registry. I figured I would share my notes here so I (as well as anyone else on the 'net) can reference them in the future…
$ read more →Using the sslsplit MITM proxy to capture Docker registry communications
This past weekend I got to debug a super fun issue! One of my Kubernetes clusters was seeing a slew of ErrImagePull errors. When I logged into one of the Kubernetes workers, the dockerd debug logs showed it had an issue pulling an image, but it didn't log WHY it couldn't pull it. Fortunately I use a private container registry, so I figured I could print the registry communications with ssldump…
$ read more →Using dockle to check docker containers for known issues
As an SRE, I'm always on the look out for tooling that can help me do my job better. The Kubernetes ecosystem is filled with amazing tools, especially ones that can validate that your clusters and container images are configured in a reliable and secure fashion. One such tool is dockle. If you haven't heard of it, dockle is a container scanning tool that can be used verify that your containers are adhering to best practices…
$ read more →Decoding JSON Web Tokens (JWTs) from the Linux command line
Over the past few months I've been spending some of my spare time trying to understand OAUTH2 and OIDC. At the core of OAUTH2 is the concept of a bearer token. The most common form of bearer token is the JWT (JSON Web Token), which is a string with three hexadecimal components separated by periods (e.g., XXXXXX.YYYYYYYY.ZZZZZZZZ). There are plenty of online tools available to decode JWTs, but being a command line warrior I wanted something I could use from a bash prompt…
$ read more →