This evening while building out a new cluster I came across another fun kubelet error:
Jan 17 20:15:53 kubworker5.prefetch.net kube-proxy.v1.9.0[23071]: E0117 20:15:53.807410 23071 proxier.go:1701] Failed to delete stale service IP 10.2.0.10 connections, error: error deleting connection tracking state for UDP service IP: 10.2.0.10, error: error looking for path of conntrack: exec: "conntrack": executable file not found in $PATH
The message was relatively straight forward. My kubelet daemon couldn’t find the conntrack executable to remove a service. If you aren’t familiar with conntrack(8) the manual page has a solid description:
conntrack provides a full featured userspace interface to the netfilter connection tracking system that is intended to replace the old /proc/net/ip_conntrack interface. This tool can be used to search, list, inspect and maintain the connec‐ tion tracking subsystem of the Linux kernel. Using conntrack , you can dump a list of all (or a filtered selection of) currently tracked connections, delete connections from the state table, and even add new ones.
What perplexed me about this was the use of exec() to interface with conntrack. I had been under the assumption that Kubernetes used the native APIs exposed to userland through the netfilter conntrack shared library. After 20-minutes of reading code I came across the ClearUDPConntrackForIP in conntrack.go which cleared that up:
err := ExecConntrackTool(execer, parameters...)
Installing the conntrack executable on my workers cleared up the issue and my service was removed. I’m learning the only way to truly learn Kubernetes is by reading code. And there’s a LOT of code. :)