This afternoon I was digging into the kube-dns service using the kuard demo application from Kubernetes: Up and Running. Kuard has a “Server side DNS query” menu which you can use to resolve a name in a pod and display the results in your browser. To create a kuard instance for testing I ran my trusty old friend kubectl:
$ kubectl run --image=gcr.io/kuar-demo/kuard-amd64:1 kuard
This spun up one pod:
$ kubectl get pods -o wide
NAME READY STATUS RESTARTS AGE IP NODE
kuard-59f4bf4795-m9bzb 1/1 Running 0 30s 10.1.4.8 kubworker4.prefetch.net
To gain access to kuard I created a port-forward from localhost:8080 to port 8080 in the pod:
$ kubectl port-forward kuard 8080:8080
Forwarding from 127.0.0.1:8080 -> 8080
When I tried to access localhost:8080 in chrome I received the following error:
Handling connection for 8080
E0203 14:12:54.651409 20574 portforward.go:331] an error occurred forwarding 8080 -> 8080: error forwarding port 8080 to pod febaeb6b4747d87036534845214f391db41cda998a592e541d4b6be7ff615ef4, uid : unable to do port forwarding: socat not found.
The errors was pretty self explanatory, the worker didn’t have the socat binary installed. I didn’t recall seeing a pre-requisite for the socat utility so I decided to start digging through the kubelet code to see how port-forward works. After poking around the kubelet source code I came across docker_streaming.go. This file contains a function portForward(…) which contains the port-forward logic. The code for port-forward is actually pretty straight forward and uses socat and nsenter to accomplish its job. First, the function checks to see if socat and nenter exist:
containerPid := container.State.Pid
socatPath, lookupErr := exec.LookPath("socat")
if lookupErr != nil {
return fmt.Errorf("unable to do port forwarding: socat not found.")
}
args := []string{"-t", fmt.Sprintf("%d", containerPid), "-n", socatPath, "-", fmt.Sprintf("TCP4:localhost:%d", port)}
nsenterPath, lookupErr := exec.LookPath("nsenter")
if lookupErr != nil {
return fmt.Errorf("unable to do port forwarding: nsenter not found.")
}
If both checks pass it will exec() nsenter passing the target process id (the PID of the pause container) to “-t” and the command to run to “-n”:
commandString := fmt.Sprintf("%s %s", nsenterPath, strings.Join(args, " "))
glog.V(4).Infof("executing port forwarding command: %s", commandString)
command := exec.Command(nsenterPath, args...)
command.Stdout = stream
This can be verified with Brendan Gregg’s execsnoop utility:
$ execsnoop -n nsenter
PCOMM PID PPID RET ARGS
nsenter 25898 976 0 /usr/bin/nsenter -t 4947 -n /usr/bin/socat - TCP4:localhost:8080
I love reading code and have a much better understanding of how port-forward works now. If you utilize this feature to access your workers you will need to make sure nsenter and socat are installed.