How the kubectl port-forward command works

This afternoon I was digging into the kube-dns service using the kuard demo application from Kubernetes: Up and Running. Kuard has a "Server side DNS query" menu which you can use to resolve a name in a pod and display the results in your browser. To create a kuard instance for testing I ran my trusty old friend kubectl:

$ kubectl run --image=gcr.io/kuar-demo/kuard-amd64:1 kuard

This spun up one pod:

$ kubectl get pods -o wide

NAME                     READY     STATUS    RESTARTS   AGE       IP
NODE
kuard-59f4bf4795-m9bzb   1/1       Running   0          30s       10.1.4.8
kubworker4.prefetch.net

To gain access to kuard I created a port-forward from localhost:8080 to port 8080 in the pod:

$ kubectl port-forward kuard 8080:8080

Forwarding from 127.0.0.1:8080 -> 8080

When I tried to access localhost:8080 in chrome I received the following error:

Handling connection for 8080
E0203 14:12:54.651409   20574 portforward.go:331] an error occurred forwarding
8080 -> 8080: error forwarding port 8080 to pod
febaeb6b4747d87036534845214f391db41cda998a592e541d4b6be7ff615ef4, uid : unable
to do port forwarding: socat not found.

The errors was pretty self explanatory, the worker didn't have the socat binary installed. I didn't recall seeing a pre-requisite for the socat utility so I decided to start digging through the kubelet code to see how port-forward works. After poking around the kubelet source code I came across docker_streaming.go. This file contains a function portForward(...) which contains the port-forward logic. The code for port-forward is actually pretty straight forward and uses socat and nsenter to accomplish its job. First, the function checks to see if socat and nenter exist:

       containerPid := container.State.Pid
        socatPath, lookupErr := exec.LookPath("socat")
        if lookupErr != nil {
                return fmt.Errorf("unable to do port forwarding: socat not
found.")
        }

       args := []string{"-t", fmt.Sprintf("%d", containerPid), "-n", socatPath,
"-", fmt.Sprintf("TCP4:localhost:%d", port)}

        nsenterPath, lookupErr := exec.LookPath("nsenter")
        if lookupErr != nil {
                return fmt.Errorf("unable to do port forwarding: nsenter not
found.")
        }

If both checks pass it will exec() nsenter passing the target process id (the PID of the pause container) to "-t" and the command to run to "-n":

        commandString := fmt.Sprintf("%s %s", nsenterPath, strings.Join(args, "
"))
        glog.V(4).Infof("executing port forwarding command: %s", commandString)

        command := exec.Command(nsenterPath, args...)
        command.Stdout = stream

This can be verified with Brendan Gregg's execsnoop utility:

$ execsnoop -n nsenter

PCOMM            PID    PPID   RET ARGS
nsenter          25898  976      0 /usr/bin/nsenter -t 4947 -n /usr/bin/socat -
TCP4:localhost:8080

I love reading code and have a much better understanding of how port-forward works now. If you utilize this feature to access your workers you will need to make sure nsenter and socat are installed.