How the kubectl port-forward command works

This afternoon I was digging into the kube-dns service using the kuard demo application from Kubernetes: Up and Running. Kuard has a “Server side DNS query” menu which you can use to resolve a name in a pod and display the results in your browser. To create a kuard instance for testing I ran my trusty old friend kubectl:

$ kubectl run kuard

This spun up one pod:

$ kubectl get pods -o wide

NAME                     READY     STATUS    RESTARTS   AGE       IP         NODE
kuard-59f4bf4795-m9bzb   1/1       Running   0          30s

To gain access to kuard I created a port-forward from localhost:8080 to port 8080 in the pod:

$ kubectl port-forward kuard 8080:8080
Forwarding from -> 8080

When I tried to access localhost:8080 in chrome I received the following error:

Handling connection for 8080
E0203 14:12:54.651409   20574 portforward.go:331] an error occurred forwarding 8080 -> 8080: error forwarding port 8080 to pod febaeb6b4747d87036534845214f391db41cda998a592e541d4b6be7ff615ef4, uid : unable to do port forwarding: socat not found.

The errors was pretty self explanatory, the worker didn’t have the socat binary installed. I didn’t recall seeing a pre-requisite for the socat utility so I decided to start digging through the kubelet code to see how port-forward works. After poking around the kubelet source code I came across docker_streaming.go. This file contains a function portForward(…) which contains the port-forward logic. The code for port-forward is actually pretty straight forward and uses socat and nsenter to accomplish its job. First, the function checks to see if socat and nenter exist:

       containerPid := container.State.Pid
        socatPath, lookupErr := exec.LookPath("socat")
        if lookupErr != nil {
                return fmt.Errorf("unable to do port forwarding: socat not found.")

       args := []string{"-t", fmt.Sprintf("%d", containerPid), "-n", socatPath, "-", fmt.Sprintf("TCP4:localhost:%d", port)}

        nsenterPath, lookupErr := exec.LookPath("nsenter")
        if lookupErr != nil {
                return fmt.Errorf("unable to do port forwarding: nsenter not found.")

If both checks pass it will exec() nsenter passing the target process id (the PID of the pause container) to “-t” and the command to run to “-n”:

        commandString := fmt.Sprintf("%s %s", nsenterPath, strings.Join(args, " "))
        glog.V(4).Infof("executing port forwarding command: %s", commandString)

        command := exec.Command(nsenterPath, args...)
        command.Stdout = stream

This can be verified with Brendan Gregg’s execsnoop utility:

$ execsnoop -n nsenter

PCOMM            PID    PPID   RET ARGS
nsenter          25898  976      0 /usr/bin/nsenter -t 4947 -n /usr/bin/socat - TCP4:localhost:8080

I love reading code and have a much better understanding of how port-forward works now. If you utilize this feature to access your workers you will need to make sure nsenter and socat are installed.

This article was posted by Matty on 2018-02-03 16:49:40 -0500 EST