Getting your kubernetes node names right


This past weekend while bootstrapping a new kubernetes cluster my kubeletes started logging the following error to the systemd journal:

Dec 30 10:26:10 kubworker1.prefetch.net kubelet[1202]: E1230 10:26:10.862904    1202 kubelet_node_status.go:106] Unable to register node "kubworker1.prefetch.net" with API server: nodes "kubworker1.prefetch.net" is forbidden: node "kubworker1" cannot modify node "kubworker1.prefetch.net"

Secure kubernetes configurations use client certificates along with the nodename to register with the control plane. My kubeconfig configuration file contained a short name:

apiVersion: v1
clusters:
- cluster:
    certificate-authority-data: STUFF
    server: https://apivip:443
  name: kubernetes
contexts:
- context:
    cluster: kubernetes
    user: system:node:kubworker1
  name: default
current-context: default
kind: Config
preferences: {}
users:
- name: system:node:kubworker1
  user:
    as-user-extra: {}
    client-certificate-data: STUFF
    client-key-data: STUFF

But the hostname assigned to the machine was fully qualified:

$ uname -n

kubworker1.prefetch.net

After re-reading the documentation there are two ways to address this. You can re-generate your certificates with the FQDN of your hosts or override the name with the kubelet ‘–hostname-override=NAME’ command line option. Passing the short name to the kubelet ‘–hostname-override’ option provided a quick fix and allowed my host to register:

$ kubectl get nodes

NAME         STATUS    ROLES     AGE       VERSION
kubworker1   Ready     <none>    13m       v1.9.0

I need to do some additional digging to see what the best practices are for kubernetes node naming. That will go on my growing list of kubernetes questions to get answered.

This article was posted by Matty on 2017-12-30 10:28:23 -0500 EST