This past weekend while bootstrapping a new kubernetes cluster my kubeletes started logging the following error to the systemd journal:
Dec 30 10:26:10 kubworker1.prefetch.net kubelet[1202]: E1230 10:26:10.862904 1202 kubelet_node_status.go:106] Unable to register node "kubworker1.prefetch.net" with API server: nodes "kubworker1.prefetch.net" is forbidden: node "kubworker1" cannot modify node "kubworker1.prefetch.net"
Secure kubernetes configurations use client certificates along with the nodename to register with the control plane. My kubeconfig configuration file contained a short name:
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: STUFF
server: https://apivip:443
name: kubernetes
contexts:
- context:
cluster: kubernetes
user: system:node:kubworker1
name: default
current-context: default
kind: Config
preferences: {}
users:
- name: system:node:kubworker1
user:
as-user-extra: {}
client-certificate-data: STUFF
client-key-data: STUFF
But the hostname assigned to the machine was fully qualified:
$ uname -n
kubworker1.prefetch.net
After re-reading the documentation there are two ways to address this. You can re-generate your certificates with the FQDN of your hosts or override the name with the kubelet ‘–hostname-override=NAME’ command line option. Passing the short name to the kubelet ‘–hostname-override’ option provided a quick fix and allowed my host to register:
$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
kubworker1 Ready <none> 13m v1.9.0
I need to do some additional digging to see what the best practices are for kubernetes node naming. That will go on my growing list of kubernetes questions to get answered.