Kubernetes is an incredible platform, but there are a lot of things that can go wrong. This is especially the case when you are new to K8S, and are overwhelmed with configuration options, deployment manifests, networking, and how containers work. Fortunately Kubernetes has matured quickly, and there are tons of opensource tools to troubleshoot and monitor your clusters. One of these tools, Popeye, is a must for any Kubernetes operator. Popeye will evaulate your clusters against best practices, and display warnings if it finds issues.
Getting going with Popeye is a breeze. If you have Krew installed, you can install the plug-in with the following command:
$ kubectl krew install popeye
To audit a cluster, you can pass the “popeye” option to kubectl:
$ kubectl popeye
This will produce a comprehensive report similar to the following:
___ ___ _____ _____ K .-'-.
| _ \___| _ \ __\ \ / / __| 8 __| `\
| _/ _ \ _/ _| \ V /| _| s `-,-`--._ `\
|_| \___/_| |___| |_| |___| [] .->' a `|-'
Biffs`em and Buffs`em! `=/ (__/_ /
\_, ` _)
`----; |
GENERAL [KIND-TEST]
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Connectivity...................................................................................✅
· MetricServer...................................................................................💥
CLUSTERS (1 SCANNED) 💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Version........................................................................................✅
✅ [POP-406] K8s version OK.
CLUSTERROLES (60 SCANNED) 💥 0 😱 0 🔊 60 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· admin..........................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· cluster-admin..................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· edit...........................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· kindnet........................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· kubeadm:get-nodes..............................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· local-path-provisioner-role....................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:aggregate-to-admin......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:aggregate-to-edit.......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:aggregate-to-view.......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:auth-delegator..........................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:basic-user..............................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:certificatesigningrequests:nodeclient...............................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:certificatesigningrequests:selfnodeclient...........................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:kube-apiserver-client-approver......................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:kube-apiserver-client-kubelet-approver..............................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:kubelet-serving-approver............................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:legacy-unknown-approver.............................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:attachdetach-controller......................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:certificate-controller.......................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:clusterrole-aggregation-controller...........................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:cronjob-controller...........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:daemon-set-controller........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:deployment-controller........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:disruption-controller........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:endpoint-controller..........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:endpointslice-controller.....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:expand-controller............................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:generic-garbage-collector....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:horizontal-pod-autoscaler....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:job-controller...............................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:namespace-controller.........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:node-controller..............................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:persistent-volume-binder.....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:pod-garbage-collector........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:pv-protection-controller.....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:pvc-protection-controller....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:replicaset-controller........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:replication-controller.......................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:resourcequota-controller.....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:route-controller.............................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:service-account-controller...................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:service-controller...........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:statefulset-controller.......................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:ttl-controller...............................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:coredns.................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:discovery...............................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:heapster................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:kube-aggregator.........................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:kube-controller-manager.................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:kube-dns................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:kube-scheduler..........................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:kubelet-api-admin.......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:node....................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:node-bootstrapper.......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:node-problem-detector...................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:node-proxier............................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:persistent-volume-provisioner...........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:public-info-viewer......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:volume-scheduler........................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· view...........................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
CLUSTERROLEBINDING 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
CONFIGMAP 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
DAEMONSET 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
DEPLOYMENT 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
HORIZONTALPODAUTOSCALER 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
INGRESS 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
NAMESPACES (1 SCANNED) 💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· default........................................................................................✅
NETWORKPOLICY 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
PERSISTENTVOLUME 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
PERSISTENTVOLUMECLAIM 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
PODS (1 SCANNED) 💥 1 😱 0 🔊 0 ✅ 0 0٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· default/centos.................................................................................💥
🔊 [POP-206] No PodDisruptionBudget defined.
😱 [POP-300] Using "default" ServiceAccount.
😱 [POP-301] Connects to API Server? ServiceAccount token is mounted.
😱 [POP-302] Pod could be running as root user. Check SecurityContext/image.
🐳 centos
💥 [POP-100] Untagged docker image in use.
😱 [POP-106] No resources requests/limits defined.
😱 [POP-102] No probes defined.
😱 [POP-306] Container could be running as root user. Check SecurityContext/Image.
PODDISRUPTIONBUDGET 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
PODSECURITYPOLICY 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
REPLICASET 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
ROLE 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
ROLEBINDING 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
SECRETS (1 SCANNED) 💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· default/default-token-zvwkm....................................................................✅
SERVICES (1 SCANNED) 💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· default/kubernetes.............................................................................✅
SERVICEACCOUNTS (1 SCANNED) 💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· default/default................................................................................✅
STATEFULSET 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
SUMMARY
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
Your cluster score: 95 -- A
o .-'-.
o __| A `\
o `-,-`--._ `\
[] .->' a `|-'
`=/ (__/_ /
\_, ` _)
`----; |
The official documentation describes the report morphology. These break down into Ok, Info, Warn and Error codes. Whenever I take ownership of an existing cluster, or help friends debug issues, Popeye and kubeaudit are run to help me understand where the cluster stands. Popeye also has a number of options to control the output that is produced:
-o, --out string Specify the output type (standard, jurassic, yaml, json, html, junit, prometheus, score) (default "standard")
This makes it super easy to add Popeye to a deployment pipeline, security dashboard, or just about anything you can think of. While Popeye won’t produce a delectable burger for Wimpy, it will help you understand issues in your cluster!