As a long time Fedora user, I like to keep up with the planning discussions that go into each release. These discussions are super useful for understanding what is coming to Redhat Enterprise Linux and CentOS. One feature I’ve been keeping my eye on is the FSTRIM enabled by default feature. Using TRIM can free unused space on your storage arrays, and is especially important if you use thin provisioned storage devices. Well the day has finally come. In Fedora 32, the fstrim timer is now enabled by default:
$ cat /etc/fedora-release
Fedora release 32 (Thirty Two)
$ systemctl status -l fstrim.timer
● fstrim.timer - Discard unused blocks once a week
Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
Active: active (waiting) since Thu 2020-05-14 20:26:14 UTC; 44s ago
Trigger: Mon 2020-05-18 00:00:00 UTC; 3 days left
Triggers: ● fstrim.service
Docs: man:fstrim
May 14 20:26:14 localhost.localdomain systemd[1]: Started Discard unused blocks once a week.
The timer is configured to kick off fstrim once a week, and within a one hour window:
$ cat /usr/lib/systemd/system/fstrim.timer
[Unit]
Description=Discard unused blocks once a week
Documentation=man:fstrim
ConditionVirtualization=!container
[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true
[Install]
WantedBy=timers.target
This is super cool, and should make a ton of storage engineers super happy!
As a long time CentOS user, I’ve always winced when I took ownership of new systems with e-mail notifications enabled. Whether this was through setting MAILTO to a distribution list, or using the default and sending it to the user who created the job. Having cron send mail has woken me up more than once when /var/spool/* filled up. Give the awesome searching capabilites that come with solutions like Elasticsearch, Loki, and Gray log, I now turn cron mailing off on all of my systems. You can disable cron mail on CentOS by adding the “-s -m off” options to the CRONDARGS variable in /etc/sysconfig/crond:
$ cat /etc/sysconfig/crond
CRONDARGS=-s -m off
I currently have an awesome dashboard to show job status, and get notified through actionable channels when critical jobs fail. While the vast majority of my scheduled activities use Kubernetes Jobs, I still have a few outliers that use crond. This has drastically reduced the number of messages I receive through e-mail, and allowed me to get true alerting wrapper around mission critical stuff. Curious how many folks still use e-mail for notifications?
Kubernetes labels are super useful. If you aren’t familiar with them, a label is a key/value pair assigned in the metadata section (either metadata.labels, or spec.template.metadata.labels) of a deployment manifest. The following example assigns three key/value labels to a deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
labels:
app: nginx
group: dse
env: staging
$ kubectl get deploy nginx -o json | jq '.metadata.labels'
{
"app": "nginx",
"env": "staging",
"group": "dse"
}
Labels get super useful when you need to apply an action to multiple resources. Actions can include get:
$ kubectl get po -l app=nginx
NAME READY STATUS RESTARTS AGE
nginx-f89759699-65ngc 1/1 Running 0 114s
nginx-f89759699-98qdd 1/1 Running 0 114s
nginx-f89759699-b7mbs 1/1 Running 0 114s
nginx-f89759699-g2xcq 1/1 Running 0 114s
nginx-f89759699-gbjb9 1/1 Running 0 114s
nginx-f89759699-j9cnx 1/1 Running 0 114s
nginx-f89759699-mpqjl 1/1 Running 0 114s
nginx-f89759699-r9csq 1/1 Running 0 114s
nginx-f89759699-t44fq 1/1 Running 0 114s
nginx-f89759699-vppwf 1/1 Running 0 114s
Which will get all pods with the label “app=nginx”. You can also use this with actions like “delete”, which will trigger the deletion of any pod matching the label passed to the “-l” option:
$ kubectl delete po -l app=nginx
pod "nginx-f89759699-65ngc" deleted
pod "nginx-f89759699-98qdd" deleted
pod "nginx-f89759699-b7mbs" deleted
pod "nginx-f89759699-g2xcq" deleted
pod "nginx-f89759699-gbjb9" deleted
pod "nginx-f89759699-j9cnx" deleted
pod "nginx-f89759699-mpqjl" deleted
pod "nginx-f89759699-r9csq" deleted
pod "nginx-f89759699-t44fq" deleted
pod "nginx-f89759699-vppwf" deleted
Additionally, this can also be used with actions like “drain”, “taint” and “untaint”. If you are working with hundreds of nodes, or thousands of pods, this will save you a ton of time when debugging and managing your infrastructure!
Kubernetes is an incredible platform, but there are a lot of things that can go wrong. This is especially the case when you are new to K8S, and are overwhelmed with configuration options, deployment manifests, networking, and how containers work. Fortunately Kubernetes has matured quickly, and there are tons of opensource tools to troubleshoot and monitor your clusters. One of these tools, Popeye, is a must for any Kubernetes operator. Popeye will evaulate your clusters against best practices, and display warnings if it finds issues.
Getting going with Popeye is a breeze. If you have Krew installed, you can install the plug-in with the following command:
$ kubectl krew install popeye
To audit a cluster, you can pass the “popeye” option to kubectl:
$ kubectl popeye
This will produce a comprehensive report similar to the following:
___ ___ _____ _____ K .-'-.
| _ \___| _ \ __\ \ / / __| 8 __| `\
| _/ _ \ _/ _| \ V /| _| s `-,-`--._ `\
|_| \___/_| |___| |_| |___| [] .->' a `|-'
Biffs`em and Buffs`em! `=/ (__/_ /
\_, ` _)
`----; |
GENERAL [KIND-TEST]
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Connectivity...................................................................................✅
· MetricServer...................................................................................💥
CLUSTERS (1 SCANNED) 💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Version........................................................................................✅
✅ [POP-406] K8s version OK.
CLUSTERROLES (60 SCANNED) 💥 0 😱 0 🔊 60 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· admin..........................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· cluster-admin..................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· edit...........................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· kindnet........................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· kubeadm:get-nodes..............................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· local-path-provisioner-role....................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:aggregate-to-admin......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:aggregate-to-edit.......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:aggregate-to-view.......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:auth-delegator..........................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:basic-user..............................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:certificatesigningrequests:nodeclient...............................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:certificatesigningrequests:selfnodeclient...........................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:kube-apiserver-client-approver......................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:kube-apiserver-client-kubelet-approver..............................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:kubelet-serving-approver............................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:certificates.k8s.io:legacy-unknown-approver.............................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:attachdetach-controller......................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:certificate-controller.......................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:clusterrole-aggregation-controller...........................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:cronjob-controller...........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:daemon-set-controller........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:deployment-controller........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:disruption-controller........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:endpoint-controller..........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:endpointslice-controller.....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:expand-controller............................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:generic-garbage-collector....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:horizontal-pod-autoscaler....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:job-controller...............................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:namespace-controller.........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:node-controller..............................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:persistent-volume-binder.....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:pod-garbage-collector........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:pv-protection-controller.....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:pvc-protection-controller....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:replicaset-controller........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:replication-controller.......................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:resourcequota-controller.....................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:route-controller.............................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:service-account-controller...................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:service-controller...........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:statefulset-controller.......................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:controller:ttl-controller...............................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:coredns.................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:discovery...............................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:heapster................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:kube-aggregator.........................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:kube-controller-manager.................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:kube-dns................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:kube-scheduler..........................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:kubelet-api-admin.......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:node....................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:node-bootstrapper.......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:node-problem-detector...................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:node-proxier............................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:persistent-volume-provisioner...........................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:public-info-viewer......................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· system:volume-scheduler........................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
· view...........................................................................................🔊
🔊 [POP-400] Used? Unable to locate resource reference.
CLUSTERROLEBINDING 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
CONFIGMAP 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
DAEMONSET 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
DEPLOYMENT 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
HORIZONTALPODAUTOSCALER 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
INGRESS 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
NAMESPACES (1 SCANNED) 💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· default........................................................................................✅
NETWORKPOLICY 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
PERSISTENTVOLUME 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
PERSISTENTVOLUMECLAIM 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
PODS (1 SCANNED) 💥 1 😱 0 🔊 0 ✅ 0 0٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· default/centos.................................................................................💥
🔊 [POP-206] No PodDisruptionBudget defined.
😱 [POP-300] Using "default" ServiceAccount.
😱 [POP-301] Connects to API Server? ServiceAccount token is mounted.
😱 [POP-302] Pod could be running as root user. Check SecurityContext/image.
🐳 centos
💥 [POP-100] Untagged docker image in use.
😱 [POP-106] No resources requests/limits defined.
😱 [POP-102] No probes defined.
😱 [POP-306] Container could be running as root user. Check SecurityContext/Image.
PODDISRUPTIONBUDGET 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
PODSECURITYPOLICY 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
REPLICASET 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
ROLE 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
ROLEBINDING 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
SECRETS (1 SCANNED) 💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· default/default-token-zvwkm....................................................................✅
SERVICES (1 SCANNED) 💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· default/kubernetes.............................................................................✅
SERVICEACCOUNTS (1 SCANNED) 💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· default/default................................................................................✅
STATEFULSET 💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
· Nothing to report.
SUMMARY
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
Your cluster score: 95 -- A
o .-'-.
o __| A `\
o `-,-`--._ `\
[] .->' a `|-'
`=/ (__/_ /
\_, ` _)
`----; |
The official documentation describes the report morphology. These break down into Ok, Info, Warn and Error codes. Whenever I take ownership of an existing cluster, or help friends debug issues, Popeye and kubeaudit are run to help me understand where the cluster stands. Popeye also has a number of options to control the output that is produced:
-o, --out string Specify the output type (standard, jurassic, yaml, json, html, junit, prometheus, score) (default "standard")
This makes it super easy to add Popeye to a deployment pipeline, security dashboard, or just about anything you can think of. While Popeye won’t produce a delectable burger for Wimpy, it will help you understand issues in your cluster!
In this day and age of security breaches, system compromises, and critical CVEs coming out almost daily, security best practices should go into everything we do. There is an almost limitless number of tools for auditing, securing, and reporting on system and network security issues. One of these tools, Kubeaudit, allows you to audit the security posture of your Kubernetes clusters. Kubeaudit ships with a numerous audit rules, which cover the major Kubernetes security best practices.
To get started with kubeadit, you will need to download a binary from the projects Github release page:
$ curl -L0 -o kubeaudit.tgz https://github.com/Shopify/kubeaudit/releases/download/v0.7.0/kubeaudit_0.7.0_linux_amd64.tar.gz
$ tar xfvz kubeaudit*.gz
Once downloaded, you can run it without any arguments to see the available options:
$ kubeaudit
Available Commands:
all Run all audits
allowpe Audit containers that allow privilege escalation
apparmor Audit containers running without AppArmor
autofix Automagically fixes a manifest to be secure
caps Audit container for capabilities
help Help about any command
image Audit container images
limits Audit containers running with limits
mountds Audit containers that mount /var/run/docker.sock
namespaces Audit Pods for hostNetwork, hostIPC and hostPID
nonroot Audit containers running as root
np Audit namespace network policies
priv Audit containers running as privileged
rootfs Audit containers with read only root filesystems
sat Audit automountServiceAccountToken = true pods against an empty (default) service account
seccomp Audit containers running without Seccomp
version Print the version number of kubeaudit
As you can see, there are options to audit pretty much every aspect of your cluster. Are image tags defined? Yup! Capabilities? For sure! Containers running without security contexts set? Indeed! To start an audit, pick the item you want to audit, and pass it as an argument to kubeaudit. The following example uses the “all” command to audit everything, and limits the audit to the default namespace:
$ kubeaudit all -n default
INFO[0000] Not running inside cluster, using local config
ERRO[0000] AllowPrivilegeEscalation not set which allows privilege escalation, please set to false Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] ReadOnlyRootFilesystem not set which results in a writable rootFS, please set to true Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] RunAsNonRoot is not set in ContainerSecurityContext, which results in root user being allowed! Container=centos KubeType=pod Name=centos Namespace=default
WARN[0000] serviceAccount is a deprecated alias for ServiceAccountName, use that one instead DSA=default KubeType=pod Name=centos Namespace=default SA=default
WARN[0000] Privileged defaults to false, which results in non privileged, which is okay. Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=AUDIT_WRITE Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=CHOWN Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=DAC_OVERRIDE Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=FOWNER Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=FSETID Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=KILL Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=MKNOD Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=NET_BIND_SERVICE Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=NET_RAW Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=SETFCAP Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=SETGID Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=SETPCAP Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=SETUID Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=SYS_CHROOT Container=centos KubeType=pod Name=centos Namespace=default
WARN[0000] Resource limit not set, please set it! KubeType=pod Name=centos Namespace=default
WARN[0000] Image tag was missing Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] AppArmor annotation missing. Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Seccomp annotation missing. Container=centos KubeType=pod Name=centos Namespace=default
Your head may spin the first time you run kubeaudit. I know mine did when I ran it with the all option, and against all namespaces. But fear not, a lot of the output is repetitive, and can be addressed with simple changes to your deployment manifests. Setting resource requests and limits, dropping unneeded capabilities, using security contexts, etc. Kubeaudit also has an “autofix” command, which can update a manifest to comply with security best practices:
$ kubectl run centos --image=centos -o yaml --dry-run > centos.yaml
$ copy centos.yaml centos-autofix.yaml
$ kubeaudit autofix -f centos-autofix.yaml
One the manifest is updated, you can use your favorite diff utility to see the changes kubeaudit made:
$ sdiff centos.yaml centos-autofix.yaml | more
> ---
apiVersion: v1 apiVersion: v1
kind: Pod kind: Pod
metadata: metadata:
creationTimestamp: null creationTimestamp: null
labels: labels:
run: centos1 run: centos1
name: centos1 name: centos1
> annotations:
> container.apparmor.security.beta.kubernetes.io/centos1: r
> seccomp.security.alpha.kubernetes.io/pod: runtime/default
spec: spec:
containers: containers:
- image: centos - image: centos
name: centos1 name: centos1
resources: {} resources: {}
> securityContext:
> allowPrivilegeEscalation: false
> capabilities:
> drop:
> - AUDIT_WRITE
> - CHOWN
I’ve addressed the vast majority of these findings by creating templates that comply with security best practices. These templates are then used when I create new deployments, and verified when new manifests are checked into version control. This is a super useful utility, and will definitely take your Kubernetes security posture to 11.