Blog O' Matty


TRIM support is enabled by default in Fedora 32

This article was posted by on 2020-05-15 00:00:00 -0500 -0500

As a long time Fedora user, I like to keep up with the planning discussions that go into each release. These discussions are super useful for understanding what is coming to Redhat Enterprise Linux and CentOS. One feature I’ve been keeping my eye on is the FSTRIM enabled by default feature. Using TRIM can free unused space on your storage arrays, and is especially important if you use thin provisioned storage devices. Well the day has finally come. In Fedora 32, the fstrim timer is now enabled by default:

$ cat /etc/fedora-release

Fedora release 32 (Thirty Two)

$ systemctl status -l fstrim.timer

● fstrim.timer - Discard unused blocks once a week
     Loaded: loaded (/usr/lib/systemd/system/fstrim.timer; enabled; vendor preset: enabled)
     Active: active (waiting) since Thu 2020-05-14 20:26:14 UTC; 44s ago
    Trigger: Mon 2020-05-18 00:00:00 UTC; 3 days left
   Triggers: ● fstrim.service
       Docs: man:fstrim

May 14 20:26:14 localhost.localdomain systemd[1]: Started Discard unused blocks once a week.

The timer is configured to kick off fstrim once a week, and within a one hour window:

$ cat /usr/lib/systemd/system/fstrim.timer

[Unit]
Description=Discard unused blocks once a week
Documentation=man:fstrim
ConditionVirtualization=!container

[Timer]
OnCalendar=weekly
AccuracySec=1h
Persistent=true

[Install]
WantedBy=timers.target

This is super cool, and should make a ton of storage engineers super happy!

Disabling cron jobs globally on CentOS machines

This article was posted by on 2020-05-14 01:00:00 -0500 -0500

As a long time CentOS user, I’ve always winced when I took ownership of new systems with e-mail notifications enabled. Whether this was through setting MAILTO to a distribution list, or using the default and sending it to the user who created the job. Having cron send mail has woken me up more than once when /var/spool/* filled up. Give the awesome searching capabilites that come with solutions like Elasticsearch, Loki, and Gray log, I now turn cron mailing off on all of my systems. You can disable cron mail on CentOS by adding the “-s -m off” options to the CRONDARGS variable in /etc/sysconfig/crond:

$ cat /etc/sysconfig/crond

CRONDARGS=-s -m off

I currently have an awesome dashboard to show job status, and get notified through actionable channels when critical jobs fail. While the vast majority of my scheduled activities use Kubernetes Jobs, I still have a few outliers that use crond. This has drastically reduced the number of messages I receive through e-mail, and allowed me to get true alerting wrapper around mission critical stuff. Curious how many folks still use e-mail for notifications?

Managing multiple Kubernetes resources by label

This article was posted by on 2020-05-14 00:00:00 -0500 -0500

Kubernetes labels are super useful. If you aren’t familiar with them, a label is a key/value pair assigned in the metadata section (either metadata.labels, or spec.template.metadata.labels) of a deployment manifest. The following example assigns three key/value labels to a deployment:

apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: nginx
    group: dse
    env: staging

$ kubectl get deploy nginx -o json | jq '.metadata.labels'

{
  "app": "nginx",
  "env": "staging",
  "group": "dse"
}

Labels get super useful when you need to apply an action to multiple resources. Actions can include get:

$ kubectl get po -l app=nginx

NAME                    READY   STATUS    RESTARTS   AGE
nginx-f89759699-65ngc   1/1     Running   0          114s
nginx-f89759699-98qdd   1/1     Running   0          114s
nginx-f89759699-b7mbs   1/1     Running   0          114s
nginx-f89759699-g2xcq   1/1     Running   0          114s
nginx-f89759699-gbjb9   1/1     Running   0          114s
nginx-f89759699-j9cnx   1/1     Running   0          114s
nginx-f89759699-mpqjl   1/1     Running   0          114s
nginx-f89759699-r9csq   1/1     Running   0          114s
nginx-f89759699-t44fq   1/1     Running   0          114s
nginx-f89759699-vppwf   1/1     Running   0          114s

Which will get all pods with the label “app=nginx”. You can also use this with actions like “delete”, which will trigger the deletion of any pod matching the label passed to the “-l” option:

$ kubectl delete po -l app=nginx

pod "nginx-f89759699-65ngc" deleted
pod "nginx-f89759699-98qdd" deleted
pod "nginx-f89759699-b7mbs" deleted
pod "nginx-f89759699-g2xcq" deleted
pod "nginx-f89759699-gbjb9" deleted
pod "nginx-f89759699-j9cnx" deleted
pod "nginx-f89759699-mpqjl" deleted
pod "nginx-f89759699-r9csq" deleted
pod "nginx-f89759699-t44fq" deleted
pod "nginx-f89759699-vppwf" deleted

Additionally, this can also be used with actions like “drain”, “taint” and “untaint”. If you are working with hundreds of nodes, or thousands of pods, this will save you a ton of time when debugging and managing your infrastructure!

Finding Kubernetes issues with Popeye

This article was posted by on 2020-05-13 00:00:00 -0500 -0500

Kubernetes is an incredible platform, but there are a lot of things that can go wrong. This is especially the case when you are new to K8S, and are overwhelmed with configuration options, deployment manifests, networking, and how containers work. Fortunately Kubernetes has matured quickly, and there are tons of opensource tools to troubleshoot and monitor your clusters. One of these tools, Popeye, is a must for any Kubernetes operator. Popeye will evaulate your clusters against best practices, and display warnings if it finds issues.

Getting going with Popeye is a breeze. If you have Krew installed, you can install the plug-in with the following command:

$ kubectl krew install popeye

To audit a cluster, you can pass the “popeye” option to kubectl:

$ kubectl popeye

This will produce a comprehensive report similar to the following:

 ___     ___ _____   _____                                                      K          .-'-.     
| _ \___| _ \ __\ \ / / __|                                                      8     __|      `\  
|  _/ _ \  _/ _| \ V /| _|                                                        s   `-,-`--._   `\
|_| \___/_| |___| |_| |___|                                                      []  .->'  a     `|-'
  Biffs`em and Buffs`em!                                                          `=/ (__/_       /  
                                                                                    \_,    `    _)  
                                                                                       `----;  |     


GENERAL [KIND-TEST]
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Connectivity...................................................................................✅
  · MetricServer...................................................................................💥


CLUSTERS (1 SCANNED)                                                         💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Version........................................................................................✅
    ✅ [POP-406] K8s version OK.


CLUSTERROLES (60 SCANNED)                                                   💥 0 😱 0 🔊 60 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · admin..........................................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · cluster-admin..................................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · edit...........................................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · kindnet........................................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · kubeadm:get-nodes..............................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · local-path-provisioner-role....................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:aggregate-to-admin......................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:aggregate-to-edit.......................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:aggregate-to-view.......................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:auth-delegator..........................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:basic-user..............................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:certificates.k8s.io:certificatesigningrequests:nodeclient...............................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:certificates.k8s.io:certificatesigningrequests:selfnodeclient...........................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:certificates.k8s.io:kube-apiserver-client-approver......................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:certificates.k8s.io:kube-apiserver-client-kubelet-approver..............................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:certificates.k8s.io:kubelet-serving-approver............................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:certificates.k8s.io:legacy-unknown-approver.............................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:attachdetach-controller......................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:certificate-controller.......................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:clusterrole-aggregation-controller...........................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:cronjob-controller...........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:daemon-set-controller........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:deployment-controller........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:disruption-controller........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:endpoint-controller..........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:endpointslice-controller.....................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:expand-controller............................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:generic-garbage-collector....................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:horizontal-pod-autoscaler....................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:job-controller...............................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:namespace-controller.........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:node-controller..............................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:persistent-volume-binder.....................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:pod-garbage-collector........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:pv-protection-controller.....................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:pvc-protection-controller....................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:replicaset-controller........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:replication-controller.......................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:resourcequota-controller.....................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:route-controller.............................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:service-account-controller...................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:service-controller...........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:statefulset-controller.......................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:controller:ttl-controller...............................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:coredns.................................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:discovery...............................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:heapster................................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:kube-aggregator.........................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:kube-controller-manager.................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:kube-dns................................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:kube-scheduler..........................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:kubelet-api-admin.......................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:node....................................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:node-bootstrapper.......................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:node-problem-detector...................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:node-proxier............................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:persistent-volume-provisioner...........................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:public-info-viewer......................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · system:volume-scheduler........................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.
  · view...........................................................................................🔊
    🔊 [POP-400] Used? Unable to locate resource reference.


CLUSTERROLEBINDING                                                           💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


CONFIGMAP                                                                    💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


DAEMONSET                                                                    💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


DEPLOYMENT                                                                   💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


HORIZONTALPODAUTOSCALER                                                      💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


INGRESS                                                                      💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


NAMESPACES (1 SCANNED)                                                       💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · default........................................................................................✅


NETWORKPOLICY                                                                💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


PERSISTENTVOLUME                                                             💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


PERSISTENTVOLUMECLAIM                                                        💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


PODS (1 SCANNED)                                                               💥 1 😱 0 🔊 0 ✅ 0 0٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · default/centos.................................................................................💥
    🔊 [POP-206] No PodDisruptionBudget defined.
    😱 [POP-300] Using "default" ServiceAccount.
    😱 [POP-301] Connects to API Server? ServiceAccount token is mounted.
    😱 [POP-302] Pod could be running as root user. Check SecurityContext/image.
    🐳 centos
      💥 [POP-100] Untagged docker image in use.
      😱 [POP-106] No resources requests/limits defined.
      😱 [POP-102] No probes defined.
      😱 [POP-306] Container could be running as root user. Check SecurityContext/Image.


PODDISRUPTIONBUDGET                                                          💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


PODSECURITYPOLICY                                                            💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


REPLICASET                                                                   💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


ROLE                                                                         💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


ROLEBINDING                                                                  💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


SECRETS (1 SCANNED)                                                          💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · default/default-token-zvwkm....................................................................✅


SERVICES (1 SCANNED)                                                         💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · default/kubernetes.............................................................................✅


SERVICEACCOUNTS (1 SCANNED)                                                  💥 0 😱 0 🔊 0 ✅ 1 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · default/default................................................................................✅


STATEFULSET                                                                  💥 0 😱 0 🔊 0 ✅ 0 100٪
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
  · Nothing to report.


SUMMARY
┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅┅
Your cluster score: 95 -- A
                                                                                o          .-'-.     
                                                                                 o     __| A    `\  
                                                                                  o   `-,-`--._   `\
                                                                                 []  .->'  a     `|-'
                                                                                  `=/ (__/_       /  
                                                                                    \_,    `    _)  
                                                                                       `----;  |     

The official documentation describes the report morphology. These break down into Ok, Info, Warn and Error codes. Whenever I take ownership of an existing cluster, or help friends debug issues, Popeye and kubeaudit are run to help me understand where the cluster stands. Popeye also has a number of options to control the output that is produced:

-o, --out string  Specify the output type (standard, jurassic, yaml, json, html, junit, prometheus, score) (default "standard")

This makes it super easy to add Popeye to a deployment pipeline, security dashboard, or just about anything you can think of. While Popeye won’t produce a delectable burger for Wimpy, it will help you understand issues in your cluster!

Auditing your Kubernetes clusters security posture with kubeaudit

This article was posted by on 2020-05-12 02:00:00 -0500 -0500

In this day and age of security breaches, system compromises, and critical CVEs coming out almost daily, security best practices should go into everything we do. There is an almost limitless number of tools for auditing, securing, and reporting on system and network security issues. One of these tools, Kubeaudit, allows you to audit the security posture of your Kubernetes clusters. Kubeaudit ships with a numerous audit rules, which cover the major Kubernetes security best practices.

To get started with kubeadit, you will need to download a binary from the projects Github release page:

$ curl -L0 -o kubeaudit.tgz https://github.com/Shopify/kubeaudit/releases/download/v0.7.0/kubeaudit_0.7.0_linux_amd64.tar.gz

$ tar xfvz kubeaudit*.gz

Once downloaded, you can run it without any arguments to see the available options:

$ kubeaudit

Available Commands:
  all         Run all audits
  allowpe     Audit containers that allow privilege escalation
  apparmor    Audit containers running without AppArmor
  autofix     Automagically fixes a manifest to be secure
  caps        Audit container for capabilities
  help        Help about any command
  image       Audit container images
  limits      Audit containers running with limits
  mountds     Audit containers that mount /var/run/docker.sock
  namespaces  Audit Pods for hostNetwork, hostIPC and hostPID
  nonroot     Audit containers running as root
  np          Audit namespace network policies
  priv        Audit containers running as privileged
  rootfs      Audit containers with read only root filesystems
  sat         Audit automountServiceAccountToken = true pods against an empty (default) service account
  seccomp     Audit containers running without Seccomp
  version     Print the version number of kubeaudit

As you can see, there are options to audit pretty much every aspect of your cluster. Are image tags defined? Yup! Capabilities? For sure! Containers running without security contexts set? Indeed! To start an audit, pick the item you want to audit, and pass it as an argument to kubeaudit. The following example uses the “all” command to audit everything, and limits the audit to the default namespace:

$ kubeaudit all -n default

INFO[0000] Not running inside cluster, using local config 
ERRO[0000] AllowPrivilegeEscalation not set which allows privilege escalation, please set to false  Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] ReadOnlyRootFilesystem not set which results in a writable rootFS, please set to true  Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] RunAsNonRoot is not set in ContainerSecurityContext, which results in root user being allowed!  Container=centos KubeType=pod Name=centos Namespace=default
WARN[0000] serviceAccount is a deprecated alias for ServiceAccountName, use that one instead  DSA=default KubeType=pod Name=centos Namespace=default SA=default
WARN[0000] Privileged defaults to false, which results in non privileged, which is okay.  Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=AUDIT_WRITE Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=CHOWN Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=DAC_OVERRIDE Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=FOWNER Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=FSETID Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=KILL Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=MKNOD Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=NET_BIND_SERVICE Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=NET_RAW Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=SETFCAP Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=SETGID Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=SETPCAP Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=SETUID Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped                        CapName=SYS_CHROOT Container=centos KubeType=pod Name=centos Namespace=default
WARN[0000] Resource limit not set, please set it!        KubeType=pod Name=centos Namespace=default
WARN[0000] Image tag was missing                         Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] AppArmor annotation missing.                  Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Seccomp annotation missing.                   Container=centos KubeType=pod Name=centos Namespace=default

Your head may spin the first time you run kubeaudit. I know mine did when I ran it with the all option, and against all namespaces. But fear not, a lot of the output is repetitive, and can be addressed with simple changes to your deployment manifests. Setting resource requests and limits, dropping unneeded capabilities, using security contexts, etc. Kubeaudit also has an “autofix” command, which can update a manifest to comply with security best practices:

$ kubectl run centos --image=centos -o yaml --dry-run > centos.yaml

$ copy centos.yaml centos-autofix.yaml

$ kubeaudit autofix -f centos-autofix.yaml

One the manifest is updated, you can use your favorite diff utility to see the changes kubeaudit made:

$ sdiff centos.yaml centos-autofix.yaml | more

                    > ---
apiVersion: v1              apiVersion: v1
kind: Pod             kind: Pod
metadata:             metadata:
  creationTimestamp: null           creationTimestamp: null
  labels:               labels:
    run: centos1                run: centos1
  name: centos1               name: centos1
                    >   annotations:
                    >     container.apparmor.security.beta.kubernetes.io/centos1: r
                    >     seccomp.security.alpha.kubernetes.io/pod: runtime/default
spec:               spec:
  containers:               containers:
  - image: centos             - image: centos
    name: centos1               name: centos1
    resources: {}               resources: {}
                    >     securityContext:
                    >       allowPrivilegeEscalation: false
                    >       capabilities:
                    >         drop:
                    >         - AUDIT_WRITE
                    >         - CHOWN

I’ve addressed the vast majority of these findings by creating templates that comply with security best practices. These templates are then used when I create new deployments, and verified when new manifests are checked into version control. This is a super useful utility, and will definitely take your Kubernetes security posture to 11.