In this day and age of security breaches, system compromises, and critical CVEs coming out almost daily, security best practices should go into everything we do. There is an almost limitless number of tools for auditing, securing, and reporting on system and network security issues. One of these tools, Kubeaudit, allows you to audit the security posture of your Kubernetes clusters. Kubeaudit ships with a numerous audit rules, which cover the major Kubernetes security best practices.
To get started with kubeadit, you will need to download a binary from the projects Github release page:
$ curl -L0 -o kubeaudit.tgz https://github.com/Shopify/kubeaudit/releases/download/v0.7.0/kubeaudit_0.7.0_linux_amd64.tar.gz
$ tar xfvz kubeaudit*.gz
Once downloaded, you can run it without any arguments to see the available options:
$ kubeaudit
Available Commands:
all Run all audits
allowpe Audit containers that allow privilege escalation
apparmor Audit containers running without AppArmor
autofix Automagically fixes a manifest to be secure
caps Audit container for capabilities
help Help about any command
image Audit container images
limits Audit containers running with limits
mountds Audit containers that mount /var/run/docker.sock
namespaces Audit Pods for hostNetwork, hostIPC and hostPID
nonroot Audit containers running as root
np Audit namespace network policies
priv Audit containers running as privileged
rootfs Audit containers with read only root filesystems
sat Audit automountServiceAccountToken = true pods against an empty (default) service account
seccomp Audit containers running without Seccomp
version Print the version number of kubeaudit
As you can see, there are options to audit pretty much every aspect of your cluster. Are image tags defined? Yup! Capabilities? For sure! Containers running without security contexts set? Indeed! To start an audit, pick the item you want to audit, and pass it as an argument to kubeaudit. The following example uses the “all” command to audit everything, and limits the audit to the default namespace:
$ kubeaudit all -n default
INFO[0000] Not running inside cluster, using local config
ERRO[0000] AllowPrivilegeEscalation not set which allows privilege escalation, please set to false Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] ReadOnlyRootFilesystem not set which results in a writable rootFS, please set to true Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] RunAsNonRoot is not set in ContainerSecurityContext, which results in root user being allowed! Container=centos KubeType=pod Name=centos Namespace=default
WARN[0000] serviceAccount is a deprecated alias for ServiceAccountName, use that one instead DSA=default KubeType=pod Name=centos Namespace=default SA=default
WARN[0000] Privileged defaults to false, which results in non privileged, which is okay. Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=AUDIT_WRITE Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=CHOWN Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=DAC_OVERRIDE Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=FOWNER Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=FSETID Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=KILL Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=MKNOD Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=NET_BIND_SERVICE Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=NET_RAW Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=SETFCAP Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=SETGID Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=SETPCAP Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=SETUID Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Capability not dropped CapName=SYS_CHROOT Container=centos KubeType=pod Name=centos Namespace=default
WARN[0000] Resource limit not set, please set it! KubeType=pod Name=centos Namespace=default
WARN[0000] Image tag was missing Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] AppArmor annotation missing. Container=centos KubeType=pod Name=centos Namespace=default
ERRO[0000] Seccomp annotation missing. Container=centos KubeType=pod Name=centos Namespace=default
Your head may spin the first time you run kubeaudit. I know mine did when I ran it with the all option, and against all namespaces. But fear not, a lot of the output is repetitive, and can be addressed with simple changes to your deployment manifests. Setting resource requests and limits, dropping unneeded capabilities, using security contexts, etc. Kubeaudit also has an “autofix” command, which can update a manifest to comply with security best practices:
$ kubectl run centos --image=centos -o yaml --dry-run > centos.yaml
$ copy centos.yaml centos-autofix.yaml
$ kubeaudit autofix -f centos-autofix.yaml
One the manifest is updated, you can use your favorite diff utility to see the changes kubeaudit made:
$ sdiff centos.yaml centos-autofix.yaml | more
> ---
apiVersion: v1 apiVersion: v1
kind: Pod kind: Pod
metadata: metadata:
creationTimestamp: null creationTimestamp: null
labels: labels:
run: centos1 run: centos1
name: centos1 name: centos1
> annotations:
> container.apparmor.security.beta.kubernetes.io/centos1: r
> seccomp.security.alpha.kubernetes.io/pod: runtime/default
spec: spec:
containers: containers:
- image: centos - image: centos
name: centos1 name: centos1
resources: {} resources: {}
> securityContext:
> allowPrivilegeEscalation: false
> capabilities:
> drop:
> - AUDIT_WRITE
> - CHOWN
I’ve addressed the vast majority of these findings by creating templates that comply with security best practices. These templates are then used when I create new deployments, and verified when new manifests are checked into version control. This is a super useful utility, and will definitely take your Kubernetes security posture to 11.