This article was posted by Matty on 2018-02-08 18:00:00 -0500 -0500
Over the past few months I’ve been trying to learn everything there is to know about Kubernetes. Kubernetes is an amazing technology for deploying and scaling containers though it comes with a cost. It’s an incredibly complex piece of software and there are a ton of bells and whistles to become familiar with. One way that I’ve found for coming up to speed is Joe Beda’s weekly TGIK live broadcast. This occurs each Friday at 4PM EST and is CHOCK full of fantastic information. In episode five Joe talks about pod parameters and probes. You can watch it here:
Here are some of my takeways from the episode:
ReplicaSets are layered on top of pods.
Deployments are layered on top of ReplicaSets.
The kubectl run –generator option can be used to generate specific types of resources at runtime.
–generator=run/v1 creates a replication controller.
–restart=Always creates a deployment.
–restart=OnFailure creates a job.
–schedule= creates a cron job.
Taking the base API object produced by kubectl get pod NAME -o yaml won’t work out of the box due to system-specific information being present (e.g., resourceVersion).
The export option provides a better way to create a base YAML file to work off of (still pretty chatty):
$ kubectl get pods ubuntu -o yaml --export
To get the base object kubectl sends to the API server you can use:
$ kubectl run ubuntu --image=ubuntu -o yaml --dry-run
The kubectl “-w” option allows you to watch for changes to a resource:
$ kubectl get pods ubuntu -w -o wide
The restart policy controls what to do when a pod fails. Three options are available though not all resource support these options:
Always
OnFailure
Never
Pods are assigned to a node for their entire lifetime.
Replica sets allow you to define the number of pods you want. Kubernetes will ensure that this number is always available. The replication controller is the Kubernetes entity responsible for managing replica sets.
Kubernetes doesn’t reschedule pods. It recreates them.
If you set RestartPolicy to Never the pod won’t be restarted. But if the pod fails it will get re-scheduled to another node.
apply is a declarative management approach where the changes you applied to a live object are maintained even if you applied other changes to the object.
Horizontal pod autoscaler allows you to automatically scale the number of pods in a replication controller, deployment or replica set based on a set of criteria (e.g., CPU utilization).
Pod format when replica sets are used:
REPLICASET-HASHOFDESCRIPTIONOFREPLICASET-UNIQUEID
Two types of probes:
readiness probes indicate that the resource is ready to receive traffic.
liveness probes indicate that the resource is functioniong correctly and should continue receiving traffic.
This article was posted by Matty on 2018-02-07 20:19:49 -0500 -0500
Over the past few months I’ve been trying to learn everything there is to know about Kubernetes. Kubernetes is an amazing technology for deploying and scaling containers though it comes with a cost. It’s an incredibly complex piece of software and there are a ton of bells and whistles to become familiar with. One way that I’ve found for coming up to speed is Joe Beda’s weekly TGIK live broadcast. This occurs each Friday at 4PM EST and is CHOCK full of fantastic information. In episode four Joe talks about Role Based Access Control (RBAC). You can watch it here:
Here are some of my takeways from the episode:
SSH command line to grab the kubeconfig from the quickstart master for local use (-W host:port can simplify this):
$ scp -i $SSH_KEY -o proxyCommand="ssh -i \${SSH_KEY}" ubuntu@1.2.3.4 nc %h %p" ubuntu@2.3.4.5:~kubeconfig ./kubeconfig
Heptio provides a super awesome RBAC cheat sheet on their website.
Kubernetes configuration file layout:
cluster: contains the cluster to connect to.
certificate-authority-data: contains the RootCA certificate.
server: contains the server to connect to (https://path-to-api-server).
contexts: used to group access parameters under a convenient name
cluster: name of the cluster
namespace: namespace to use
user: username (this is the CN in the X509 certificate)
users: Contains the individual users and their security credentials
name: name of the user
user: attributes for the user (e.g., certificate and private key location)
You can specify the kubeconfig you want to use with the KUBECONFIG environment variable.
Each Kubernetes configuration file can contain multiple clusters and users.
The kubectl “-v” option can be used to review the request/response headers sent between kubectl and the API server.
$ kubectl -v 100 get pods
View your Kubernetes configuration file:
$ kubectl config view
Kubernetes provides an API for signing certificates. To use it you can create a YAML file and base64 encode your CSR and assign it to the “request:” attribute. Then you can use kubectl to sign, approve and d/l your certificate (Jakub’s write up explains the commands further):
$ kubectl create -f cert-request.yaml
$ kubectl certificate approve matty
$ kubectl get csr matty -o jsonpath='(.status.certificate} | base64 -d > matty.crt
Currently no way to revoke certificates.
Joe suggested using one cluster configuration for each cluster.
Authentication defines who can do something. Authorizations defines what you can do.
Authorization modes:
ABAC mode attribute based access control (mostly deprecated)
Webhook allows you to call a third party to control what something can can do
Node authorization limits what individual nodes can do
RBAC uses the authorization API to determine what something can do
RBAC was back ported from openshift.
Four key RBAC objects:
Roles
ClusterRoles
Rolbindings
ClusterRoleBindings
Some objects in kubernetes (csr and nodes for example) don’t live in a specific namespace. These are cluster wide resources.
Retrieve all roles and clusterroles:
$ kubectl get roles --all-namespaces
$ kubectl get clusterroles
Individual services (daemonsets for example) in Kubernetes typically run under their own role.
The cluster-admin role provides super use privileges (use cautiously).
View the details of a specific role:
$ kubectl get clusterroles system:basic-user -o yaml
The verb in a role indicates the actions that can be performed.
You can generate role binding templates with the dry-run option:
$ kubectl create rolebinding matty --clusterrole=admin --user=users:matty --dry-run -o yaml
Namespaces and RBAC are a great combo for isolating users to their own namespace and limiting what they can do in that namespace.
Service accounts are used to connect to the cluster. These can be listed with kubectl:
$ kubectl get sa
If Francis Ford Coppola made a film about Kubernetes Joe would be the kubfather.
Things I need to learn more about:
Learn more about the kubernetes certificate API.
Learn more about how to properly use serviceaccounts.
This article was posted by Matty on 2018-02-07 20:00:00 -0500 -0500
*** UPDATE 05/15/2020 ***
I have since learned about Weave’s footloose project, which provides a VM-like experience, but utilizes containers under the covers. Amazing project!
*** Original Post ***
As I dig deeper into Kubernetes it’s been useful to spin up a pod with one of my favorite Operating Systems and review the configuration for that pod. This is super easy to do with kubectl run’s –image option. To create an Ubuntu VM you can pass the ubuntu image name to “–image”
$ kubectl run -it --rm --image=unbuntu ubuntu -- bash
To create a CentOS pod you can use the centos image:
$ kubectl run -it --rm --image=centos centos -- bash
To create a Fedora pod you can use the fedora image:
$ kubectl run -it --rm --image=fedora fedora -- bash
And to create an alpine pod you can use the alpine image:
$ kubectl run -it --rm --image=alpine alpine -- ash
If you want to run a specific version of an OS you can add the version to the image name:
$ kubectl run -it --rm --image=fedora:26 fedora26 -- bash
Once the POD is up you can start reviewing the various Kubernetes objects that were created to support the pod. You can also use the “–dry-run” and output options to view the object that is created and sent to the API server:
$ kubectl run --image=alpine alpine --dry-run -o yaml
The YAML above can be copied to a file, modified and re-deployed with the kubectl create command. This is a fantastic way to tinker with all of the Kubernetes deployment options.
This article was posted by Matty on 2018-02-06 18:00:00 -0500 -0500
Over the past few months I’ve been trying to learn everything there is to know about Kubernetes. Kubernetes is an amazing technology for deploying and scaling containers though it comes with a cost. It’s an incredibly complex piece of software and there are a ton of bells and whistles to become familiar with. One way that I’ve found for coming up to speed is Joe Beda’s weekly TGIK live broadcast. This occurs each Friday at 4PM EST and is CHOCK full of fantastic information. In episode three Joe talks about istio. You can watch it here:
Here are some of my takeways from the episode:
Allows services in a microservice architecture to talk to each other with some added benefits:
Control the flow of traffic and API calls between services.
Locating other services in the mesh.
Securing services by identifying who is calling them and enforcing policy.
Allows things like distributed tracing to be incorporated without application changes.
Linkerd is built on top of the concepts learned from finagle.
istio runs as a sidecar which lives in the namepsace of your pod.
Sidecars are buddy processes used to give other containers additional capabilities.
Kubernetes has a pluggable authentication an authorization framework.
Zipkin is a distributed tracing system that can be used to better understand a microservices architecture.
You can get the logs from a pod with the logs command:
$ kubectl logs mypod
Istio makes heavy use of the LoadBalancer service. Be careful if running in AWS. These can cost you real $$ if you are using the quickstart and don’t delete them.
Crash loops and back offs are incredibly powerful patterns.
Third party resource allow you to create new things that are application specific. Kubernetes can be used to store the configuration and objects created by TPR.
Third party resources (TPR) are going away. CRDs will replace them.
A RoleBinding grants the permissions defined in a role to a user or set of users in a namespace.
A ClusterRoleBinding grants the permissions defined in a role to a user or set of users cluster wide (be careful with these).
Istio pods created during the installation process:
This article was posted by Matty on 2018-02-04 18:43:54 -0500 -0500
Over the past few months I’ve been trying to learn everything there is to know about Kubernetes. Kubernetes is an amazing technology for deploying and scaling containers though it comes with a cost. It’s an incredibly complex piece of software and there are a ton of bells and whistles to become familiar with. One way that I’ve found for coming up to speed is Joe Beda’s weekly TGIK live broadcast. This occurs each Friday at 4PM EST and is CHOCK full of fantastic information. In episode two Joe goes into Kubernetes networking. You can watch it here:
Here are some of my takeways from the episode:
Each pod is assigned a unique IP.
Assigning a unique IP per POD helps overcome the challenges associated with sharing ports on a machine.
Communication takes three forms:
Pod to pod
Pod to host
Host to pod
Outbound traffic from the cluster to external entities works out of the box via NAT.
Inbound connectivity to pods requires some extra plumbing.
Kubernetes has built-in service discovery via the endpoints API.
CNI provides specifications and libraries to configure network interfaces in Linux containers
CNI IPAM is how IPs (POD CIDR) are managed on the host.
Services and deployments utilize labels to coordinate their efforts.
Deployments use the “selector:” to pick the containers that the service will point to.
Multiple types of services (5 types of objects):
None
ClusterIP - exposes the service on a cluster-internal IP which is only accessible from the cluster.
NodePort - expose a port on the node that forwards to the service.
LoadBalancer - creates an external load-balancer and configures it to point to the NodePort on each worker node.
ExternalName - provides a way to return an alias to an external service residing outside the cluster.
ClusterIPs are picked from the service network and kube-proxy maps the service IP to a set of endpoints with iptables.
ClusterIPs live for the life of a service. They don’t change.
DNS names take the following format (CLUSTER_DOMAIN is configured with –cluster-domain):
NAME.SVC.NAMESPACE.CLUSTER_DOMAIN
Services can be exposed with the kubectl expose command:
$ kubectl expose deployment kuard --type=ClusterIP --target-port=8080 --port 80
service “kuard” exposed
Things I need to learn more about:
Learn more about the endpoint service discovery API.
Learn more about the CNI API.
Study iptables-save to see how pods and services are stitched together.