I’ve been spending a good amount of my spare time trying to learn the ins and outs of kubernetes and terraform. To really get the gist of how Kubernetes works under the covers I’ve been automating Kubernetes the hard way with terraform and ansible. There are a a couple of dependencies in the Kubernetes world. One dependency is the control plane’s reliance on etcd. After configuring and starting my etcd cluster I wanted to check the cluster health before moving forward. You can retrieve the health status of an etcd node with the endpoint health option:
etcdctl endpoint health
http://127.0.0.1:2379 is healthy: successfully committed proposal: took = 651.381µs
Ansible provides a really cool feature to assist with these situations: the do-until loop. The do-until loop allows you to run a command a fixed number of times (the retries parameter contains the #) and continue once the until criteria is met. In my case I had ansible check for ‘is healthy’ in the stdout:
--- - hosts: kubcontrollers tasks: - shell: etcdctl --endpoints=[http://127.0.0.1:2379] endpoint health register: result until: result.stdout.find("is healthy") != -1 retries: 5 delay: 10
I’ve read through a few playbooks that use this to accomplish rolling restarts and upgrades. Nifty feature!