I've been digging into Kubernetes persistent volumes and storage classes and wanted to create a new cluster to storage my volume claims. Ansible version 1.9 ships with the gluster_volume module which makes creating a new cluster crazy easy. The module is written in Python and you can find it in /usr/lib/python2.7/site-packages/ansible/modules/system/gluster_volume.py on Fedora-derived distributions. To create a new cluster you need to create a new file system and then define a gluster_volume task to actually create the cluster:
- name: create gluster volume
gluster_volume:
state: present
name: "{{ gluster_volume_name }}"
bricks: "{{ gluster_filesystem/brick_name }}"
rebalance: no
start_on_create: yes
replicas: "{{ gluster_replica_count }}"
transport: tcp
options:
auth.allow: "{{ gluster_cal }}"
cluster: "{{ groups.gluster | join(',') }}"
host: "{{ inventory_hostname }}"
run_once: true
Jeff Geerling describes how to build a cluster in gory detail and I would definitely recommend checking that out if you want to learn more about the module options listed above (his Ansible for DevOps book is also a must have). Gluster is a complex piece of software and I wanted to learn more about what was happening under the covers. While I could have spent a couple of hours reading the source code to see how the module worked I decided to fast track my learning with execsnoop. Prior to running my playbook I ran the bcc execsnoop utility and grep'ed for the string gluster:
$ execsnoop | grep cluster
chmod 8851 8844 0 /usr/bin/chmod u+x
/root/.ansible/tmp/ansible-tmp-1519562230.06-193658562566672//root/.ansible/tmp/
ansible-tmp-1519562230.06-193658562566672/gluster_volume.py
bash 8852 6539 0 /bin/bash -c /bin/sh -c '/usr/bin/python
/root/.ansible/tmp/ansible-tmp-1519562230.06-193658562566672/gluster_volume.py;
rm -rf "/root/.ansib
sh 8852 6539 0 /bin/sh -c /usr/bin/python
/root/.ansible/tmp/ansible-tmp-1519562230.06-193658562566672/gluster_volume.py;
rm -rf "/root/.ansible/tmp/ansib
python 8865 8852 0 /usr/bin/python
/root/.ansible/tmp/ansible-tmp-1519562230.06-193658562566672/gluster_volume.py
python 8866 8865 0 /usr/bin/python
/tmp/ansible_2rTonm/ansible_module_gluster_volume.py
gluster 8867 8866 0 /usr/sbin/gluster --mode=script peer status
gluster 8873 8866 0 /usr/sbin/gluster --mode=script volume info
gluster 8879 8866 0 /usr/sbin/gluster --mode=script peer probe
gluster01.prefetch.net
gluster 8885 8866 0 /usr/sbin/gluster --mode=script volume
create kubernetes-volume1 replica 3 transport tcp
gluster01.prefetch.net:/bits/kub1 gluster02.prefetch.net:/bits/kub1
gluster03.prefetch.net:/bits/kub1
S10selinux-labe 8891 6387 0
/var/lib/glusterd/hooks/1/create/post/S10selinux-label-brick.sh
--volname=kubernetes-volume1
gluster 8892 8866 0 /usr/sbin/gluster --mode=script volume info
gluster 8898 8866 0 /usr/sbin/gluster --mode=script volume start
kubernetes-volume1
glusterfsd 8904 6387 0 /usr/sbin/glusterfsd -s
gluster01.prefetch.net --volfile-id kubernetes-volume1.
gluster01.prefetch.net.bits-kub1 -p
/var/run/gluster/vols/kubernetes-volume1/gluster01.prefetch.net-bits-kub1.pid
-S /var/run/gluster/e2eaf5fee7c4e7d6647dc6ef94f8a342.socket --brick-name
/bits/kub1 -l /var/log/glusterfs/bricks/bits-kub1.log --xlator-option
*-posix.glusterd-uuid=6aa04dc6-b578-46a5-9b9e-4394c431ce42 --brick-port 49152
--xlator -option kubernetes-volume1-server.listen-port=49152
glusterfs 8925 8924 0 /usr/sbin/glusterfs -s localhost
--volfile-id gluster/glustershd -p /var/run/gluster/glustershd/glustershd.pid
-l /var/log/glusterfs/glustershd.log -S
/var/run/gluster/6eb5edea939911f91690c1181fb07f5d.socket --xlator-option
*replicate*.node-uuid=6aa04dc6-b578-46a5-9b9e-4394c431ce42
S29CTDBsetup.sh 8935 6387 0
/var/lib/glusterd/hooks/1/start/post/S29CTDBsetup.sh
--volname=kubernetes-volume1 --first=yes --version=1 --volume-op=start
--gd-workdir=/var/lib/glusterd
getopt 8937 8935 0 /usr/bin/getopt -l volname: -name ctdb
--volname=kubernetes-volume1 --first=yes --version=1 --volume-op=start
--gd-workdir=/var/lib/glusterd
S30samba-start. 8938 6387 0
/var/lib/glusterd/hooks/1/start/post/S30samba-start.sh
--volname=kubernetes-volume1 --first=yes --version=1 --volume-op=start
--gd-workdir=/var/lib/glusterd
getopt 8939 8938 0 /usr/bin/getopt -l volname:,gd-workdir:
-name Ssamba-start --volname=kubernetes- volume1 --first=yes --version=1
--volume-op=start --gd-workdir=/var/lib/glusterd
grep 8945 8944 0 /usr/bin/grep user.smb
/var/lib/glusterd/vols/kubernetes-volume1/info
gluster 8947 8866 0 /usr/sbin/gluster --mode=script volume set
kubernetes-volume1 auth.allow 192.168.2.*
S30samba-set.sh 8956 6387 0
/var/lib/glusterd/hooks/1/set/post/S30samba-set.sh --volname=kubernetes-volume1
-o auth.allow=192.168.2.* --gd-workdir=/var/lib/glusterd
getopt 8959 8956 0 /usr/bin/getopt -l volname:,gd-workdir:
--name Ssamba-set -o o: -- --volname=kubernetes-volume1 -o
auth.allow=192.168.2.* --gd-workdir=/var/lib/glusterd
grep 8965 8964 0 /usr/bin/grep status
/var/lib/glusterd/vols/kubernetes-volume1/info
S32gluster_enab 8967 6387 0
/var/lib/glusterd/hooks/1/set/post/S32gluster_enable_shared_storage.sh
--volname=kubernetes-volume1 -o auth.allow=192.168.2.*
--gd-workdir=/var/lib/glusterd
gluster 8974 8866 0 /usr/sbin/gluster --mode=script volume info
The first five lines are used to setup up the ansiballz module and the remaining lines contain the commands ansible runs to create the cluster on one of my gluster noes. While I love reading code sometimes you just don't have time and this was one of them.