Updating a file with ansible if the current file is X days old

I’m a heavy user of the Logstash geoip features which utilize the GeoLite database. To keep up to date with the latest mappings I updated my logstash ansible role to check the current database and retrieve a new one if its older than a certain number of days. This was super easy to do with ansible. To get started I defined a couple of variables in group_vars:

geoip_directory: "/elk/logstash/geoip"
geoip_source: http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.mmdb.gz
geoip_upgrade_days: 30

These variables define the location to put the geoip database, the URL to the latest database and how how often to update the file. To check if a file is outdated I used the stat module’s mtime attribute along with a when conditional:

- name: Get the GeoIP database file name
  set_fact: geoip_compressed_file_name="{{ geoip_source | basename }}"

- name: Get the GeoIP database file name
  set_fact: geoip_uncompressed_file_name="{{ geoip_compressed_file_name | replace('.gz', '') }}"

- name: Retrieving file stat data from {{ geoip_directory }}/{{ geoip_uncompressed_file_name }}"
  stat:
    path: "{{ geoip_directory }}/{{ geoip_uncompressed_file_name }}"
  register: stat_results

- name: "Download the latest GeoIP database from {{ geoip_source }}"
  get_url:
    url: "{{ geoip_source }}"
    dest: "{{ geoip_directory }}"
    mode: 0600
  when: ((ansible_date_time.epoch|int - stat_results.stat.mtime) > (geoip_upgrade_days * 60 * 60 * 24))
  register: downloaded_geoip_file

- name: "Uncompressing the GeoIP file {{ geoip_directory }}/{{ geoip_compressed_file_name }}"
  shell: gunzip -f "{{ geoip_directory }}/{{ geoip_compressed_file_name }}"
  when: downloaded_geoip_file.changed

I still need to add a couple of checks to deal with edge conditions but this is definitely a step up from what I was doing previously. Viva la ansible!

Using Ansible to verify remote file checksums with get_url, lookup() and stat

Being an extremely security minded operations guy I take every precaution to verify that the files I download are legit. In this day and age of servers and data getting compromised this should be an operational standard. There are numerous ways to verify checksums. You can use openssl’s various hashing options or a simple wrapper script similar to this. I prefer to automate everything so I typically offload these types of tasks to ansible and chef. Both configuration management systems give you a number of ways to tackle this and I thought I would discuss a couple of my favorite ansible recipes in this blog post.

To illustrate how easy this process is with ansible lets say you are setting up an elasticsearch cluster and want to keep up to date with the latest GEO IP database from maxmind.net. To retrieve the database and verify the checksum you can use the ansible get_url module with the checksum parameter and lookup filter:

---
- hosts: localhost
  connection: local
  vars:
    geoip_dir: "/tmp/geoip"
    geoip_db_url: "http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz"
    geoip_db_md5sum_url: "http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz.md5"
    geoip_db_compressed_file_name: "{{ geoip_db_url | basename }}"
    geoip_db_md5sum: "md5: {{ lookup('url', geoip_db_md5sum_url) }}"
  gather_facts: false
  tasks:
     - name: Create geoip directory if it doesn't exist
       file:
         path: "{{ geoip_dir }}"
         state: directory
         mode: 0700

     - name: "Downloading the latest GeoIP and comparing it to checksum {{ geoip_db_md5sum }}"
       get_url:
         url: "{{ geoip_db_url }}"
         dest: "{{ geoip_dir }}/{{ geoip_db_compressed_file_name }}"
         mode: 0600
         checksum: "{{ geoip_db_md5sum }}"

In the example above the lookup() filter will retrieve the MD5SUM from a remote file and assign that to the checksum parameter passed to get_url. If the remote file checkum matches the value of geoip_db_md5sum the file will be downloaded to the directory specified in the dest parameter. This is useful to show how versatile ansible is but the security conscious admin should be bugging out about retrieving a payload and checksum from the same server. Taking this a step further lets say you retrieved the checksum from a secure source and assigned it to the variable geoip_db_md5sum. This variable can then be referenced by the get_url checksum parameter:

---
- hosts: localhost
  connection: local
  vars:
    geoip_dir: "/tmp/geoip"
    geoip_db_url: "http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz"
    geoip_db_md5sum_url: "http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz.md5"
    geoip_db_compressed_file_name: "{{ geoip_db_url | basename }}"
    geoip_db_uncompressed_file_name: "{{ geoip_db_url | basename | replace('.tar.gz','')}}"
    geoip_db_md5sum: "md5: ca82582c02c4a4e57ec9d23a97adaa72"
  gather_facts: false
  tasks:
     - name: Create geoip directory if it doesn't exist
       file:
         path: "{{ geoip_dir }}"
         state: directory
         mode: 0700

     - name: "Downloading the latest GeoIP file"
       get_url:
         url: "{{ geoip_db_url }}"
         dest: "{{ geoip_dir }}/{{ geoip_db_compressed_file_name }}"
         checksum: "{{ geoip_db_md5sum }}"
         mode: 0600

Simple, elegant, but wait, there’s more! Ansible also has a stat module which you can use to retrieve the checksum of a file. This can be combined with get_url to achieve the same result (this isn’t the ideal way to solve this problem but is included to show the power of ansible):

---
- hosts: localhost
  connection: local
  vars:
    geoip_dir: "/tmp/geoip"
    geoip_db_url: "http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz"
    geoip_db_md5sum_url: "http://geolite.maxmind.com/download/geoip/database/GeoLite2-City.tar.gz.md5"
    geoip_db_compressed_file_name: "{{ geoip_db_url | basename }}"
    geoip_db_uncompressed_file_name: "{{ geoip_db_url | basename | replace('.tar.gz','')}}"
    geoip_db_md5sum: "ca82582c02c4a4e57ec9d23a97adaa72"
  gather_facts: false
  tasks:
     - name: Create geoip directory if it doesn't exist
       file:
         path: "{{ geoip_dir }}"
         state: directory
         mode: 0700

     - name: "Downloading the latest GeoIP file"
       get_url:
         url: "{{ geoip_db_url }}"
         dest: "{{ geoip_dir }}/{{ geoip_db_compressed_file_name }}"
         mode: 0600

     - name: "Checking {{ geoip_db_compressed_file_name }} against checksum {{ geoip_db_md5sum }}"
       stat:
          path: "{{ geoip_dir }}/{{ geoip_db_compressed_file_name }}"
       register: st

     - name: extract the geo IP database
       unarchive:
         src: "{{ geoip_dir }}/{{ geoip_db_compressed_file_name }}"
         dest: "{{ geoip_dir }}"
       when: geoip_db_md5sum == st.stat.md5

In the example above the GEO IP database is downloaded and then the stat module is called to retrieve a variety of file metadata. One piece of metadata is the md5 parameter which contains an MD5 hash of the file passed as an argument to path. This value is compared against the value stored in geoip_db_md5sum and if the comparison succeeds the module (unarchive in this case) will run. Ansible supports a number of hash algorithms (SHA256, SHA512, etc.) which should be used in place of MD5 if you have the ability to do so. Gotta loves you some Ansible!

Retrieving a file name from a URL with ansible

Ansible has several extremely powerful modules for interacting with web resources. The get_url module allows you to retrieve resources, uri allows you to interact with web services and the templating and filtering capabilities provided by Jinja2 allow you to slice and dice the results in a variety of ways. One pattern that I find useful is applying the basename filter to the URL to grab a file name:

---
- hosts: 127.0.0.1
  connection: local
  vars:
    url: "http://foo.com/a/b/c/filename.tar.gz"
    file_name: "{{ url | basename }}"
  gather_facts: false
  tasks:
     - name: "Grab the file name from {{ url }}"
       debug:
         msg: "Base name is {{ file_name }}"

After the filter is applied file_name will contain the last component in the “/” separated URL:

$ ansible-playbook get-basename.yml

PLAY [127.0.0.1] **************************************************************************************************

TASK [Grab the file name from http://foo.com/a/b/c/filename.tar.gz] ***********************************************
ok: [127.0.0.1] => {
    "msg": "Base name is filename.tar.gz"
}

PLAY RECAP ********************************************************************************************************
127.0.0.1                  : ok=1    changed=0    unreachable=0    failed=0

I’ll share some valid uses for this in a future blog post.

Automating cron jobs with the ansible cron module

Over the past month I have been rewriting some cron scripts to enhance monitoring and observability. I’ve also been refactoring my ansible
playbooks to handle deploying these scripts in a consistent fashion. Ansible ships with the cron module which makes this process a breeze.
The cron module has all of the familiar cron attributes (hour, minute, second, program to run, etc.) and takes the following form:

- name: Cron job to prune old elasticsearch indexes
  cron:
    name: cleanup-elasticsearch-indexes
    minute: 0
    hour: 0
    job: /scripts/curator/curator_clean_logs.sh
    state: present
    user: curator

When I first played around with this module I noticed that each playbook run would result in a cron entry being added. So instead of getting one curator log cleanup job when the play is executed I would get a one entry per run. This is obviously very bad. When I read back through the cron module documentation I came across this little nugget for the “name” parameter:

Note that if name is not set and state=present, then a new crontab entry will always be created, regardless of existing ones.

Ansible uses the name to tag the entry and if the tag already exists a new cron job won’t be added to the system (in case your interested this is implemented by the find_job() method in cron.py). Small subtleties like this really bring to light the importance of a robust test environment. I am currently using vagrant to solve this problem but there are also a number of solutions documented in the Ansible testing strategies guide.

Debugging ansible playbooks, plays and tasks

I am a long time ansible user and have wrangled it into automating just about everything I do. As my roles and playbooks have increased in quantity and size I’ve found it’s essential to have a good grasp of the debugging capabilities built into ansible. These are useful for detecting syntax errors, finding ordering issues and most importantly for learning how ansible works under the covers. In this post I’m going to cover a number of methods to test playbooks and troubleshoot issues when they pop up. In a future post I’ll go into some tips and tricks I’ve learned while developing new ansible modules. Here are some of my favorites:

1. Checking the syntax of your playbooks to find YAML problems

One of the first things I encountered while developing roles were YAML formatting subtleties. A space missing here, a hyphen missing there or a filter not returning the correct results. They all add up to a pissed off ansible-playbook run! I use atom for editing my playbooks though sometimes I’ve been known to fire up vim when I need to test something quickly from an SSH session. Ansible has a “–syntax-check” option which can be used to make sure your YAML is properly structured:

$ ansible-playbook --syntax-check sshd-disable-root-login.yml
ERROR! 'tsks' is not a valid attribute for a Play

The error appears to have been in
'/ansible/playbooks/sshd-disable-root-login.yml': line 2, column 3,
but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

---
- hosts: localhost
  ^ here

The ansible check error output is extremely verbose and 99% of the time you will read the error and say doh!

2. Testing playbooks before they are committed to a git repository

Like most admins I use git repos for everything I touch. This allows me to see what changed, when it changed, and provides an easy way to share changes with my colleagues. Git provides a set of hooks which you can use to run commands at various stages of the commit process. You can read about hooks here. To ensure that my playbooks are syntactically correct prior to check-in I like to use a pre-commit hook to check syntax:

$ git commit -a -m "Added debug statement to playbook disable root login"
ERROR: Found a syntax error with ./playbooks/sshd-disable-root-login.yml
ERROR: Please run ansible-playbook --syntax-check
./playbooks/sshd-disable-root-login.yml to view the error

If I snarbled something git won’t let me commit the change and I get a simple error message that points me to the problematic file. This goes hand in-hand with a CI system to test your playbooks when they are committed.

3. Getting verbose output during playbook runs

Periodically issues arise and you want to see what ansible is doing when a playbook is applied. Running ansible with multiple verbose flags provides a significant amount of detail:

$ ansible-playbook -vvv playbooks/sshd-disable-root-login.yml
Using /ansible/ansible.cfg as config file

PLAYBOOK: sshd-disable-root-login.yml ******************************************
1 plays in playbooks/sshd-disable-root-login.yml

PLAY [localhost] ***************************************************************

TASK [Disable SSH root logins] *************************************************
task path: /ansible/playbooks/sshd-disable-root-login.yml:6
Using module file
/usr/lib/python2.7/site-packages/ansible/modules/core/files/lineinfile.py
<127.0.0.1> ESTABLISH LOCAL CONNECTION FOR USER: ansible
<127.0.0.1> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo
~/.ansible/tmp/ansible-tmp-1485817022.67-136944116724824 `" && echo
ansible-tmp-1485817022.67-136944116724824="` echo
~/.ansible/tmp/ansible-tmp-1485817022.67-136944116724824 `" ) && sleep
0'
<127.0.0.1> PUT /tmp/tmp2jJjuc TO
/home/ansible/.ansible/tmp/ansible-tmp-1485817022.67-136944116724824/lineinfile.py
<127.0.0.1> EXEC /bin/sh -c 'chmod u+x
/home/ansible/.ansible/tmp/ansible-tmp-1485817022.67-136944116724824/
/home/ansible/.ansible/tmp/ansible-tmp-1485817022.67-136944116724824/lineinfile.py
&& sleep 0'
<127.0.0.1> EXEC /bin/sh -c 'sudo -H -S -n -u root /bin/sh -c
'"'"'echo BECOME-SUCCESS-ukbdiwfpcoxjqveqmvkrnhhjcorflacb;
/usr/bin/python2
/home/ansible/.ansible/tmp/ansible-tmp-1485817022.67-136944116724824/lineinfile.py;
rm -rf "/home/ansible/.ansible/tmp/ansible-tmp-1485817022.67-136944116724824/"
> /dev/null 2>&1'"'"' && sleep 0'
changed: [localhost] => {
    "backup": "",
    "changed": true,
    "diff": [
        {
            "after": "",
            "after_header": "/etc/ssh/sshd_config (content)",
            "before": "",
            "before_header": "/etc/ssh/sshd_config (content)"
        },
        {
            "after_header": "/etc/ssh/sshd_config (file attributes)",
            "before_header": "/etc/ssh/sshd_config (file attributes)"
        }
        ......
}

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0

In the output above you can see ansible perform a ESTABLISH, EXEC and PUT to get the ansiballz module deployed to a system. It also prints the playbook execution variables associated with the task. Super handy!

4. Using dry run mode to see what will change

In addition to syntax checking ansible has a “–check” option to perform a dry run. Modules that support the check feature will print the items they are going to change but won’t actually make any changes to your systems. Here is a sample run:

$ ansible-playbook --check playbooks/disable-root-sshd-login.yml

PLAY [control] *****************************************************************

TASK [setup] *******************************************************************
ok: [ansible]

TASK [Disable remote SSH logins as root] ***************************************
changed: [ansible]

PLAY RECAP *********************************************************************
ansible      : ok=2    changed=1    unreachable=0    failed=0   

In the output above we can see that the the second task will change the system configuration. If you have a complex ansible layout this option can be invaluable for helping you understand what is getting updated prior to anything changing.

5. Stepping through playbook execution

Another super useful ansible debugging option is the ability to step through your playbooks one task at a time. If your playbook contains a series of tasks that build off of each other this option can be used to stop execution so you can review what a task did to a system. To step through execution you can use the “–step” option:

$ ansible-playbook --step playbooks/disable-root-sshd-login.yml

PLAY [control] *****************************************************************
Perform task: TASK: setup (N)o/(y)es/(c)ontinue: y

Perform task: TASK: setup (N)o/(y)es/(c)ontinue: *******************************

TASK [setup] *******************************************************************
ok: [ansible.homefetch.net]
Perform task: TASK: Disable remote SSH logins as root (N)o/(y)es/(c)ontinue: y

Perform task: TASK: Disable remote SSH logins as root (N)o/(y)es/(c)ontinue: ***

TASK [Disable remote SSH logins as root] ***************************************
changed: [ansible.homefetch.net]

PLAY RECAP *********************************************************************
ansible.homefetch.net      : ok=2    changed=1    unreachable=0    failed=0   

Prior to each task running you will be prompted with an option to run the task, to ignore a task and one to run the rest of the playbook in its entirety. This is one of the best tools in my ansible bat belt.

6. Print variables and debugging information inside your playbooks

When you are testing new playbooks its super useful to be able to print debugging information during playbook runs. Ansible has a debug module which you can use to print text strings, variables and fact values. Variables are enclosed in {{ }} pairs and the list of system facts can be viewed with `ansible-playbook -m setup`. The following shows how debug can be used to print the network interfaces and the default IPv4 address for my control group:

---
- hosts: control
  gather_facts: yes
  tasks:
     - debug: 
         msg: "Network interfaces assigned to {{ inventory_hostname }}: {{ ansible_interfaces }}"
     - debug: 
         msg: "Default IPv4 address assign to {{ inventory_hostname }}: {{ ansible_default_ipv4.address }}"

If this playbook is run the values will be printed to stdout:

$ ansible-playbook playbooks/vars.yml 

PLAY [control] *****************************************************************

TASK [setup] *******************************************************************
ok: [ansible]

TASK [debug] *******************************************************************
ok: [ansible] => {
    "msg": "Network interfaces assigned to ansible: [u'lo', u'eno16780032']"
}

TASK [debug] *******************************************************************
ok: [ansible] => {
    "msg": "Default IPv4 address assign to ansible: 192.168.1.7"
}

PLAY RECAP *********************************************************************
ansible      : ok=3    changed=0    unreachable=0    failed=0   

If you are using the command and shell modules you can register the output from the command and print it using debug:

---
- hosts: control
  gather_facts: yes
  tasks:
    - command: ls -1 /tmp
      register: tmp_files

    - debug: 
        msg: "Files in /tmp: {{ tmp_files.stdout_lines }}"

$ ansible-playbook playbooks/vars.yml 

PLAY [control] *****************************************************************

TASK [setup] *******************************************************************
ok: [ansible]

TASK [command] *****************************************************************
changed: [ansible]

TASK [debug] *******************************************************************
ok: [ansible] => {
    "msg": "Files in /tmp: [u'ansible_dlMYA3', u'systemd-private-0778c17a65c04f01a4ba8765903a26fc-gorp.service-hc9Yf6', u'systemd-private-53253bf62ed94cfb9b4c5b14c5755193-gorp.service-5opN22']"
}

PLAY RECAP *********************************************************************
ansible      : ok=3    changed=1    unreachable=0    failed=0   

In the output above you can see a list that contains the files that currently reside in /tmp.

The debugging tips listed above are super useful but the best debugging tool is the ansible documentation. Reading every page and knowing exactly how modules are supposed to work will save you a lot of angst and pain. If you have any additional tips to share please leave me a comment. I would like to thank Jesse Keating for writing the AMAZING Mastering Ansible book. This well written book is chock full of great information!