Normalizing date strings with the Python dateutil module

I recently attended a Python workshop with Jeff Cohen and he answered a ton of random questions from students. One student mentioned that he was overwhelmed with the number of Python modules and Jeff told us that he has evolved his Python skills by learning at least one new module each week. I’ve started doing this as well and it’s been a HUGE help. Each Sunday I find a module I’m not familiar with in PiPY or the standard library and read through the documents. Then I do a number of coding exercises to see how the module works. Once I’m comfortable using it I try to read through the source code to see how it’s implemented under the covers. The last part is time consuming but it’s a great way to really understand how the module works.

While perusing the date modules last weekend I came across dateutil. This handy little module provides a built-in parser to take arbitrary dates and normalize them into a datetime object. If you are dealing with different data sources without a common set of formatting standards you will love this little guy! To see how this works say you have two dates and need to get the number of days between them. The following snippet does this.

>>> import dateutil.parser

>>> date1 = "2020-06-23T16:56:05Z"

>>> date2 = "June 22 2018 09:23:45"

>>> d1 = dateutil.parser.parse(date1, ignoretz=True)

>>> d2 = dateutil.parser.parse(date2, ignoretz=True)

>>> print (d1-d2).days

If you need higher resolution you can use the min, seconds, total_seconds and microseconds attributes to drill down further. This useful module made rewriting a breeze!

Monitoring DNS domain name expiration with dns-domain-expiration-checker

Several years ago I wrote a simple bash script to check the expiration date of the DNS domains I own. At the time I wrote this purely for my own needs but after receiving 100s of e-mails from folks who were using it (and submitting patches) I decided to enhance it to be more useful. As time has gone on Registrar WHOIS data formats have changed and I came to the realization that there is no standard time format for expiration records. I needed a more suitable solution so I spent last weekend re-writing my original script in Python. The new version is available on github and solves all of the issues I previously encountered.

To work around the Registrar WHOIS data format issue I created a list of strings which can be easily extended as new ones are encountered (or changed):

EXPIRE_STRINGS = [ "Registry Expiry Date:",
                   "Domain Expiration Date"

These strings are checked against the WHOIS data returned from `which whois` and if a match is found I save off $NF which should be the date. To get around the date formatting issues I pulled in the dateutil module which does an AMAZING job of normalizing dates. This module allows me to feed it random date formats which are then normalized to datetime objects which I can perform math on. The github README contains several examples showing how to use the script. The most basic form allows you to see expiration data for a domain or a set of domains in a file:

$ --domainname --interactive
Domain Name                Registrar             Expiration Date                 Days Left               DNC Holdings, Inc.    2020-06-23 16:56:05             1056

The script also provides SMTP alerting and Nagios support will be added in the near future. If you use the script and encounter any bugs please shoot me an issue on github.

Using awk character classes to simplify parsing complex strings

This week I was reading a shell script in a github repository to see if it would be good candidate to automate a task. As I was digging through the code I noticed a lengthy shell pipeline to parse a string similar to this:

Thu Jul 20 18:13:04 EDT 2017 snarble foo bar (gorp): blatch (fmep): gak+

Here is the code she/he was using to extract the string “gorp”:

$ cat /foo/bar.txt | grep “snarble” | awk ‘{print $10}’ | awk -F'(‘ ‘{print $2}’ | awk -F’)’ ‘{print $1}’

After my eyes recovered I thought this would be a good candidate to simplify with awk character classes. These are incredibly useful for applying numerous field separators to a given line of input. I took what the original author had and simplified it to this:

$ awk -F'[()]+’ ‘/snarble/ {print $2}’ /foo/bar.txt

The argument passed to the field separated option (-F) contains a list of characters to use as delimiters. The string inside the slashes are used to match all lines that contain the word snarble. I find the second a bit easier to read and character classes are a super useful!

Debugging ansible playbooks, plays and tasks

I am a long time ansible user and have wrangled it into automating just about everything I do. As my roles and playbooks have increased in quantity and size I’ve found it’s essential to have a good grasp of the debugging capabilities built into ansible. These are useful for detecting syntax errors, finding ordering issues and most importantly for learning how ansible works under the covers. In this post I’m going to cover a number of methods to test playbooks and troubleshoot issues when they pop up. In a future post I’ll go into some tips and tricks I’ve learned while developing new ansible modules. Here are some of my favorites:

1. Checking the syntax of your playbooks to find YAML problems

One of the first things I encountered while developing roles were YAML formatting subtleties. A space missing here, a hyphen missing there or a filter not returning the correct results. They all add up to a pissed off ansible-playbook run! I use atom for editing my playbooks though sometimes I’ve been known to fire up vim when I need to test something quickly from an SSH session. Ansible has a “–syntax-check” option which can be used to make sure your YAML is properly structured:

$ ansible-playbook --syntax-check sshd-disable-root-login.yml
ERROR! 'tsks' is not a valid attribute for a Play

The error appears to have been in
'/ansible/playbooks/sshd-disable-root-login.yml': line 2, column 3,
but may
be elsewhere in the file depending on the exact syntax problem.

The offending line appears to be:

- hosts: localhost
  ^ here

The ansible check error output is extremely verbose and 99% of the time you will read the error and say doh!

2. Testing playbooks before they are committed to a git repository

Like most admins I use git repos for everything I touch. This allows me to see what changed, when it changed, and provides an easy way to share changes with my colleagues. Git provides a set of hooks which you can use to run commands at various stages of the commit process. You can read about hooks here. To ensure that my playbooks are syntactically correct prior to check-in I like to use a pre-commit hook to check syntax:

$ git commit -a -m "Added debug statement to playbook disable root login"
ERROR: Found a syntax error with ./playbooks/sshd-disable-root-login.yml
ERROR: Please run ansible-playbook --syntax-check
./playbooks/sshd-disable-root-login.yml to view the error

If I snarbled something git won’t let me commit the change and I get a simple error message that points me to the problematic file. This goes hand in-hand with a CI system to test your playbooks when they are committed.

3. Getting verbose output during playbook runs

Periodically issues arise and you want to see what ansible is doing when a playbook is applied. Running ansible with multiple verbose flags provides a significant amount of detail:

$ ansible-playbook -vvv playbooks/sshd-disable-root-login.yml
Using /ansible/ansible.cfg as config file

PLAYBOOK: sshd-disable-root-login.yml ******************************************
1 plays in playbooks/sshd-disable-root-login.yml

PLAY [localhost] ***************************************************************

TASK [Disable SSH root logins] *************************************************
task path: /ansible/playbooks/sshd-disable-root-login.yml:6
Using module file
<> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo
~/.ansible/tmp/ansible-tmp-1485817022.67-136944116724824 `" && echo
ansible-tmp-1485817022.67-136944116724824="` echo
~/.ansible/tmp/ansible-tmp-1485817022.67-136944116724824 `" ) && sleep
<> PUT /tmp/tmp2jJjuc TO
<> EXEC /bin/sh -c 'chmod u+x
&& sleep 0'
<> EXEC /bin/sh -c 'sudo -H -S -n -u root /bin/sh -c
'"'"'echo BECOME-SUCCESS-ukbdiwfpcoxjqveqmvkrnhhjcorflacb;
rm -rf "/home/ansible/.ansible/tmp/ansible-tmp-1485817022.67-136944116724824/"
> /dev/null 2>&1'"'"' && sleep 0'
changed: [localhost] => {
    "backup": "",
    "changed": true,
    "diff": [
            "after": "",
            "after_header": "/etc/ssh/sshd_config (content)",
            "before": "",
            "before_header": "/etc/ssh/sshd_config (content)"
            "after_header": "/etc/ssh/sshd_config (file attributes)",
            "before_header": "/etc/ssh/sshd_config (file attributes)"

PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=1    unreachable=0    failed=0

In the output above you can see ansible perform a ESTABLISH, EXEC and PUT to get the ansiballz module deployed to a system. It also prints the playbook execution variables associated with the task. Super handy!

4. Using dry run mode to see what will change

In addition to syntax checking ansible has a “–check” option to perform a dry run. Modules that support the check feature will print the items they are going to change but won’t actually make any changes to your systems. Here is a sample run:

$ ansible-playbook --check playbooks/disable-root-sshd-login.yml

PLAY [control] *****************************************************************

TASK [setup] *******************************************************************
ok: [ansible]

TASK [Disable remote SSH logins as root] ***************************************
changed: [ansible]

PLAY RECAP *********************************************************************
ansible      : ok=2    changed=1    unreachable=0    failed=0   

In the output above we can see that the the second task will change the system configuration. If you have a complex ansible layout this option can be invaluable for helping you understand what is getting updated prior to anything changing.

5. Stepping through playbook execution

Another super useful ansible debugging option is the ability to step through your playbooks one task at a time. If your playbook contains a series of tasks that build off of each other this option can be used to stop execution so you can review what a task did to a system. To step through execution you can use the “–step” option:

$ ansible-playbook --step playbooks/disable-root-sshd-login.yml

PLAY [control] *****************************************************************
Perform task: TASK: setup (N)o/(y)es/(c)ontinue: y

Perform task: TASK: setup (N)o/(y)es/(c)ontinue: *******************************

TASK [setup] *******************************************************************
ok: []
Perform task: TASK: Disable remote SSH logins as root (N)o/(y)es/(c)ontinue: y

Perform task: TASK: Disable remote SSH logins as root (N)o/(y)es/(c)ontinue: ***

TASK [Disable remote SSH logins as root] ***************************************
changed: []

PLAY RECAP *********************************************************************      : ok=2    changed=1    unreachable=0    failed=0   

Prior to each task running you will be prompted with an option to run the task, to ignore a task and one to run the rest of the playbook in its entirety. This is one of the best tools in my ansible bat belt.

6. Print variables and debugging information inside your playbooks

When you are testing new playbooks its super useful to be able to print debugging information during playbook runs. Ansible has a debug module which you can use to print text strings, variables and fact values. Variables are enclosed in {{ }} pairs and the list of system facts can be viewed with `ansible-playbook -m setup`. The following shows how debug can be used to print the network interfaces and the default IPv4 address for my control group:

- hosts: control
  gather_facts: yes
     - debug: 
         msg: "Network interfaces assigned to {{ inventory_hostname }}: {{ ansible_interfaces }}"
     - debug: 
         msg: "Default IPv4 address assign to {{ inventory_hostname }}: {{ ansible_default_ipv4.address }}"

If this playbook is run the values will be printed to stdout:

$ ansible-playbook playbooks/vars.yml 

PLAY [control] *****************************************************************

TASK [setup] *******************************************************************
ok: [ansible]

TASK [debug] *******************************************************************
ok: [ansible] => {
    "msg": "Network interfaces assigned to ansible: [u'lo', u'eno16780032']"

TASK [debug] *******************************************************************
ok: [ansible] => {
    "msg": "Default IPv4 address assign to ansible:"

PLAY RECAP *********************************************************************
ansible      : ok=3    changed=0    unreachable=0    failed=0   

If you are using the command and shell modules you can register the output from the command and print it using debug:

- hosts: control
  gather_facts: yes
    - command: ls -1 /tmp
      register: tmp_files

    - debug: 
        msg: "Files in /tmp: {{ tmp_files.stdout_lines }}"

$ ansible-playbook playbooks/vars.yml 

PLAY [control] *****************************************************************

TASK [setup] *******************************************************************
ok: [ansible]

TASK [command] *****************************************************************
changed: [ansible]

TASK [debug] *******************************************************************
ok: [ansible] => {
    "msg": "Files in /tmp: [u'ansible_dlMYA3', u'systemd-private-0778c17a65c04f01a4ba8765903a26fc-gorp.service-hc9Yf6', u'systemd-private-53253bf62ed94cfb9b4c5b14c5755193-gorp.service-5opN22']"

PLAY RECAP *********************************************************************
ansible      : ok=3    changed=1    unreachable=0    failed=0   

In the output above you can see a list that contains the files that currently reside in /tmp.

The debugging tips listed above are super useful but the best debugging tool is the ansible documentation. Reading every page and knowing exactly how modules are supposed to work will save you a lot of angst and pain. If you have any additional tips to share please leave me a comment. I would like to thank Jesse Keating for writing the AMAZING Mastering Ansible book. This well written book is chock full of great information!

Simplifying python development with virtual environments

Over the past year I’ve spent a considerable amount of time beefing up my development skills. One of the best ways I’ve found to improve my skills is by reading code from well respected developers, watching udemy and youtube videos, fixing bugs in code I’m not familiar with (you will find some doozies this way) and benchmarking code to see what the system and resource ramifications of a change are.

To allow myself to experiment in isolation I’ve relied heavily on Python virtual environments. Virtual environments are code sandboxes and everything (libraries, packages, etc.) in a given virtual environment is isolated from everything else on your system. You can hop between these environments quite easily and if you break something the breakage won’t leak out of the environment. To use virtual environments on a Ubuntu host you will first need to install a couple of packages with apt (packages are also available for CentOS):

$ sudo apt install virtualenvwrapper

Once the virtual environment packages are installed you will need to define the PROJECT_HOME (older versions use the WORKON_HOME environment varaible) environment variable to tell virtualenv where to store your environments:

$ mkdir /home/matty/python-virtualenv

$ export PROJECT_HOME=/home/matty/python-virtualenv

To get a better feel for how virtual environments work lets say you want to modify the foo module to see if a new algorithm would perform better than what is currently in place. To create a new environment to test this hypothesis you can run the mkproject helper with the name of the environment you want to create:

$ mkproject foo

This will create a new virtual environment and initialize it with an isolated python runtime environment. To see the environments on your system you can run lsvirtualenv:

$ lsvirtualenv



To sandbox yourself in the new enviroment you can run the workon helper with the name of the environment you want to work in:

$ workon foo

Once you are in a virtual environment you can install new packages with pip, pull down code with git or alter system libraries to your liking. All of these changes will be isolated to this environment. If you change a library or python program and something breaks that breakage will only exist in the virtual environment. This is incredibly useful for debugging modules from PyPi and evaluating changes without “polluting” a branch (less of an issue with a good CI suite) or your system libraries. To leave a virtual environment you can use the deactivate helper:

$ deactivate

Once you are finished vetting your change you can delete a virtual environment with the rmvirtualenv helper:

$ rmvirtualenv foo

The virtualenvwrapper package comes with several other useful helpers. showvirtualenv will show you details about a specific virtual environment, cpvirtualenv will duplicate an existing virtual environment and the various sitepackages helpers will allow you to work with the site-packages that are in your virtual environment. Great stuff!

Using VirtualBox host networks with vagrant

I have a few VirtualBox VMs that I use to test clustering, Gluster, Ceph and various other technologies. These hosts have a NAT interface as well as a host network for internal communications. This week I revamped several of my Vagrantfiles to use a host network and learned a few things in the process. To get a machine to ‘vagrant up’ and connect to an existing VirtualBox host network I first had to install the vbguest plug-in:

$ vagrant plugin install vagrant-vbguest

Copy iso file /usr/share/virtualbox/VBoxGuestAdditions.iso into the
box /tmp/VBoxGuestAdditions.iso
mount: /dev/loop0 is write-protected, mounting read-only
Installing Virtualbox Guest Additions 5.1.6 – guest version is unknown
Verifying archive integrity… All good.
Uncompressing VirtualBox 5.1.6 Guest Additions for Linux………..
VirtualBox Guest Additions installer
Copying additional installer modules …
Installing additional modules … Building Guest Additions kernel modules.

Once this plugin was installed I added a “private_network” stanza similar to the following to each Vagrantfile: “private_network”, ip: “”

Then I ran ‘vagrant up’ and my machines could see the host networks my VirtualBox VMs reside on. Niiiiice!