Monitoring DNS domain name expiration with dns-domain-expiration-checker

Several years ago I wrote a simple bash script to check the expiration date of the DNS domains I own. At the time I wrote this purely for my own needs but after receiving 100s of e-mails from folks who were using it (and submitting patches) I decided to enhance it to be more useful. As time has gone on Registrar WHOIS data formats have changed and I came to the realization that there is no standard time format for expiration records. I needed a more suitable solution so I spent last weekend re-writing my original script in Python. The new version is available on github and solves all of the issues I previously encountered.

To work around the Registrar WHOIS data format issue I created a list of strings which can be easily extended as new ones are encountered (or changed):

EXPIRE_STRINGS = [ "Registry Expiry Date:",
                   "Expiration:",
                   "Domain Expiration Date"
                 ]

These strings are checked against the WHOIS data returned from `which whois` and if a match is found I save off $NF which should be the date. To get around the date formatting issues I pulled in the dateutil module which does an AMAZING job of normalizing dates. This module allows me to feed it random date formats which are then normalized to datetime objects which I can perform math on. The github README contains several examples showing how to use the script. The most basic form allows you to see expiration data for a domain or a set of domains in a file:

$ dns-domain-expiration-checker.py --domainname prefetch.net --interactive
Domain Name                Registrar             Expiration Date                 Days Left
prefetch.net               DNC Holdings, Inc.    2020-06-23 16:56:05             1056

The script also provides SMTP alerting and Nagios support will be added in the near future. If you use the script and encounter any bugs please shoot me an issue on github.

Simplifying python development with virtual environments

Over the past year I’ve spent a considerable amount of time beefing up my development skills. One of the best ways I’ve found to improve my skills is by reading code from well respected developers, watching udemy and youtube videos, fixing bugs in code I’m not familiar with (you will find some doozies this way) and benchmarking code to see what the system and resource ramifications of a change are.

To allow myself to experiment in isolation I’ve relied heavily on Python virtual environments. Virtual environments are code sandboxes and everything (libraries, packages, etc.) in a given virtual environment is isolated from everything else on your system. You can hop between these environments quite easily and if you break something the breakage won’t leak out of the environment. To use virtual environments on a Ubuntu host you will first need to install a couple of packages with apt (packages are also available for CentOS):

$ sudo apt install virtualenvwrapper

Once the virtual environment packages are installed you will need to define the PROJECT_HOME (older versions use the WORKON_HOME environment varaible) environment variable to tell virtualenv where to store your environments:

$ mkdir /home/matty/python-virtualenv

$ export PROJECT_HOME=/home/matty/python-virtualenv

To get a better feel for how virtual environments work lets say you want to modify the foo module to see if a new algorithm would perform better than what is currently in place. To create a new environment to test this hypothesis you can run the mkproject helper with the name of the environment you want to create:

$ mkproject foo

This will create a new virtual environment and initialize it with an isolated python runtime environment. To see the environments on your system you can run lsvirtualenv:

$ lsvirtualenv

bar
===

foo
===

To sandbox yourself in the new enviroment you can run the workon helper with the name of the environment you want to work in:

$ workon foo

Once you are in a virtual environment you can install new packages with pip, pull down code with git or alter system libraries to your liking. All of these changes will be isolated to this environment. If you change a library or python program and something breaks that breakage will only exist in the virtual environment. This is incredibly useful for debugging modules from PyPi and evaluating changes without “polluting” a branch (less of an issue with a good CI suite) or your system libraries. To leave a virtual environment you can use the deactivate helper:

$ deactivate

Once you are finished vetting your change you can delete a virtual environment with the rmvirtualenv helper:

$ rmvirtualenv foo

The virtualenvwrapper package comes with several other useful helpers. showvirtualenv will show you details about a specific virtual environment, cpvirtualenv will duplicate an existing virtual environment and the various sitepackages helpers will allow you to work with the site-packages that are in your virtual environment. Great stuff!

Making Python code more readable with list comprehensions

This post is added as a reference for myself.

As I mentioned previously I’ve been spending a good deal of time learning how to write efficient and readable Python code. Jeff Knupp’s Idiomatic Python videos are absolutely incredible and have definitely helped me become a better programmer. In his first video he turned me on to list comprehensions and how they can make code more readable. Prior to watching this video I had a tendency to use multi line iterators similar to this:

def lower_a_list(list):
    ml = []
    for word in list:
        ml.append(word.lower())
    return ml

After watching Jeff do some crazy awesome python sorcery on bad.py I’ve started to use list comprehensions. Here is a refactored lower_a_list() that uses list comprehensions:

def lower_a_list(list):
    return [word.lower() for word in list]

I find the second much more readable and you can get the gist of what’s going on by scanning from right to left. Huge thanks to Jeff for putting out this amazing series!

Python generators you had me at first yield

This post is added as a reference for myself.

I’ve been reading a lot about writing clean, readable and performant Python code. One feature that I now adore are generators. In the words of Python guru David Beazley a generator is:

“a function that produces a sequence of results instead of a single value”

The presentation above is an incredible overview of this amazing feature and I’ve been taking advantage of it whenever I can. Here is one case where I was able to use this today:

#!/usr/bin/env python

import os
import re

SYSFS_TAPE_PATH = "/sys/class/scsi_tape/"

def find_drives():
    """
       Return a list of tape drives
    """
    for drive in os.listdir(SYSFS_TAPE_PATH):
        if re.match(r'^nst[0-9]+$', drive):
            yield drive

for drive in find_drives():
    print drive

Running the script produces all of the drives on the system:

$ ./generator 
nst0
nst1
nst2
nst3

In the past I would have created a list and returned it after processing all of the drives in the scsi_tape directory. With generators that is no longer necessary and I find the code is much more readable. A huge thanks to David for the great presentation! Hopefully one day I will get to take one of his awesome Python programming classes.

Python console editing with VIM

I’m a long time console guy and haven’t found a graphical Python development tool that suits my needs as well as vim. It takes a couple of amazing vim plug-ins but the experience is great IMHO. I’m currently using the vim-jedi, pylint and vim-syntastic plug-ins which can be installed with yum on CentOS/Fedora machines:

$ sudo yum install pylint

$ sudo yum install vim-jedi

$ sudo yum install vim-syntastic-python

To enable syntax highlighting and auto indentation you can add the following to your $HOME/.vimrc:

syntax on
filetype indent plugin on

Once these are in place your code will be highlighted with a nice set of colors, hitting return will generate the correct spacing and pylint will be run automatically to note problems with your code. If you are using other plug-ins let me know via a comment.

Reading a file into a Python string

I’ve learned a number of useful things from the Google learn Python video series. One of the tips I got to use today. That tip was Python’s ability to read a file into a string:

$ cat foo
this
is
a
test
file
of
words

$ python
>>> f = open(“foo”,”r”)
>>> string = f.read()
>>> string
‘this\nis \na \ntest\nfile\nof \nwords\n’

This has a few interesting uses, and I plan to put this to use this weekend when I finish up a Python project I’m working on. I really, really dig Python. It’s quite swell. :)