Archive
Posts from 2017
Troubleshooting a bizarre logstash CPU problem
This week I was converting some legacy shell scripts to ansible roles and wandered into a bizarre issue with one of my elasticsearch servers. After committing a couple of changes my CI system rejected the commit due to a system resource issue. When I logged into the system to troubleshoot the issue I noticed the CPU was pegged: This system is used solely to test changes so it should have been 100% idle. Htop showed the Logstash java process as the top CPU consumer so I ventured off to the log directory to see if anything was awry…
$ read more →How I saved myself 3-months of frustration by finding and fixing a sendmail bug on a vendors appliance
I recently upgraded a vendors appliance to a new version of code to address a couple of known isues. After the upgrade I noticed that the appliance wasn't sending e-mail (yes some vendors still use e-mail) alerts when problems were detected. Luckily the appliance runs Linux and provides shell access through SSH if you enable it. My previous support experiences have been less than stellar so I figured I would root cause the problem and send them a bug report instead of opening a support case…
$ read more →Using Ansible to verify remote file checksums with get_url, lookup() and stat
Being an extremely security minded operations guy I take every precaution to verify that the files I download are legit. In this day and age of servers and data getting compromised this should be an operational standard. There are numerous ways to verify checksums. You can use openssl's various hashing options or a simple wrapper script similar to this…
$ read more →Retrieving a file name from a URL with ansible
Ansible has several extremely powerful modules for interacting with web resources. The get_url module allows you to retrieve resources, uri allows you to interact with web services and the templating and filtering capabilities provided by Jinja2 allow you to slice and dice the results in a variety of ways. One pattern that I find useful is applying the basename filter to the URL to grab a file name: After the filter is applied file_name will contain the last component in the "/" separated URL: I'll share some valid uses for this in a future blog post.
$ read more →How elasticsearch bootstrap checks affect development and production mode
One of my friends reached out to me earlier this week to help him with an elasticsearch issue. He was trying to bring up a new cluster to see how ES compares to splunk and was getting a "bootstrap checks failed" error at startup. This was causing his elasticsearch java processes to bind to localhost instead of the hostname he assigned to the network.host value. Here is a snippet of what I saw when I reviewed the logs: Elasticsearch has two modes of operation: development and production…
$ read more →