Archive
Posts in Linux
A couple of quick and easy ways to display JSON data on the Linux command line
I interact with RESTful services daily and periodically need to review the JSON objects exposed through one or more endpoints. There are several Linux utilities that can take a JSON object and print the object in an easily readable form. The pygmentize utility (available in the python-pygments package) can be fed a JSON object via a file or STDIN: In the output above I'm retrieving a JSON object from the Bind statistics server and feeding it to pygmentize via STDIN. Pygmentize will take the object is given and produce a nightly formatted JSON object on STDOUT…
$ read more →Conditionally restarting systemd services
In a previous post I discussed how one of my systemd services was getting continuously restarted causing the CPU to spike. This isn't ideal and after re-reading the systemd manual page I came across a couple of useful options to control when and how frequently a systemd service will restart. The first option is RestartSec which controls how long systemd will wait to restart a process after a failure occurs. Systemd also has the RestartForceExitStatus and RestartPreventExitStatus which allow you to define the signals that should or should not cause a restart…
$ read more →Troubleshooting a bizarre logstash CPU problem
This week I was converting some legacy shell scripts to ansible roles and wandered into a bizarre issue with one of my elasticsearch servers. After committing a couple of changes my CI system rejected the commit due to a system resource issue. When I logged into the system to troubleshoot the issue I noticed the CPU was pegged: This system is used solely to test changes so it should have been 100% idle. Htop showed the Logstash java process as the top CPU consumer so I ventured off to the log directory to see if anything was awry…
$ read more →How I saved myself 3-months of frustration by finding and fixing a sendmail bug on a vendors appliance
I recently upgraded a vendors appliance to a new version of code to address a couple of known isues. After the upgrade I noticed that the appliance wasn't sending e-mail (yes some vendors still use e-mail) alerts when problems were detected. Luckily the appliance runs Linux and provides shell access through SSH if you enable it. My previous support experiences have been less than stellar so I figured I would root cause the problem and send them a bug report instead of opening a support case…
$ read more →Forwarding the systemd journal to syslog
I am a big fan of the ELK stack and use it daily for various parts of my job. All of the logs from my physical systems, VMs and containers get funneled into elasticsearch, indexed and are available for me to slice and dice with Kibana. In addition to syslog data I also like to funnel the systemd journal into elasticsearch. This is easily accomplished by changing the journald.conf ForwardToSyslog configuration directive to yes: This small change will cause all journal entries to get routed to the local syslog daemon…
$ read more →