Reading a file into a Python string

I’ve learned a number of useful things from the Google learn Python video series. One of the tips I got to use today. That tip was Python’s ability to read a file into a string:

$ cat foo
this
is
a
test
file
of
words

$ python
>>> f = open(“foo”,”r”)
>>> string = f.read()
>>> string
‘this\nis \na \ntest\nfile\nof \nwords\n’

This has a few interesting uses, and I plan to put this to use this weekend when I finish up a Python project I’m working on. I really, really dig Python. It’s quite swell. :)

Getting MySQL running on a CentOS Linux server

I started playing with MySQL back in the 4.X days, but never invested a lot of my time since my day job required me to support Oracle databases. I’m trying to branch out more now, and recently picked up a copy of MySQL, MySQL High Availability and PHP And MySQL. There are a slew of things I would like to web-enable, so I’m hoping to learn everything I can about PHP and MySQL in the next few months.

To allow me to start experimenting with PHP and MySQL, I needed to create a test environment. My MySQL environment consists of two CentOS 6 virtual machines running MySQL 5.1.X. Getting MySQL working on these two machines was amazingly easy. First, I installed the mysql packages with yum:

$ yum install mysql mysql-server

Next I started up the MySQL services and made sure they started at boot:

$ chkconfig mysqld on

$ service mysqld start

And finally I secured my MySQL installation by running the mysql_secure_installation script:

$ /usr/bin/mysql_secure_installation

NOTE: RUNNING ALL PARTS OF THIS SCRIPT IS RECOMMENDED FOR ALL MySQL
      SERVERS IN PRODUCTION USE!  PLEASE READ EACH STEP CAREFULLY!

In order to log into MySQL to secure it, we'll need the current
password for the root user.  If you've just installed MySQL, and
you haven't set the root password yet, the password will be blank,
so you should just press enter here.

Enter current password for root (enter for none): 
OK, successfully used password, moving on...

Setting the root password ensures that nobody can log into the MySQL
root user without the proper authorisation.

Set root password? [Y/n] y
New password: 
Re-enter new password: 
Password updated successfully!
Reloading privilege tables..
 ... Success!

By default, a MySQL installation has an anonymous user, allowing anyone
to log into MySQL without having to have a user account created for
them.  This is intended only for testing, and to make the installation
go a bit smoother.  You should remove them before moving into a
production environment.

Remove anonymous users? [Y/n] y
 ... Success!

Normally, root should only be allowed to connect from 'localhost'.  This
ensures that someone cannot guess at the root password from the network.

Disallow root login remotely? [Y/n] y
 ... Success!

By default, MySQL comes with a database named 'test' that anyone can
access.  This is also intended only for testing, and should be removed
before moving into a production environment.

Remove test database and access to it? [Y/n] y
 - Dropping test database...
 ... Success!
 - Removing privileges on test database...
 ... Success!

Reloading the privilege tables will ensure that all changes made so far
will take effect immediately.

Reload privilege tables now? [Y/n] y
 ... Success!

Cleaning up...

All done!  If you've completed all of the above steps, your MySQL
installation should now be secure.

Thanks for using MySQL!

Once this completed I restarted MySQL and could login using the root user (additional accounts will be added in the near future):

$ service mysqld restart

Stopping mysqld:                                           [  OK  ]
Starting mysqld:                                           [  OK  ]

$ mysql -h localhost -u root –password=XXXXXXXX

Welcome to the MySQL monitor.  Commands end with ; or \g.
Your MySQL connection id is 2
Server version: 5.1.52-log Source distribution

Copyright (c) 2000, 2010, Oracle and/or its affiliates. All rights reserved.
This software comes with ABSOLUTELY NO WARRANTY. This is free software,
and you are welcome to modify and redistribute it under the GPL v2 license

Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.

mysql> 

It took me 5 minutes to get MySQL to a state were I could enable replication and create databases. Nice!

Integrating ssh-agent into your login process

Most of my readers utilize SSH keys to access remote systems. The security benefits are well known, and key-based authentication makes automating remote tasks a whole lot easier. When you use key-based authentication it becomes imperative to protect your private key, since a third party could access your systems if they were able to gain access to your account. The SSH key generator (ssh-keygen) will attempt to encrypt your private key by default, and can also be used ssh-keygen to add a password to a private key after the fact.

With passwords comes prompts, and with prompts comes frustration. To alleviate this frustration you can use the ssh-agent process to minimize the number of times you need to type your password. Ssh-agent stores your private keys securely in memory, and hands them out to the ssh process when you attempt to connect to remote systems. Keys are added to ssh-agent through the ssh-add command line utility, which will prompt you for your private key password prior to adding them to the keys held in memory by ssh-agent.

Each time you access a remote system the ssh client will contact the ssh-agent process to acquire your private keys. If you start ssh-agent and run ssh-add to add your private keys when you login to a server, you will now be able to access other hosts using key-based authentication without a password for the length of the shell session. You will find this especially useful when you are using tools like clusterit to manage remote machines.

To automate the process above, I like to modify my bash environment to prompt me for the password when I login to my servers. The ssh-add prompt I get looks similar to this:

$ ssh proxy.prefetch.net

Last login: Tue Jan 17 20:37:54 2012 from 192.168.1.121
Starting an ssh-agent process
Enter passphrase for /home/matty/.ssh/id_dsa:
Identity added: /home/matty/.ssh/id_dsa (/home/matty/.ssh/id_dsa)

Once I’ve input the correct password I can then access other systems freely and without a password. So what exactly did I do to integrate ssh-agent into my shell environment? First I added an exec statement to create an ssh-agent process and make the bash process a child of it (the reasons why this is required are documented in the SSH FAQ):

$ grep ssh-agent .bash_profile

# Start an ssh-agent process and start bash as a child of it
echo “Starting an ssh-agent process”
exec ssh-agent bash

Once ssh-agent is tied into the environment I call ssh-add from my .bashrc to add my private keys:

$ grep ssh-add .bashrc

# Add the encrypted private keys to my in-memory key store with ssh-add.
echo “Calling ssh-add to add my private key to ssh-agent”
ssh-add

The second entry is what causes the following password prompt when I login to my servers:

Enter passphrase for /home/matty/.ssh/id_dsa:

I always try to do everything I can to improve security, and this is definitely one of those every admin should do to protect their beloved private key(s). :) If you are doing something differently, please share your thoughts in the comment section below.

Free video tutorials for C, Java, PHP, HTML5, Python, MySQL and more …

I just came across the new boston video tutorial series. I’ve watched 20 of the PHP videos and am hooked. The production quality is great, and the content is really, really good! Once I finish the 200 PHP videos I plan to watch their MySQL and HTML5 videos. Can’t recommend these videos enough, and the fact that they’re free makes them even better! Nice!

The importance of keeping your storage array firmware up to date

A couple of weeks back I attempted to migrate a pair of clustered Solaris 10 servers to a new disk storage array. After rebooting into single user mode to pick up the new devices, I went to add the new quorum disk with clquorum. This resulted in both nodes panicking with the following panic string:

panic[cpu3]/thread=fffffe800125bc60: Reservation Conflict
Disk: /scsi_vhci/disk@g6000d310002c6700000000000000003e

fffffe800125ba40 fffffffff7959e39 ()
fffffe800125ba70 sd:sd_pkt_status_reservation_conflict+c8 ()
fffffe800125bab0 sd:sdintr+431 ()
fffffe800125bb50 scsi_vhci:vhci_intr+3da ()
fffffe800125bb70 fcp:ssfcp_post_callback+4a ()
fffffe800125bba0 fcp:ssfcp_cmd_callback+4c ()
fffffe800125bc00 qlc:ql_task_thread+756 ()
fffffe800125bc40 qlc:ql_task_daemon+94 ()
fffffe800125bc50 unix:thread_start+8 ()

At first I thought I was doing something wrong, but after a lot of research I figured out that there were a couple of Solaris-related bugs in the version of the storage array firmware we were using. One of the bugs was triggering the panic above, and after the array was patched everything worked as expected. Keeping up to date with firmware is just as important as keeping up to date with OS patches. It’s amazing how many firmware bugs there are, and they bite you in the oddest ways.

How to figure out if a processes has been chroot()’ed

A number of applications (e.g., custom chroot jails, openssh, vsftp, apache) support the ability to chroot themselves. To find out if a process called chroot() at startup, you can check the /proc/<pid>/root entry for the process. For non-chrooted processes this entry will point to /:

$ ps auxwww | grep [s]endmail

root      3643  0.0  0.1  69032  2344 ?        Ss    2011   0:01 sendmail: accepting connections
smmsp     3651  0.0  0.0  59784  1780 ?        Ss    2011   0:01 sendmail: Queue runner@01:00:00 for /var/spool/clientmqueue

$ cd /proc/3643

$ ls -lad root

lrwxrwxrwx 1 root root 0 Jan 22 10:23 root -> /

For a chrooted process the root directory will point to the directory passed to the chroot() system call:

$ ps auxwww | grep [n]amed

named    18298  0.0  2.3 243632 49084 ?        Ssl   2011  15:16 /usr/sbin/named -u named -t /var/named/chroot

$ cd /proc/18298

$ ls -lad root

lrwxrwxrwx 1 named named 0 Jan 22 10:19 root -> /var/named/chroot

Chroot environments can be made secure, especially if you follow the coding practices discussed in Building Secure Software and Using Chroot Securely. These are must reads for anyone who plans to use chroot()!