Blog O’ Matty

CFengine 3 Tutorial — Part 4 — Client failsafe.cf and update.cf

As stated in part 1 of this tutorial series, normal client-side operations of CFEngine is for cf-agent to:
1. Execute against /var/cfengine/inputs/failsafe.cf (which calls update.cf)
2. Execute against /var/cfengine/inputs/promises.cf

We stated that we dont want to break failsafe.cf or update.cf. When we write new CFEngine policies to implement, we import and call them from promises.cf. If we make a mistake and break the syntax, failsafe.cf / update.cf are still in pristine state. It will allow the clients to self-recover from the config breakage once we make the change through SVN.

Lets take a look at failsafe.cf

$ cat -n /var/cfengine/inputs/failsafe.cf
     1  # failsafe.cf
     2
     3  # Whatever you do, DO NOT MODIFY THIS FILE.  If you do, you can break the whole
     4  # CFEngine infrastructure as failsafe.cf and update.cf is the "failsafe" to restore things back into a working
     5  # state.  With a broken update.cf or failsafe.cf, you will be touching boxes manually
     6  # to recover.  If you break CFEngine, may shame be brought upon you and your offspring for generations.
     7  # You _should_ be modifying promises.cf to add additional bundlesequences and input files to extend CFEngine.
     8
     9  body common control
    10  {
    11          bundlesequence  =>      { "update" };
    12          inputs          =>      { "update.cf" };
    13  }
    14
    15  ############################################
    16  bundle common g
    17  {
    18  # Define Master Policy Servers
    19  vars:
    20          "phost" string  =>      "192.168.1.10";
    21  }
    22
    23  ############################################
    24
    25  body depth_search recurse(d)
    26
    27  {
    28  depth => "$(d)";
    29  }

Nice. Breaking this down line-by-line again.

1. Lines 9-13 define our “body common control” stanza. Remember, this is like the main() function in Java / C / Python. This is the starting off point for our policy to execute.
2. Line 12 We input “update.cf” into the execution of this policy. By using input statements, we can break configurations out into multiple files. Think of this like importing Apache SSL configurations in /etc/www/conf/httpd.conf from an external file like /etc/www/conf/ssl.conf
3. Line 11 We start our execution from the “update” promise. This is located in update.cf.
4. Line 16-21 We define a global variable to be used throughout execution of this policy. Here, phost is defined as a single IP address. This is the IP address of our Master Policy Server. If we wanted to extend our CFEngine infrastructure over multiple Master Policy Hosts, you would extend that network information here.
5. Line 25-29 Defines a function called “recurse” that takes an argument. This argument tells the recurse function what depth to search to. We’ll see this being called in update.cf

So, this is all pretty straightforward stuff. Lets see the interesting bits in update.cf.

$ cat -n /var/cfengine/inputs/update.cf
     1  # update.cf
     2
     3  bundle agent update
     4  {
     5  vars:
     6
     7  "master_location" string        =>      "/var/cfengine/masterfiles/client_inputs";
     8  "master_modules" string         =>      "/var/cfengine/masterfiles/client_modules";
     9
    10  files:
    11          # /var/cfengine should remain 0700.  Nobody but the root user should be poking around in here
    12          "/var/cfengine/"
    13                  perms           =>      m_u_g("0700","root","root"),
    14                  depth_search    =>      avoid_inputs_recurse("inf"),
    15                  action          =>      immediate;
    16
    17          # Update the config files from the policy master servers
    18          "/var/cfengine/inputs"
    19                  perms           =>      m_u_g("0600","root","root"),
    20                  copy_from       =>      remote_copy("$(master_location)","$(g.phost)"),
    21                  depth_search    =>      recurse("inf"),
    22                  action          =>      immediate;
    23
    24          # Update the modules from the policy master servers
    25          "/var/cfengine/modules"
    26                  perms           =>      m_u_g("0700","root","root"),
    27                  copy_from       =>      remote_copy("$(master_modules)","$(g.phost)"),
    28                  depth_search    =>      recurse("inf"),
    29                  action          =>      immediate;
    30
    31          # Update the binaries from the sbin directory
    32          "/var/cfengine/bin"
    33                   perms          =>      m_u_g("0700","root","root"),
    34                  copy_from       =>      mycopy("/var/cfengine/sbin","localhost"),
    35                  depth_search    =>      recurse("inf"),
    36                  action          =>      immediate;
    37  }
    38  ############################################
    39  body perms m_u_g(m,u,g)
    40  {
    41          mode            =>      "$(m)";
    42          owners          =>      { "$(u)" };
    43          groups          =>      { "$(g)" };
    44  }
    45  #########################################################
    46  body copy_from mycopy(from,server)
    47  {
    48          source          =>      "$(from)";
    49          compare         =>      "digest";
    50          purge           =>      "true";
    51  }
    52
    53  #########################################################
    54  body action immediate
    55  {
    56          ifelapsed       =>      "1";
    57  }
    58
    59  #########################################################
    60  body copy_from remote_copy(sourcedir,sourceserver)
    61  {
    62          source          =>      "$(sourcedir)";
    63          servers         =>      { "$(sourceserver)" };
    64          copy_backup     =>      "true";
    65          purge           =>      "true";
    66          trustkey        =>      "true";
    67          compare         =>      "digest";
    68          encrypt         =>      "true";
    69          verify          =>      "true";
    70  }
    71  body depth_search avoid_inputs_recurse(d)
    72  {
    73          depth           =>      "$(d)";
    74          exclude_dirs    =>      { "/var/cfengine/inputs", "/var/cfengine/state" };
    75  }

Finally. Some really interesting CFengine stuff to talk about. Lots of things are happening in those 75 lines. Lets break it down line by line again.

* Line 3 defines “bundle agent update” which is what we imported into bundlesequence from failsafe.cf on line 11. This is how the logic / flow of control is passed between different config files in CFEngine.
* LIne 5-8 defines some string variables. There are all sorts of variables that can be defined in CFengine. (integers, strings, arrays, etc..)
* Line 12 defines some actions we want to have taken on the /var/cfengine directory tree
** Line 13 executes the m_u_g function that is defined on line 39. We pass it the UNIX permissions for mode, user, and group we want to have applied.
*** Note, that “perms” is offered on line 13, but “perms” is also defined as the promise type in the m_u_g function on line 39. This is by design.
** Lines 14 and 15 also call other functions defined lower in updates.cf
* Lines 17-22 updates /var/cfengine/inputs on the clients from data on the Master Policy Servers. This is what will update promises.cf should we break it.
** Line 20 calls remote_copy, a function that we define starting on line 60.
** Line 20 calls remote_copy using two variables. The first variable, $master_location, we defined in “bundle agent update” i.e. this current scope — so we can call this variable directly. The variable $g.phost is “outside” the current scope. We defined $phost in failsafe.cf on line 20 in “bundle common g”. We prefix the name of the variable with the bundle it was called from. We called the bundle “g” on line 16 of failsafe.cf — hence the use of $g.phost to define which bundle / variable we want to access.
*** Line 60-70 defines the remote_copy function. Note that we encrypt the transport, we compare config files using a MD5 digest, verify the files were transferred correctly, etc…
*** Line 63 accepts an array of strings. We passed g.phost into this function, which is passed into “sourceservers” and finally into “servers.” But instead of a single entry in g.phost, we could have several and line 63 would be able to accept all of them. This isn’t a string variable. Its an array of strings.
* Lines 24-29 populates /var/cfengine/modules on the clients. Modules are shell scripts used to define custom classes on clients. More on this in part 5 of this tutorial.
* Lines 31-36 verifies that the contents of /var/cfengine/bin are idendical to /var/cfengine/sbin. The sbin is considered the “pristine” binary directory. If cf-agent detects that there is some change on the binaries which get executed in /var/cfengine/bin, then they will be re-populated from the pristine source. This is an attempt to make the clients auto-recover in a resilient hands off fashion should one of the binaries be corrupted.

That’s it! We’ve looked at our first complex CFEngine policy and saw how cf-agent is going to behave. This will be used to download new policies / auto recover in the case of a broken promises.cf policy.

One more note: Before we run this for the first time to “phone home” to the master policy servers, we need to generate a public / private key for this client machine. This works exactly like SSH public key authencation. On first contact of the client to the server, the server saves a copy of the public key from the client. Every attempt from the client to the server hence forward authenticates using this key. Data is encrypted between cf-serverd and cf-agent using this key. The client also saves a copy of the master policy server’s keys.

This key only needs to be generated once in the client’s lifetime. To generate the key (and I execute this from a postinstall package script) execute the cf-key binary.

 $ /var/cfengine/bin/cf-key
Making a key pair for cfengine, please wait, this could take a minute...

 ls -l /var/cfengine/ppkeys/localhost.p*
-rw-------   1 root     root        1743 Jul  2 12:33 /var/cfengine/ppkeys/localhost.priv
-rw-------   1 root     root         426 Jul  2 12:33 /var/cfengine/ppkeys/localhost.pub

$ cat /var/cfengine/ppkeys/localhost.pub
-----BEGIN RSA PUBLIC KEY-----
MIIBCAKCAQEA57cGTBfsqTwfuawgyO9K9tLt7IOvns7lAku/8XcyUkJ0AY0AATVK
TVjI7E1HT/moTvvLo+t6QuCD6Eo3+K++OaeP4pmSXhcGRFhuK4IVSLjfuDtYfwmn
Kd730gP2KONQZiIiVkQfsd1ADMTxTtldv/UR1COG49wexKA3f13iBNEj7d6YehHy
PFabbFpjcGmelg5yu0nDopUrGGg402BLAc8Z9H/7QxrzktH9uVrFuLitGE8reyJQ
2A8wQErRgtgpVBC2M1NFo4bWIk6mkLCukF6EOuUxzEgUjcToCc8p5sr5j2kpj+Vi
n5m16pcjkQo+EX+t7wbnRFy1PK0d98SrfwIBIw==
-----END RSA PUBLIC KEY-----

Keys are saved in /var/cfengine/ppkeys. If you rebuild a client, or regenerate keys, you’ll need to remove the old public key entry on the master policy servers. Since clients also cache the public key of the master policy servers — if the server is rebuilt or keys regenerated then this old key will need to be removed from all the clients so they can re-cache the new key. In short: treat these keys like you would with SSH keys. If you loose / damage a private key, it could be a PITA the recover from.

One last note: lets execute failsafe.cf and watch corrections take place. Note, we do not specify the absolute path in -f because by default, cf-agent will look in /var/cfengine/inputs (where failsafe.cf and update.cf live). We execute in –inform mode using -I so we see the good changes that cf-agent makes and -K so we aren’t held on locks.

/var/cfengine $ touch mike
/var/cfengine $ chmod 777 mike
/var/cfengine $ /var/cfengine/bin/cf-agent -f failsafe.cf -I -K
 -> Object /var/cfengine/mike had permission 777, changed it to 700

/var/cfengine $ rm /var/cfengine/inputs/promises.cf
/var/cfengine $ echo 'this is a garbage statement' > /var/cfengine/inputs/promises.cf
/var/cfengine $ /var/cfengine/bin/cf-agent -f failsafe.cf -I -K
 -> Updated /var/cfengine/inputs/promises.cf from source /var/cfengine/masterfiles/client_inputs/promises.cf on 192.168.1.10