After several colleagues recommending Puppet, I hesitantly began the slow, brain fscking process of:

1. Understanding exactly what I had accomplished with cfEngine.
2. Understanding Ruby (ugh, I’m so thankful for Perl)
3. Understanding how to express my cfengine feelings in a way Puppet will understand without hurting it’s feelings
4. …
5. Profit.

cfEngine makes some things incredibly easy to manage.  Nearly every command allows you to “define” new classes based on various conditions.  This allows to modify a configuration file, and then tell the daemon associated with that config file to restart.  However, when I needed to do something highly specialized, I had to create a shell script, copy the shell script to the server and then run the shell script.  Passing data back to do something was possible, though it seemed a bit hacky.  It separated the customized actions being performed from the dependent actions in the cfEngine configs.  If I had to go back later and make changes, I had to look at both the .cf file and the custom shell script in a completely different directory.

With Puppet, these things can be done relatively simply inside the same class file.  Also, Puppet can be extended simply through the use of defines (think macros) or complexly through the use of modules.  Additionally, Puppet supports templating, classes, inheritance, and explicit order.  Where with cfengine I’d have to do something like this:

copy:
  s_snmpd.dc_has_snmp::
     $(distribute)/snmpd.conf	dest=/etc/snmp/snmpd.conf mode=644

				server=$(policyhost)
				type=sum
				define=dc_restart_snmpd

shellcommands:
   s_snmpd.dc_restart_snmpd::
	"/sbin/service snmpd restart"

class ssh {
    package {
        [ "openssh-clients", "openssh-server" ]:
        ensure => latest
    }
    file { "/etc/ssh/sshd_config":
        mode  => 0600,
        owner => root,
        group => root,
        mode => 644,
        require => Package["openssh-server"],
        content => template("sshd_config.erb")
    }
    service { sshd:
        subscribe => File["/etc/ssh/sshd_config"],
        ensure    => running,
        enable    => true
    }
}

 

With this syntax it’s easy to read that the file /etc/ssh/sshd_config is dependent on the openssh-server package and that the sshd service is dependent on that file.  Puppet also feels more “cross-platform” as the “service” directive allows me to abstractly describe the service without having to hard code a call to /sbin/service.

Puppet is not without it’s drawbacks.  The first of which is that it is Ruby.  If you’re not using Ruby on your systems, this means more package installations on those servers.  If you’ve been programming in another language, like Perl or Python, it’s another language you have to fight with.  The memory usage is much higher than I expected.  On some virtual servers, this may be a huge drawback. Consider:

Memory usage for puppetmasterd

Not too bad, but this is shocking:

Memory Usage at ~250 MB prior to restart

Compare this to cfegine:

Memory Usage for cfservd, Yes, Memory Leak.

and:

Memory Usage for cfexecd

Hell, even a long running Perl program using POE and Net::Pcap to decode all packets on our uplink at work (which bursts to ~75mb/sec) isn’t using that much memory:

Memory Usage for PoCo::Pcap based Traffic Inspector

Ultimately, RAM is cheap and my time is expensive.  After kludging together configuration management in cfengine for the past three years, I’ve decided to ditch it in favor of a more sane and extensible configuration with Puppet.  I’ve got a lot to learn about Puppet still, so as I learn new and more exciting things and Puppet grows, I’ll be sure to share how it’s helping.

Configuring the cfengine master server, with cfegine!

The easiest way to do secure file transfer without passwords would be ssh + public key authentication. This will grant us a reasonable level of security, which we can fine tune with products like scponly. For now, we’ll just play around with basics.

The first thing to do is to setup a user on your cfengine server to accept the file transfers. Make this user unprivileged and make sure they are allowed to login with ssh. I restrict ssh connectivity using groups. I have a special group for utility accounts on my servers called ‘localssh’. I’m going to create a user named ‘util’ to handle this setup.

cfmaster# adduser -n -g localssh -h /home/util util

We need passwordless authentication, so we’re using ssh-keys. However, we don’t want to generate those keys as they will be too much work. We also want to make sure we keep that key under lock and barrel to ensure it’s safety. I’ll use cfengine to configure the master server, and regenerate a utility key everyday. This will ensure limited exposure of the key on the network.

Here’s the master section of our cfengine copyback.cf:

groups:
  hg_cfmaster     = ( cfmaster.domain.com )

control:
  any::
    util_keydir     = ( /usr/local/cfkeys )
    util_privkey   = ( /usr/local/cfkeys/util.dsa )
    util_pubkey   = ( /usr/local/cfkeys/util.dsa.pub )
    util_updir       = ( /home/util/cfin )
    actionsequence = ( directories tidy shellcommands )

directories:
  any::
    $(util_keydir)        mode=700 owner=root group=root fix=all

  hg_cfmaster::
    /home/util              mode=700 owner=util group=localssh fix=all
    $(util_updir)           mode=700 owner=util group=localssh fix=all

tidy:
  hg_cfmaster::
     $(util_keydir)   pattern=util.dsa age=1 r=0 define=dc_util_genkey

shellcommands:
  dc_util_genkey::
     "/usr/bin/ssh-keygen -t dsa -b 1024 -N '' -C 'util@domain.com' -f $(util_privkey)"

copy:
  hg_cfmaster::
    $(util_pubkey)      dest=/home/util/.ssh/authorized_keys mode=600 owner=util group=localssh type=sum

What have we done!?@?!@?

Well, the control and groups sections setup our variables. The ‘directories’ section creates the directories and makes sure the permissions are nice and tight. This ensures that cfengine keeps it that way.

The neat trick is my use of “dynamic classes” to take care of key regeneration. The tidy section looks in the $(util_keydir) for anything matching “util.dsa” and removes it if it’s older than 1 day old. The “define” section defines a dynamic class for the tidy statement if and only if files were deleted.

Then in shellcommands, if our dynamic class “dc_util_genkey” is active, we issue the ssh-keygen command to create our new key.

Last, the copy section moves the generated public_key into the ~/.ssh/authorized_keys file for our util user. This enables the key for logging in without a password. We can get fancier, but like I said, for now, its simple.

Distributing the private key to the clients

The cfengine clients are going to need the private key to be able to authenticate to our cfmaster server. This is a quick addition in the aforementioned ‘copy’ block so it looks like this:

copy:
  hg_cfmaster::
    $(util_pubkey)      dest=/home/util/.ssh/authorized_keys mode=600 owner=util group=localssh type=sum

  !hg_cfmaster::
     $(util_privkey)     dest=$(util_keydir) mode=0600 owner=root group=root type=sum server=$(policyhost)

That’s it. Now all the clients will decide if they need the key based on the checksum and replace it as a newer copy becomes available. So, now we have an account that we can use to send files back to our cfengine master server.

Sending a file to our cfengine master server

What do we do now? Well, I used this technique to issue a certificate request to my cfengine master server for a security tool called OSSEC-HIDS. This meant cfengine could manage the configurations and keys from my clients, making deployment completely automated. Here’s an example using the key to scp a file back:

shellcommands:
  !hg_cfmaster::
    "/usr/bin/scp -i $(util_privkey) /tmp/somefilewithinformation.txt util@$(policyhost):~/$(host).txt"

There ya go! I’ll be putting up a page on the OSSEC-HIDS Wiki on how I used this technique to manage all my clients configurations relatively soon.