• Passwords
• Storage Devices
• Certificates (if you are using Encryption)

Well, over the past few week’s I’ve been able to work around this problem by storing additional information in a CouchDB DB.

It is not the ideal solution, but it is a start and I’m okay with that. I should also warn you, I do HORRIBLE things here with Bash and JSON. Since Bash doesn’t know about JSON, I rely upon awk. I know, I know, I should re-write all of this in a nice new language like Python or Ruby…

First thing is first, I had to create a new database:

$ curl -kX PUT https://puppet.bayphoto.local/bacula_meta
{"ok":true}

I’m not going to worry about that name, to me it is a database that contains some metadata of our clients.

My client creation tool that I posted in my previous Bacula article has been updated to do a little bit more. Aside from no longer using TEMPLATE files, I’ve added some additional code to push a few details into this new bacula_meta database. Here is my “write_json” function:

# Some NEW Variables:
export COUCH_SERVER="https://puppet.bayphoto.local"
export DB="bacula_meta"
export CERTDIR="$BDIR/certs"
 
write_json()
{
   curl -H "Content-Type: application/json" -kX PUT -d '{ "_id": "'${HOSTNAME}'","passhash": "'${PASSHASH}'" }' $COUCH_SERVER/$DB/$HOSTNAME
}

The document I create is simple, it is named after the short hostname of the client added to backups, and for this first run we store that and the password.

The “main” function of the script first tests to see if a document in the bacula_meta db exists, and if not it will create a new client. If it does exist, you can either continue and re-create the bacula client’s configuration, or quite:

TEST=`curl -k -s -X GET $COUCH_SERVER/$DB/$HOSTNAME`
if [[ $TEST == *not_found* ]]
then
       # Generate a bacula password.
       export PASSHASH=`dd if=/dev/random bs=6 count=4 2>/dev/null | openssl base64`
 
       # This is the actual .conf configuration
       print_client_conf
 
       # Create a new client document in $DB
       write_json
 
       # Create SSL key-pair
       create_keys
 
       # Adding the client .conf file for the director to source.
       echo \@$BDIR/clients.d/$HOSTNAME.conf >> $BDIR/clients.conf
 
       echo 'created client definition: '$BDIR/clients.d/$HOSTNAME.conf
       echo 'for '$HOSTNAME
else
       echo 'client '$HOSTNAME 'already exists.'
       echo 'Do you want to override the current configuration for:'
       echo '      '$HOSTNAME
       read -p "[y/N] " prompt
       prompt=${prompt,,}
       if [[ $prompt =~ ^(yes|y)$ ]]
       then
               # if we choose to override, a new client conf will be generated and added and commited.
 
               # Lets re-obtain our stored client password first
               export PASSHASH=`curl -k  -X GET $COUCH_SERVER/$DB/$HOSTNAME | awk -F: 'gsub("{|}","") { print $5 }'`
 
               # print out a new cliend.conf
               print_client_conf
 
               # Push the clients key-pair back to couchdb       
               curl -k -X PUT $COUCH_SERVER/$DB/$HOSTNAME/$FQDN-fd.pem?rev=$DOC_REV --data-binary @$CERTDIR/$FQDN-fd.pem  -H "Content-Type: application/octet-stream"
 
                grep -w $HOSTNAME $BDIR/clients.conf
                if [ $? -eq 0 ]
                then
                        echo 'client '$HOSTNAME 'already exists...'
                else
                        echo \@$BDIR/clients.d/$HOSTNAME.conf >> $BDIR/clients.conf
                fi
 
       else
               echo "Ok, no clients were modified or added!"
       fi
fi

The other addition was a create_keys function. Our clients encrypt their data to the storage node (we send some backup volumes to S3 storage, which is over http and not stored in any sort of encrypted format), and we needed a decent way to distribute the keys (using Puppet…).

This was difficult for me to do. What I failed to understand about adding attachments to CouchDB is you have to reference the current document _rev, and after a LOT of trial and error I finally got it. The DOC_REV variable grabs the current documents revision:

DOC_REV=`curl -k -s -X GET $COUCH_SERVER/$DB/$HOSTNAME | awk -F ':|"' '{ print $10}'`

Which is then used when I actually PUT the file in there:

curl -k -X PUT $COUCH_SERVER/$DB/$HOSTNAME/$CN-fd.pem?rev=$DOC_REV --data-binary @${CERTDIR}/$CN-fd.pem  -H "Content-Type: application/octet-stream"

create_keys()
{
  DOC_REV=`curl -k -s -X GET $COUCH_SERVER/$DB/$HOSTNAME | awk -F ':|"' '{ print $10}'`
  C="US"
  ST="California"
  L="Santa Cruz"
  O="Bay Photo Lab"
  OU="IT"
  CN="${HOSTNAME}.bayphoto.local"
  EMAIL="bayit@bayphoto.com"
 
openssl genrsa -out ${CERTDIR}/${CN}.key 2048
openssl req -new -key ${CERTDIR}/${CN}.key -x509 -out ${CERTDIR}/${CN}.cert <<EOF
${C}
${ST}
${L}
${O}
${OU}
${CN}
$EMAIL
EOF
echo ""
 
cat ${CERTDIR}/${CN}.key ${CERTDIR}/${CN}.cert > ${CERTDIR}/${CN}-fd.pem
 
curl -k -X PUT $COUCH_SERVER/$DB/$HOSTNAME/$CN-fd.pem?rev=$DOC_REV --data-binary @${CERTDIR}/$CN-fd.pem  -H "Content-Type: application/octet-stream"
}

So what does adding a new client look like using this updated tool?

# ./cclient.bash -s Standard -h client-a
INSERT 0 1
{"ok":true,"id":"client-a","rev":"1-0841684988ec85c6d2b16cb941a739ac"}
Generating RSA private key, 2048 bit long modulus
..............................................................+++
..............+++
e is 65537 (0x10001)
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:State or Province Name (full name) [Some-State]:Locality Name (eg, city) []:Organization Name (eg, company) [Internet Widgits Pty Ltd]:Organizational Unit Name (eg, section) []:Common Name (eg, YOUR name) []:Email Address []:
{"ok":true,"id":"client-a","rev":"2-21d4e7bc019c2176dfa2583b320387ab"}
created client definition: /usr/local/etc/bacula/clients.d/client-a.conf
for client-a

And my new record in CouchDB has all the right data:

curl -kX GET https://puppet.bayphoto.local/bacula_meta/client-a
{"_id":"client-a","_rev":"2-21d4e7bc019c2176dfa2583b320387ab","hostname":"client-a","passhash":"y9WBgacrd8JbZjrefeZHKbPk9Kda5UQc","_attachments":{"client-a.bayphoto.local-fd.pem":{"content_type":"application/octet-stream","revpos":2,"digest":"md5-kqi8ODloPxT6D6IxZbCoVg==","length":3411,"stub":true}}}

Thats ugly… how about a nice screenshot!

Now that we have the Bacula tool pushing passwords and and certificates, we need to get Puppet to pull the data.

I found a github project called couchdblookup:
https://github.com/camptocamp/puppet-couchdb/blob/master/lib/puppet/parser/functions/couchdblookup.rb

I placed that couchdblookup.rb file into one of my Puppet modules (etc/puppet/environments/production/bacula/lib/puppet/parser/functions/couchdblookup.rb), and created a bacula::fd::cert class:

 
class bacula::fd::cert inherits bacula::fd {
 
  # Pull bacula client password from our
  # CouchDB server
  $couchdb_bind_address = "puppet.bayphoto.local"
  $couchdb_port = "5984"
  $couchdb_base_url = "https://${couchdb_bind_address}:${couchdb_port}"
  $bacula_meta = "${couchdb_base_url}/bacula_meta/${hostname}"
  $bacula_fd_cert = "${couchdb_base_url}/bacula_meta/${hostname}/${fqdn}-fd.pem"
 
  $bacula_fd_passhash = couchdblookup($bacula_meta, "passhash")
 
  file { "master.cert":
    name    => $operatingsystem ? {
      FreeBSD  => "/usr/local/etc/bacula/certs/master.cert",
      windows  => "C:\Program Files\Bacula\master.cert",
      default  => "/etc/bacula/certs/master.cert",
    },
    owner   => 0,
    mode    => 0640,
    source  => "puppet:///bacula/master.cert",
  }
 
  exec { "fd.cert":
    path    => ["/usr/bin","/usr/local/bin","/bin","/sbin","/usr/sbin","/usr/local/sbin","/usr/local/libexec","/usr/libexec"],
    command => $operatingsystem ? {
      FreeBSD  => "fetch -o /usr/local/etc/bacula/certs/${fqdn}-fd.pem $bacula_fd_cert",
      windows  => "C:/scripts/curl.exe -sk $bacula_fd_cert -o \"\Program Files\Bacula\\${::fqdn}-fd.pem\"",
      default  => "curl -sk $bacula_fd_cert -o /etc/bacula/certs/${fqdn}-fd.pem",
    },
  }
 
}

As you can see, I’m also working on getting Windows systems into our Puppet environment.

It is incredibly immature right now, and Windows lacks a lot of tools I take for granted. It would make my life a lot easier if Microsoft just tool all the BSD licensed userland tools like diff, fetch (or curl), md5, ssh, etc… to make my Puppet automation easier. You NEED diff.exe to use Puppet on windows, otherwise templating won’t work.

Aside from the windows side of things being a pain, this has been working out well enough.

Aside from some very simple system configuration management (I’ve not yet dived too deep into puppet. I mostly use it for configuring system authentication and ensuring a particular computer security baseline), I though it would have been great to store the client’s “facts” into a accessable database.

Sometime last year, Puppet added the ability to store facts into a Couch Database: http://www.puppetlabs.com/blog/couchdb-facts-terminus-for-puppet/

I’m not using CentOS/RHEL or a Yum backed package system. I’m using FreeBSD with its Ports tree, so here is the equivalent command:

$ portinstall databases/couchdb

Why one instead of the 3? Because Ruby 1.8 already exists on the system, as it get installed with the necessary tools such as portupgrade. That also includes the gem tool.

The next component the article listed was the couchrest rubygem:

$ gem install couchrest -v1.0.0

That is where I had a slight problem. FreeBSD has a bunch of rubygem-* Ports, but not this one.

It is not that I can’t install the couchrest gem this way, FreeBSD can use Gem’s just like any OS. My problem is when you move out of the package management system your updating process ends up fragmented. I try very hard not to let this happen.

So how do I continue? I make a Port!

This was actually REALLY simple. First, I found another rubygem based port and copied it to a new structure called databases/rubygem-couchrest. I pulled up the FreeBSD’s Porters Handbook and followed the quick instructions.

What is pretty handy about the Ports tree is all the heavy lifting it does for you. For example, if a project is hosted on SourceForge, you can define the MASTER_SITE variable as:

MASTER_SITE=    SF

And it will look up the project on SF (using other variables such as PORTNAME and PORTVERSION).

Ruby Forge is also supported, so I just had to set the MASTER_SITES to RG and define the name and version. It took all but 5 minutes to make, build a checksum, and create this shell archive:

# This is a shell archive.  Save it in a file, remove anything before
# this line, and then unpack it by entering "sh file".  Note, it may
# create directories; files and directories will be owned by you and
# have default permissions.
#
# This archive contains:
#
#	rubygem-couchrest
#	rubygem-couchrest/Makefile
#	rubygem-couchrest/distinfo
#	rubygem-couchrest/pkg-descr
#
echo c - rubygem-couchrest
mkdir -p rubygem-couchrest > /dev/null 2>&1
echo x - rubygem-couchrest/Makefile
sed 's/^X//' >rubygem-couchrest/Makefile << '656a34e951167ae5d8b3571b3b30bce9'
X# Ports collection makefile for:	rubygem-couchrest
X# Date created:				20 June 2011
X# Whom:						Mike Carlson (carlson39@llnl.gov)
X#
X
XPORTNAME=	couchrest
XPORTVERSION=	1.0.2
XCATEGORIES=	databases rubygems
XMASTER_SITES=	RG
X
XMAINTAINER=	ruby@FreeBSD.org
XCOMMENT=	provides a simple interface on top of CouchDBs RESTful HTTP API
X
XRUN_DEPENDS=	rubygem-json>=${PORTVERSION}:${PORTSDIR}/devel/rubygem-json \
X		rubygem-mime-types>=${PORTVERSION}:${PORTSDIR}/misc/rubygem-mime-types \
X		rubygem-rest-client>=${PORTVERSION}:${PORTSDIR}/www/rubygem-rest-client
X
XUSE_RUBY=   yes
XUSE_RUBYGEMS=   yes
XRUBYGEM_AUTOPLIST=  yes
X
X.include <bsd.port.mk>
656a34e951167ae5d8b3571b3b30bce9
echo x - rubygem-couchrest/distinfo
sed 's/^X//' >rubygem-couchrest/distinfo << 'fed48567690d00cc0666b4be7236926b'
XSHA256 (rubygem/couchrest-1.0.2.gem) = 55b62424feedba1cb936d7187c94875fbea8daf17962813deba0cf0c41c93c72
XSIZE (rubygem/couchrest-1.0.2.gem) = 45056
fed48567690d00cc0666b4be7236926b
echo x - rubygem-couchrest/pkg-descr
sed 's/^X//' >rubygem-couchrest/pkg-descr << '72aefcf6ac70736e90967f68ab1bb671'
XCouchRest provides a simple interface on top of CouchDB's RESTful HTTP API, as well as including some utility scripts for managing views and attachments.
X
XWWW: http://rubygems.org/gems/couchrest
72aefcf6ac70736e90967f68ab1bb671
exit

After running the gamut of “make” tests listed in the handbook, I submitted a PR with the command-line tool called send-pr(1) that is build in to FreeBSD. You can see it in the queue here

I don’t know if it will get approved, or how long it will take, but I’m happy that I took a step in contributing to the project. I’ve filed a few bugs, and I write about FreeBSD a lot, so this seems to be a good logical progression.

The end result is now I can follow the Puppet Blog entry on setting up and maintaining a CouchDB store for facts, and have a consistent updating method (via Ports that is). My hope is to have this become a source for our configuration management tool, and help us reconcile and keep hardware and OS information up to date.

What also made this conference unique, is how honest the Puppet team and community were about the projects strength and weaknesses. Those that have deployed Puppet on a larger scale (MessageOne and Google) seemed to go through the same iterations in attempting to scale out their Puppetmaster’s. From WEBrick (which is what I’m currently running Puppet with :) ), which is hated by all since its a single process/thread web server that can only handle one request at a time. To Mongrel, which you have to manage a mongrel cluster script, feed it lots of memory, and then throw an apache proxy server in front of them. Now, people are starting to settle on using Passenger/mod_rack, which is what I spent most of yesterday looking into and setting up. This allows apache to mount a rails instance, and then you don’t actually have to run puppetmasterd. This still requires some decent hardware, and I’m currently running my puppetmaster on a VM with 2GB or memory, so I’ll have to watch out for that. Chris, the one who introduced me to Puppet, said he still uses WEBrick for all of his DB, Tomcat, and Apache servers (I think he said something like 200 systems) and it has been working out nicely. He, like the guys at Google, also doesn’t run puppet as a daemon.

Anyway, the point is, we learned a lot about the project, way more than if a sales person had come to us and just told us the things puppet does well, or how it operates on paper (*cough* LANDesk *cough*). It was really awesome to talk with Andrew Pollock and Nigel Kersten from Google. See, I was a little unsure about Puppet in our environment, where we have multi-purpose servers, computer servers, and desktops that we have to manage. It seemed, at a first glance, that most of the Puppet users out there have a homogeneous environment, and Andrew (Shafer) had stressed the concept of single role servers. After talking with them, I felt a lot more comfortable pursuing Puppet across our servers and desktops. Did I mention they were super cool and friendly?

We also learned a lot about the Puppet developers, which had its own interesting advantage. I have a lot of respect for what Luke Kanies has been able to do, and by the end of the conference, he showed significant mastery in what he has done, as well as some humility and admitting what he has not been able to do and why. I was a little put off the first day though, when both him and Andrew came off a little arrogant and crass. It did make me step back and think, “Is this project going to be well managed in the future with personalities like this in charge? Is their answer of ‘don’t do that!’ tongue in cheek, or are they not supportive of a diverse environment?”. In the end, I have more respect for the project than ever, and with it still being a young project, I hope they listened to some of the feedback, and I also can’t wait to see where it ends up in the next year.

Andrew, the Puppet Andrew, came up to us a lot during the conference, and he was fun to talk too, and he’s very academic and he had a lot of abstract concepts to talk about. Also, he said this was the first conference he has arranged, and I think he did a fantastic job. Jenny had commented that this was the first conference she had lasted the entire duration, so that says a lot about the pacing and content of PuppetCamp. I felt the same way, every session was incredibly engaging, and how Andrew had setup the democratic and chaotic Open Sessions was very impressive. Lets put it this way, I even got up there and pitched a topic, which is something I would have never done. Hurray for me stepping outside of my comfort zone!

Warning: side topic!

Now that I’ve had the weekend to google all the cool technologies I was exposed too, I’m also reminded why I really like having a FreeBSD server at my disposal. They had talked about CouchDB, so on a whim I did a

~> cd /usr/ports
/usr/ports> make search name=couchdb
Port: couchdb-0.9.0_1,1
Path: /usr/ports/databases/couchdb
Info: A document database server, accessible via a RESTful JSON API
Maint: till@php.net
B-deps: ca_root_nss-3.11.9_2 curl-7.19.6_1 erlang-lite-r13b01_6,1 gettext-0.17_1 gmake-3.81_3 icu-3.8.1_2 libiconv-1.13.1 libtool-2.2.6a nspr-4.8 perl-5.8.9_3 spidermonkey-1.7.0
R-deps: ca_root_nss-3.11.9_2 curl-7.19.6_1 erlang-lite-r13b01_6,1 gettext-0.17_1 gmake-3.81_3 icu-3.8.1_2 libiconv-1.13.1 libtool-2.2.6a nspr-4.8 perl-5.8.9_3 spidermonkey-1.7.0
WWW: http://couchdb.org/

Port: py26-simplecouchdb-0.9.26
Path: /usr/ports/databases/py-simplecouchdb
Info: Simple Librairy to Allow Python Applicationto Use CouchDB
Maint: wenheping@gmail.com
B-deps: py26-httplib2-0.5.0 py26-py-restclient-1.3.2 py26-setuptools-0.6c9 python26-2.6.2_3
R-deps: py26-httplib2-0.5.0 py26-py-restclient-1.3.2 py26-setuptools-0.6c9 python26-2.6.2_3
WWW: http://code.google.com/p/py-simplecouchdb/

I did a ‘make install’, and I had a cool little couchdb up and running. What is also cool is FreeBSD likes to give you very helpful information when you install something. For example, this is what is printed out when you install the CouchDB port:

===> COMPATIBILITY NOTE:
CouchDB is still pre-stable; between 0.8 and 0.9 the database format
changed which breaks BC. In current trunk, the format changed again, so
please double-check in case you are updating an existing installation.

More info:
* http://wiki.apache.org/couchdb/Breaking_changes?action=show&redirect=BreakingChanges
* http://wiki.apache.org/couchdb/BreakingChangesUpdateTrunkTo0Dot9

See, isn’t that helpful? Best of all, I didn’t have to enable additional repositories, or fetch the src manually, and its dependencies and then figure out how to run the right configure script flags… FreeBSD makes it easy, and since it automatically uses what you already have with what is required, its an incredibly stable build. Removing it is pretty simple as well, just:

> pkg_deinstall -R couchdb
---> Deinstalling 'couchdb-0.9.0_1,1'
---> Deinstalling 'erlang-lite-r13b02,1'
[Updating the pkgdb
in /var/db/pkg ... - 118 packages found (-1 +0) (...) done]
---> Deinstalling 'curl-7.19.6_1'
[Updating the pkgdb
in /var/db/pkg ... - 117 packages found (-1 +0) (...) done]
---> Deinstalling 'ca_root_nss-3.11.9_2'
---> Deinstalling 'spidermonkey-1.7.0'
---> Deinstalling 'nspr-4.8'
[Updating the pkgdb
in /var/db/pkg ... - 116 packages found (-1 +0) (...) done]
---> Deinstalling 'gmake-3.81_3'
[Updating the pkgdb
in /var/db/pkg ... - 115 packages found (-1 +0) (...) done]
---> Deinstalling 'perl-threaded-5.8.9_3'
[Updating the pkgdb
in /var/db/pkg ... - 114 packages found (-1 +0) (...) done]
---> Deinstalling 'gettext-0.17_1'
---> Deinstalling 'libiconv-1.13.1'
---> Deinstalling 'icu-3.8.1_2'
---> Deinstalling 'libtool-2.2.6a'
** Listing the failed packages (-:ignored / *:skipped / !:failed)
! curl-7.19.6_1 (pkg_delete failed)
! ca_root_nss-3.11.9_2 (pkg_delete failed)
! perl-threaded-5.8.9_3 (pkg_delete failed)
! gettext-0.17_1 (pkg_delete failed)
! libiconv-1.13.1 (pkg_delete failed)

This does a upwards recursive dependency removal. Also, if one dependency is relied on by another, it wont get removed. Like, if Perl58 was a dependency of a package, it wouldn’t be removed if perl58 is used by many other packages. This is smart. So, above, the packages that failed to deinstall where ones that are required dependencies of other installed packages.

Speaking of package management; have you ever installed something that ended up having a few dozen dependencies, then you want to uninstall that package with a “rpm -e cba8″, or something equivalent, but what about all the other cruft that came along with it? You would have to keep track of each dependency, and specify all of them and hope you don’t break another program. FreeBSD has a few tools to do this, one in particular, portmaster can remove all ports that were once a dependency but no longer used:

> portmaster -s
Information for neon28-0.28.4:
Comment:
An HTTP and WebDAV client library for Unix systems
===>>> neon28-0.28.4 is no longer depended on, delete? [n] y
===>>> Delete old and new distfiles for www/neon28
without prompting? [n] y
===>>> Running pkg_delete -f neon28-0.28.4
Information for rubygem-actionwebservice-1.2.6:
...

I ended up removing 4 packages that were no longer used.

CentOS and RHEL are the larger Puppet consumers, I’m still a big proponent for FreeBSD, and at work, it has allowed me to quickly build an Apache + Puppet + RubyPassenger/mod_rack stack with the minimal dependencies installed. So, the puppet server is still pretty lean, which means updates are smaller and faster. It still surprises me that its relatively unknown, even though Netcraft always has it listed in the top domains with the best uptime and consistently growing over the years. Why do I feel like an AmigaOS fan sometimes?

Hmm, it is sort of weird that this turned into a FreeBSD ports management entry :)

Okay, final word: PuppetCamp09 was Freaking awesome. There were a lot of smart developers and sysadmins there. We even got a very cool git howto, which I found useful. It was very diverse, which is strange for a conference based on one project in particular.