Our motivation has not been the more difficult part, nor has our muse left us ( The two of us have been able to work out some really fun segments ). I’ll have to post some of it when we can get a decent recording, right now we have been using my phone to record which only works to remind us of what we have done.

The only problem we’ve had is finding a solid place to practice.

Between the drum kit and my guitar, we generate a fair amount of noise.

I mean “structured noise”, not just regular noise. Noise with a rhythm.

Our first foray was the Tattoo Shop after hours. At the time, the shop was closed on Tuesday so we could sneak in there to practice. The problem with this spot was: we had to setup, and lugging around a drum set, setting up and finally, tearing it down and driving it back to Glenn and Erin’s soaked up at least an hour of our time. On a Tuesday night, loosing an hour is detrimental to me.

Glenn and Erin recently moved, so in between places, we jammed in the empty front room:

Blurry Glenn

After some though, Glenn and Eric decided to rent out a space in Seaside that was supposed to be art and music friendly. It was located in an industrial part of town where it seemed no one was around:

Plenty of space for the two of us.

We were incredibly excited! Setup was as easy as my unloading my amp and hooking it up. We had no distractions, privacy, and we could pin up song notes and new ideas on the wall.

Then, after our first practice, we heard a knock on the door…

A mother, who was clearly worked up asked us to stop and that we were disrupting her daughters sleep.

Both Glenn and I fell over each other in sincere apologies. We had no idea that people even lived in that area. So, after wrapping it up, and talking it over, we agreed that ~11pm was too late and we would wrap it up around 9:30pm the next time. He also wrote up a nice flier for the two houses we found within a two block radius:

Hello Neighbor!

I very recently began renting a studio space on Redwood and I play music a few days a week, electric on Tuesdays and acoustic on M, W, Fr. Please let me know if the noise is too loud or disruptive! Please feel free to call or text me if I am being too loud and I will gladly accommodate!

Thanks All! Be Well!
<3 Glenn

His number was included (I omitted it here for privacy sake) and we both were excited to work things out.

Two days later, the Manager of the studio contacted Glenn and said there had been a complaint about the noise. He was very understanding, and he had even know that Glenn posted fliers (this was admitted to the manager from the “anonymous” individual). He said that even though it was part of the rental agreement that music was allowed, he would have to make sure we did not continue if there were any more complaints.

We had two more sessions after that. On the last one, we felt great about everything. We stopped at 9:30pm, we did as much as we could to diffuse and direct the sound. Our music was just starting to find its own voice, and we’d hi-five each other and say how cool this was.

The next Friday, we received the final complaint. The “anonymous” submitter said we kept her daughter up late, and we were so loud she had to turn up her TV to compensate.

At this point, we were a bit frustrated. The person never attempted to work things out with us. We gave her the easiest and guilt free mode of communication (a text message) to ask us to stop. We could have started practice earlier and stopped earlier had she given us the chance to work things out. We have to stop playing there and find a new place.

So, for at least a few weeks, we can play at a farm in Aromas:

Outside in nature! And no, I wont take better pictures, my phone sucks as a camera in most conditions but its all I have

Here is a better one from this week. Glenn plays the drums and keyboard, such a talented musician. I just hold the guitar.

I’ll bust out the ol’ headlamp, and I promise not to blind anyone.

After the farm, who knows were we will go. We could scale it back to his living room again, and we pull out the micro-amps. Its not the same though, it feels good to have my amp turned up a little (saturate those tubes!) and the to feel the kick of the bass drum is nice.

Speaking of that, when we first practiced, I was so unaccustomed to the sound of a real live and up close drum set that I thought I would pee’d myself… just a little… like a puppy startled by a loud sneeze. I have never played at a high volume, nothing beyond what only I could hear in my garage. That is also a startling realization, that I actually have to PLAY and be HEARD by more than just me :)

That made me spring a leak just thinking about it…

Performance wise, I was always fairly happy with ESXi knowing that it was a virtual machine of course. There were a few issues I recall:

• silent data corruption (fsck’ing Linux and FreeBSD volumes would reveal inconsistent filesystem information, but there were never any errors reported to the VM)
• Live migrations were not always stable. In fact, most of the time it would result in an unexpected shutdown

I doubt it was solely VMWares fault, as it could have been a series of mis-configurations and poor implementation descisions. Again, that is not a black mark against VMWare, it is just part of my own personal user experience.

What IS a black mark against VMWare is how expensive their product is, and I found out recently that version 5 now charges by how much memory your servers have (and not by processor socket, which is how I think it used to be). vRAM as they like to call it.

This is a problem in our environment, as memory is really valuable in a virtualized environment; shared disk subsysterms will ALWAYS be an order of magnitude slower than having dedicated disks. By allowing the OS’s virtual memory layer to do all of the caching in RAM, we can load up a hypervisor server with as much ram as reasonably priced and give a VM a few gig’s to work with.

That is what we did at Bay Photo Lab :)

We spec’d two servers with 4 AMD Opteron 6220 processors, each with 128GB of system memory each.

That gave us 64 cores, and ~254GB of usable memory (Dom0 uses ~2GB). The price point for using AMD over Intel was too much to pass up. We were able to get double the amount of resources for nearly half the price.

Rob had already built up a nice GlusterFS cluster, so we use the Gluster FUSE client on our Xen servers to access the shared storage. This works out nicely, it isn’t the fastest storage (currently we are using a 1Gb network), however, we have the option to upgrade various points like moving to InfiniBand or 10Gb ethernet.

Before this new environment, we were manually creating a Xen configuration file. This is OK for maybe Rob and I. This is not OK for anyone else though, so I started looking for a user interface.

The simple route was to install Virt Manager, VNC Server (or use SSH tunneling). That still would only work for Rob and myself, as access to the Xen servers should be limited.

I came across Convirture, with has a free/open source edition of their hypervisor manager. Convirture uses libvirt, and will work with either KVM or Xen.

They also offer a nice Appliance you can download and run “xm create” on one of your hypervisor servers. After messing around with rolling my own version, I decided that the virtual appliance was much better to use.

Main Page. Here you can quickly see the status of your VM's.

FYI: The VM’s that are greyed out are VMs that were created outside of Convirture. I had created one VM on the new Xen cluster before Rob had created the new FreeBSD Template to install from. I’ll have to convert it at some point.

Speaking of Templates, we only have a few. This is still a growing project.

Connecting to a Virtual Machine's console is SUPER easy.

You can either choose to launch the Java applet (works great on Windows), or, you can choose to use your own VNC client. Since I decided not to install Java on my FreeBSD Desktop, I chose to connect with my own VNC viewer. I simply open up a Shell after I tell Convirture to connect me to a console and type:

mikec@b-bot ~> vncviewer convirt.bayphoto.local:6900

And I’m presented with this wonderful window:

A simple and fast console

From there, I can install portinstall + puppet and then its all up to our node definition to do the rest.

There are a lot of things you can do with the Open Source version, and after some configuration file tuning, I was able to get Live Migrations to work.

Templates are really easy to work with, and I think I’d like to get a base FreeBSD image created that already has Puppet installed and ready to go. That would make deploying new VM’s a lot faster. Same with Windows too.

Now the fun part… How much trouble was setting all of this up?

As I mentioned above, we use Xen here. To me at this point, it honestly doesn’t matter if KVM is heads over heels better than KVM. What matters is we already have a stable environment.

I tried various Distro’s: Debian, CentOS, Fedora and Scientific Linux

I also tried Two major kernel branches: 3.2 and 2.6.32

There was a problem getting GlusterFS to work on Debian versus Fedora, and Fedora is a KVM environment, and I wanted to jump through as little hoops as possible.

It took a few days to notice, but GlusterFS would not work reliably on 3.2, the system would simply reboot when I tried to write data out to the gluster filesystem. No warning, no nice error log, nothing.

In the end, I fell back to a RHEL based Distro (Scientific Linux), as Gluster seems to work well with a RHEL based OS. Also, since my Puppet manifests already have CentOS, it was easy for me to shove Scientific Linux in those modules.

I also fell back to using the 2.6.32 kernel, which is also why I was unable to use Debian. It only had Xen 4 and 3.2 kernels to choose from.

Scientific Linux is RHEL based, so it has KVM packages. Not Xen packages. So, I followed this guide:
http://www.crc.id.au/xen-on-rhel6-scientific-linux-6-centos-6-howto/

Then I found out Libvirt does not have Xen support build in, so I had to use this guide as well:
http://wiki.xensource.com/xenwiki/RHEL6Xen4Tutorial

Also, Scientific Linux strayed from the RHEL/Linux ethernet device convention, ethX, to emX (what would have been eth0 is now em0). This caused a little hiccup with the configuration I was managing. It was a small annoyance.

Here is my main beef: I tried everything I could to stick with a stock OS that met the following requirements:

• Xen 4.x
• GlusterFS support

That was shockingly difficult to do. In the end I had to re-compile a few packages, and I still get these fun messages:

brctl show
bridge name	bridge id		STP enabled	interfaces
virbr0		/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
/sys/class/net/virbr0/bridge: No such file or directory
8000.52540043692c	yes		virbr0-nic
xenbr0		/sys/class/net/xenbr0/bridge: No such file or directory
/sys/class/net/xenbr0/bridge: No such file or directory
/sys/class/net/xenbr0/bridge: No such file or directory
/sys/class/net/xenbr0/bridge: No such file or directory
/sys/class/net/xenbr0/bridge: No such file or directory
/sys/class/net/xenbr0/bridge: No such file or directory
/sys/class/net/xenbr0/bridge: No such file or directory
/sys/class/net/xenbr0/bridge: No such file or directory
/sys/class/net/xenbr0/bridge: No such file or directory
/sys/class/net/xenbr0/bridge: No such file or directory
/sys/class/net/xenbr0/bridge: No such file or directory
/sys/class/net/xenbr0/bridge: No such file or directory
8000.d067e5fbefa7	no		em1
							tap39.0
							tap40.0
							tap41.0
							tap43.0
							vif39.0
							vif40.0
							vif41.0
							vif43.0

New RHEL 6 based OS’s have support for IGMP inspection, and bridge-utils does not have the proper support for this. Its annoying, and it took up a good day of research to come to a reasonable resolution.

With all of this, I would have really been pleased if BHyVe (“Bee hive”, the BSD hypervisor project) was ready and at production quality.

All of these little issues I had with different kernel, base packages, vendor packages, are just par for the course when dealing with Linux. It can sometimes be mindbogglingly frustrating to deal with such a chaotic environment when you come from a very stable and well documented environment like FreeBSD.

FreeBSD lacks support, and that is for various reasons. The biggest is FreeBSD has less of a market share, a smaller developer and user community, and very VERY little marketing. Linux is still the right tool for this job, but what a sloppy tool it can be. Maybe in a few years, BHyVe will be a viable alternative.

In the end, I’m happy with this environment. I think Xen is great for our environment and works for our needs. Plus, the cost of doing business is much lower when we utilize free software. We can feasibly deploy 32 dual cpu VM’s with 8GB ram each for what costs us about $400 per instance. That is being overly generous too!

I’ve been using our TV for my own nefarious deeds, unbeknownst to Caralyne, as an educational tool in science and skepticism. Between Good Eats, Mythbusters and a few PBS clips of Neil Degrasse Tyson being awesome, something was bound to stick.

Also, as a daughter and child of mine, she is constantly seeking my affection and approval… which I am not above exploiting these features she no doubt got from me, so I give her a LOT of positive feedback and attention when we are watching these shows.

I ask her questions like “What do you think will happen?” and “Why are they doing that?”, and sometimes I’ll ask her to summarize.

It certainly has paid off! She signed up for “Science Day” at school, and when the Schoolastic book catalog came around, she picked out a Mythbusters Experiments book. It is a sweet, kid friendly book that has a collection of myths to bust. Well, some are myths, and some are just plain cool.

Since this was during the time I was going back and forth between Antioch and Santa Cruz, I tried to make our weekends together a little more exciting by doing one of the activities in her Mythbustere’s book. The first was making a Volcano:

Waiting for Mt. El Monte's eruption!

Our eerie blue lava did not startle our fearless Aussie, he in fact licked up as much as he could (i bet it tasted like fish and chips...)

We repeated this 3 to 4 times, it was a lot of fun

I somehow got the idea in my head that I should raise the bar and give something to Caralyne that was a little more challenging to work on. I talked with my buddy Steve and he reminded me that Fry’s has a neat collection of robotic kits, and that has been our latest project :)

We built it together; I did all of the soldering, and she did the rest of the assembly. I think it worked out well. Next time I’ll have her try her hand at soldering, maybe pull put the Volt/Ohm meter and let her verify the resistors.

It took a few weekends, which got stretched out over a few months due to the big move and all, but the process was very rewarding.

Caralyne showing off the circuit board and gearbox

Caralyne with her finished robot :) And Owen... Well, Owen has a bag for a hat. I'm not sure about that kid...

Both of them watched the little robot crawl around, and Caralyne was excited to see that its sensor on the top would react to the environment.

I was actually surprised it worked. It has been a while since I’ve assembled something like this (not since Heald), and I remember when I did this kind of stuff in high school, a few of these kits never worked.

After a few more of these under her belt, I’d like to get her one that you can program. If I can get her started on learning a simple programming language before she turns into a teenager that hates me, I will feel a lot more secure in her future.

• Passwords
• Storage Devices
• Certificates (if you are using Encryption)

Well, over the past few week’s I’ve been able to work around this problem by storing additional information in a CouchDB DB.

It is not the ideal solution, but it is a start and I’m okay with that. I should also warn you, I do HORRIBLE things here with Bash and JSON. Since Bash doesn’t know about JSON, I rely upon awk. I know, I know, I should re-write all of this in a nice new language like Python or Ruby…

First thing is first, I had to create a new database:

$ curl -kX PUT https://puppet.bayphoto.local/bacula_meta
{"ok":true}

I’m not going to worry about that name, to me it is a database that contains some metadata of our clients.

My client creation tool that I posted in my previous Bacula article has been updated to do a little bit more. Aside from no longer using TEMPLATE files, I’ve added some additional code to push a few details into this new bacula_meta database. Here is my “write_json” function:

# Some NEW Variables:
export COUCH_SERVER="https://puppet.bayphoto.local"
export DB="bacula_meta"
export CERTDIR="$BDIR/certs"
 
write_json()
{
   curl -H "Content-Type: application/json" -kX PUT -d '{ "_id": "'${HOSTNAME}'","passhash": "'${PASSHASH}'" }' $COUCH_SERVER/$DB/$HOSTNAME
}

The document I create is simple, it is named after the short hostname of the client added to backups, and for this first run we store that and the password.

The “main” function of the script first tests to see if a document in the bacula_meta db exists, and if not it will create a new client. If it does exist, you can either continue and re-create the bacula client’s configuration, or quite:

TEST=`curl -k -s -X GET $COUCH_SERVER/$DB/$HOSTNAME`
if [[ $TEST == *not_found* ]]
then
       # Generate a bacula password.
       export PASSHASH=`dd if=/dev/random bs=6 count=4 2>/dev/null | openssl base64`
 
       # This is the actual .conf configuration
       print_client_conf
 
       # Create a new client document in $DB
       write_json
 
       # Create SSL key-pair
       create_keys
 
       # Adding the client .conf file for the director to source.
       echo \@$BDIR/clients.d/$HOSTNAME.conf >> $BDIR/clients.conf
 
       echo 'created client definition: '$BDIR/clients.d/$HOSTNAME.conf
       echo 'for '$HOSTNAME
else
       echo 'client '$HOSTNAME 'already exists.'
       echo 'Do you want to override the current configuration for:'
       echo '      '$HOSTNAME
       read -p "[y/N] " prompt
       prompt=${prompt,,}
       if [[ $prompt =~ ^(yes|y)$ ]]
       then
               # if we choose to override, a new client conf will be generated and added and commited.
 
               # Lets re-obtain our stored client password first
               export PASSHASH=`curl -k  -X GET $COUCH_SERVER/$DB/$HOSTNAME | awk -F: 'gsub("{|}","") { print $5 }'`
 
               # print out a new cliend.conf
               print_client_conf
 
               # Push the clients key-pair back to couchdb       
               curl -k -X PUT $COUCH_SERVER/$DB/$HOSTNAME/$FQDN-fd.pem?rev=$DOC_REV --data-binary @$CERTDIR/$FQDN-fd.pem  -H "Content-Type: application/octet-stream"
 
                grep -w $HOSTNAME $BDIR/clients.conf
                if [ $? -eq 0 ]
                then
                        echo 'client '$HOSTNAME 'already exists...'
                else
                        echo \@$BDIR/clients.d/$HOSTNAME.conf >> $BDIR/clients.conf
                fi
 
       else
               echo "Ok, no clients were modified or added!"
       fi
fi

The other addition was a create_keys function. Our clients encrypt their data to the storage node (we send some backup volumes to S3 storage, which is over http and not stored in any sort of encrypted format), and we needed a decent way to distribute the keys (using Puppet…).

This was difficult for me to do. What I failed to understand about adding attachments to CouchDB is you have to reference the current document _rev, and after a LOT of trial and error I finally got it. The DOC_REV variable grabs the current documents revision:

DOC_REV=`curl -k -s -X GET $COUCH_SERVER/$DB/$HOSTNAME | awk -F ':|"' '{ print $10}'`

Which is then used when I actually PUT the file in there:

curl -k -X PUT $COUCH_SERVER/$DB/$HOSTNAME/$CN-fd.pem?rev=$DOC_REV --data-binary @${CERTDIR}/$CN-fd.pem  -H "Content-Type: application/octet-stream"

create_keys()
{
  DOC_REV=`curl -k -s -X GET $COUCH_SERVER/$DB/$HOSTNAME | awk -F ':|"' '{ print $10}'`
  C="US"
  ST="California"
  L="Santa Cruz"
  O="Bay Photo Lab"
  OU="IT"
  CN="${HOSTNAME}.bayphoto.local"
  EMAIL="bayit@bayphoto.com"
 
openssl genrsa -out ${CERTDIR}/${CN}.key 2048
openssl req -new -key ${CERTDIR}/${CN}.key -x509 -out ${CERTDIR}/${CN}.cert <<EOF
${C}
${ST}
${L}
${O}
${OU}
${CN}
$EMAIL
EOF
echo ""
 
cat ${CERTDIR}/${CN}.key ${CERTDIR}/${CN}.cert > ${CERTDIR}/${CN}-fd.pem
 
curl -k -X PUT $COUCH_SERVER/$DB/$HOSTNAME/$CN-fd.pem?rev=$DOC_REV --data-binary @${CERTDIR}/$CN-fd.pem  -H "Content-Type: application/octet-stream"
}

So what does adding a new client look like using this updated tool?

# ./cclient.bash -s Standard -h client-a
INSERT 0 1
{"ok":true,"id":"client-a","rev":"1-0841684988ec85c6d2b16cb941a739ac"}
Generating RSA private key, 2048 bit long modulus
..............................................................+++
..............+++
e is 65537 (0x10001)
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [AU]:State or Province Name (full name) [Some-State]:Locality Name (eg, city) []:Organization Name (eg, company) [Internet Widgits Pty Ltd]:Organizational Unit Name (eg, section) []:Common Name (eg, YOUR name) []:Email Address []:
{"ok":true,"id":"client-a","rev":"2-21d4e7bc019c2176dfa2583b320387ab"}
created client definition: /usr/local/etc/bacula/clients.d/client-a.conf
for client-a

And my new record in CouchDB has all the right data:

curl -kX GET https://puppet.bayphoto.local/bacula_meta/client-a
{"_id":"client-a","_rev":"2-21d4e7bc019c2176dfa2583b320387ab","hostname":"client-a","passhash":"y9WBgacrd8JbZjrefeZHKbPk9Kda5UQc","_attachments":{"client-a.bayphoto.local-fd.pem":{"content_type":"application/octet-stream","revpos":2,"digest":"md5-kqi8ODloPxT6D6IxZbCoVg==","length":3411,"stub":true}}}

Thats ugly… how about a nice screenshot!

Now that we have the Bacula tool pushing passwords and and certificates, we need to get Puppet to pull the data.

I found a github project called couchdblookup:
https://github.com/camptocamp/puppet-couchdb/blob/master/lib/puppet/parser/functions/couchdblookup.rb

I placed that couchdblookup.rb file into one of my Puppet modules (etc/puppet/environments/production/bacula/lib/puppet/parser/functions/couchdblookup.rb), and created a bacula::fd::cert class:

 
class bacula::fd::cert inherits bacula::fd {
 
  # Pull bacula client password from our
  # CouchDB server
  $couchdb_bind_address = "puppet.bayphoto.local"
  $couchdb_port = "5984"
  $couchdb_base_url = "https://${couchdb_bind_address}:${couchdb_port}"
  $bacula_meta = "${couchdb_base_url}/bacula_meta/${hostname}"
  $bacula_fd_cert = "${couchdb_base_url}/bacula_meta/${hostname}/${fqdn}-fd.pem"
 
  $bacula_fd_passhash = couchdblookup($bacula_meta, "passhash")
 
  file { "master.cert":
    name    => $operatingsystem ? {
      FreeBSD  => "/usr/local/etc/bacula/certs/master.cert",
      windows  => "C:\Program Files\Bacula\master.cert",
      default  => "/etc/bacula/certs/master.cert",
    },
    owner   => 0,
    mode    => 0640,
    source  => "puppet:///bacula/master.cert",
  }
 
  exec { "fd.cert":
    path    => ["/usr/bin","/usr/local/bin","/bin","/sbin","/usr/sbin","/usr/local/sbin","/usr/local/libexec","/usr/libexec"],
    command => $operatingsystem ? {
      FreeBSD  => "fetch -o /usr/local/etc/bacula/certs/${fqdn}-fd.pem $bacula_fd_cert",
      windows  => "C:/scripts/curl.exe -sk $bacula_fd_cert -o \"\Program Files\Bacula\\${::fqdn}-fd.pem\"",
      default  => "curl -sk $bacula_fd_cert -o /etc/bacula/certs/${fqdn}-fd.pem",
    },
  }
 
}

As you can see, I’m also working on getting Windows systems into our Puppet environment.

It is incredibly immature right now, and Windows lacks a lot of tools I take for granted. It would make my life a lot easier if Microsoft just tool all the BSD licensed userland tools like diff, fetch (or curl), md5, ssh, etc… to make my Puppet automation easier. You NEED diff.exe to use Puppet on windows, otherwise templating won’t work.

Aside from the windows side of things being a pain, this has been working out well enough.

From one Lab to another

After 9.5 years with one employer (LLNL), I joined Bay Photo Lab in Santa Cruz.

This has brought on many changes, not just a career change but a significant change in my way of life.

First off, I’m renting a room until my family gets down here. I have this weird double life now, where during the week I live alone in a small room with no heating, and then I go back home on the weekend. Not seeing the kids is the hardest part. We typically do a video chat on Wednesday nights, but when I come home Owen is usually a little stand-offish and Caralyne is unhappy during the week (especially when I have to leave). Its sad.

Here was the bathroom in my place for the first week. As in, I did not have one.

Like a kid fresh out of his parents home, I have to do things like laundry, and all of my own dishes.

I'm in no hurry to hurry back

My commute is significantly different. I drive a lot less during the week, I’m about 3 miles from work. Some days I choose to walk, and other days I’ll drive in after jogging back home during lunch. It is pretty awesome, it has been 12 years since I have lived and worked in the same place.

Let’s talk about Bay Photo. I had a slight idea of what they did when I initially interviewed with them IT team here (Rob and Patrick), and I did some simple Google-ing. However, after working here for two months, it is incredibly impressive how much they do and how great the people are.

Bay Photo Lab has a nice small company feel to it, and it is inCREDIBLY friendly here. People are pretty passionate about the work they do here, and so is the IT team.

This has been really awesome, as passion (and really, it should be called “the desire to do the right thing”) was something LLNL did not reward or encourage.

Rob, my counterpart here, has been essentially on his own and he has a lot of great ideas that I can help him implement. We both really love technology and open source, so we hardly have to bring each other up to speed. I no longer have to talk until I’m blue in the face to extol the virtues of FreeBSD, Puppet, Bacula, Wiki’s, Git, code management tools, etc… they are accepted as a matter of fact. I’m also allowed to run whatever OS I want for whatever reason. Rob and I were talking about DragonflyBSD’s HAMMER filesystem, and how we are both just looking for an excuse to use it.

I couldn’t even afford to entertain that though while at LLNL. It was hard enough getting OS’s approved, because it was never about technical merit, and the thought of trying to defend yet another OS was enough to prevent me from using it.

Work has been rewarding, and I can honestly say that I’ve never worked harder. I’ve refined my workflow a bit, and with the help of my friend Steve, I’ve been shown how to use JIRA and FishEye to keep track of my projects. Now when I come in to work, I don’t have to sit here and think about where I was yesterday, I an just pull up my project page and review my tasks.

I also now have somewhat of a social life, which is pretty exciting! Tuesday nights I drive to Monterey and have dinner with Glenn and Erin:

Glenn and Eric cookin' in the kitchen

Delicious pizza!

We have dinner, converse, and then we bust out the instruments. I had to drop my music class at DVC, but having someone to be musical with is very fulfilling. No tour dates or T-Shirts just yet, but we have a great time. I’m looking forward to getting my gear down here so we can get really crazy

I also get to see Steve and Summer a whole lot more, and we typically meet up on Thursday nights. It will be nice when I’m finally moved down in the area so we can do that on the weekend, have a BBQ or something.

Well, that about sums up the last two months I’ve been here. I’ll be moving to Felton in the beginning of the month, so more after that!

Coal was the dog I dreamed of having; he was the combination of all the Aussies I’ve had growing up.

He was the best. That dog never seemed to have a down moment. He was always happy, and always looking for trouble. I also felt like he was an extension to me, but I suspect that is me projecting my own personality on him. Lets be honest, he wasn’t an extension of me, he was better than me. He was me and more. Filled with unconditional love and never had a bad day, a temper tantrum, and a moment of self-involved narcissism. I was lucky to be with him.

I’ve always had an Australian Shepard; Max, Buddy, and then my parents both had Shasta and Happy. I loved their energy, friendliness, how expressive they seemed to be with their eyes. Even the ridiculous smile, which they always greeted me with.

They all lived to a average and predictable age (Shasta is still going strong) and I was prepared for their natural deaths. Over the years, both of my parents have gotten Aussie’s, and I’ve loved them too. I couldn’t be happier when I would visit and show them all the affection I could.

Not Coal. I’m not prepared to accept that he is gone, it still hasn’t fully processed yet. I keep looking at this pictures and even though I get upset, I still think that I’ll see him when I get home. Or when I look in the backyard, I expect to see him up to his usual shenanigans.

I didn’t get enough time with him, and I’m mad that I let him out early that morning instead of just keeping him in the house. I know that he just had it in him to escape and run around. He loved it, he loved exploring and being free, and that was noticeable when we first got him and he tried to escape the backyard.

His attempts to escape got really bad after Zoey died. It seemed that having her around helped ground him, especially when he got scared from loud noises like fireworks or gun shots. We knew we wanted to get out of Antioch for the kids, but when Coal started getting scared and trying to take off if we were not home, I knew he needed to move to a safer place as well. I just thought if he could make it until we moved, he would be happier and better off.

He just kept getting out though, and he got lucky every time. People would find him and bring him home to us, and I’d give them what ever I could to show my appreciation. He’d be happier than ever that he got to run around and then get delivered back to us, and it was concerning to me that he never came home on his own. That night I closed up where I noticed he had escaped from

I’m so sad that I didn’t get him to Santa Cruz, and that I wont get to hug him, kick the soccer ball around with him, go on walks, watch him sleep, kiss his cute snout with that black button nose. I’m going to miss him terribly.

I’ve shared an album on Picasa, please enjoy:

Coal

Goodbye Coal, I’m so sorry I couldn’t keep you safe. This is how I’ll always remember the two of us.

Well, after about 9 and a half years with Lawrence Livermore National Laboratory 11/09 was my last day. Now, I wasn’t the only admin besides Jenny (thats the aquilino1@llnl.gov email you see there), but she was my closest friend and peer while I was there. There was a little poetic license there, but it was accurate.

It is hard to quantify the emotions about leaving LLNL. I do not know if I have the appropriate words to describe the feeling.

How about this:

Remember when you left the comfortable confines of high school? That microcosm was your entire world. You had spend the majority of your adolescent life being a part of that world; your friends, you responsibilities, and learning that environment. When you left high school, it was sort of scary, and it was hard to imagine how things would be after leaving high school. The only certainty was the regiment of “work” that you learned in school would continue on (hopefully to college).

Its a big scary world out there, and when you are “institutionalized” you may not want to leave even if it is time.

This is where I am at. It was time for me to forge a new path in my career, and I was starting to feel a little confined by the Lab’s philosophy of what I.T. is. I was also feeling a bit over-specialized. Not in what I specifically do, which is *NIX Administration, but things that only a national lab has to deal with. That is a bit more complicated, so despite the fact that I could have made a lifetime out of LLNL, I was not getting the fulfillment I require. I look for technical challenges, not inter-personally communication challenges.

Like high school, I will keep some of the friends I had made there. Also like high school, I will soon find myself in a different environment where my vantage point and perspective will be force to change and adapt to the new culture. There is nothing wrong with that as long as I continue to have a healthy respect for the past, it is after-all what brought me here.

Most importantly, I will have my anchor: the “work”. The core of my skill set may be re-balanced for the new job, but it will not fundamentally change and I am happy to accept that.

So far, after two full days have gone by, I’m pretty excited about the growing list of projects and tasks that we are facing. Renting in a shared house with two other room mates isn’t so awesome though, I’ll be a full fledged germaphobe (mysophobia) in no time. I clean my dishes but I sort of freak out of my sponge touches anything else but my dish.

Here is to new adventures!

The City only publishes this as a PDF image, comparing the year selected, and the previous year. It also only goes back to 2005.

To help view this as a possible trend, and not just a snapshot in time, I typed up all of them in Google Docs – City of Antioch Crime Stats

It does some some improvement after a few years of growth, it is still however an order of magnitude worse than other cities in California.

According to Homesurfer.com’s Crime report, even the Sister city to Antioch, Pittsburg had 323 per 100,000 “incidents” in one year (2008). While Antioch had 869 per 100,000.

Compare that to places like Moraga which had 35 incidents per 100,000 or heck, even Oakley’s 248, which borders Antioch on the eastern side.

Here is a decent table output:

I applaud the police department for making a dent over the past two years, and even more so for publishing this data (Concord and Pittsburg did not have this information easily accessible). Even with their effort, I don’t think I’ll be hanging around to see if it pays off.

I hope that Antioch becomes a nice bedding community again. The type of place where people like my grandfather, only had to drawn their weapon once during a domestic dispute, and never had to fire it.

I enjoyed it a lot, because I really like to share information. It is especially rewarding when it is something I have taken a great interest in, like the FreeBSD Operating System.

The design of the course was simple, I did my best to fill in the gap that a Solaris or Linux administrator might have. Everyone on the team has experience with Unix, mostly Solaris and Red Hat, so I skipped a lot of the basics.

This was also done in 1.5 hours increments over a few months. None of us had the time to dedicate four or so days in a real class room environment where I could concoct lab exercises. I wish we could have, that would have been GREAT. Alas, I wrote these up in my spare time and put them up on the calendar for the team.

I even created a certificate for those that attended all of them :)

Cert: fbsd-admin-template

So, here is what I’m going to do. I think it is important to share, so I converted all of my slides and material into Google’s Presentation document and made them Public.

Both to read and write.

So, if you feel that there are typo’s, errors, or something can be added, I want to extend to opportunity to anyone to correct them.

Part1:
FreeBSD Training #1 – Introduction
Part 2:
FreeBSD Training #2 – Software Management
Part 3:
FreeBSD Training #3 – Service Management
Part 4:
FreeBSD Training #4 – Hardware Management

The last class was me running through SysInstall. My logic for that was simple, everyone should first know how to install and manage software and services before they begin installing a fresh OS.

There you go Internets, enjoy, and please don’t add Viagra links in my presentations :)

As mentioned many times, this is a FreeBSD based environment. Some good sysinfo output below:

Operating system release: FreeBSD 8.2-RELEASE
OS architecture: amd64
Kernel build dir location: /usr/obj/usr/src/sys/GENERIC
Currently booted kernel: /boot/kernel/kernel

Currently loaded kernel modules (kldstat(8)):
zfs.ko
opensolaris.ko

Bootloader settings for the Director/Database node:

The /boot/loader.conf has the following contents:
kern.ipc.semmni=1024
kern.ipc.semmns=2048
kern.ipc.semmnu=1024

All of the storage nodes and the director are running a GENERIC kernel with very few system tweaking. One of the storage nodes has a Chelsio 10Gb controller, but that hasn’t had a high enough load to crack the 1Gb/sec barrier.

I’m using Bacula from the ports tree, and the directory has a special Make flag to build with gcc’s debugging symbols. Jenny worked on getting that setup when we were having some stability issues.

The Bacula configuration one the director node is backed by a git repository. It adds a little bit of complexity for a systems administrator, when they want to add a client, but the benefit is clear. This backup project actually enforces change control and tracks all of the commits by who.

I’ve also setup Redmine as a project front-end, and I’ve begun to file tickets and reference what commit fixed what. This not only tracks my progress, but it is the first time I’ve had a backup server that was clearly documented and had some type of accountability.

A snippet of the Redmine site

The Structure

I’ve compared projects like bacula to a large box of LegosTM. It doesn’t enforce a structure by any means, and I’ve taken it upon myself to add meaning to the otherwise flat and incomprehensible bacula-dir.conf

The Bacula Port on FreeBSD installs all configuration files in /usr/local/etc.

Write, the Director, only contains the following in /usr/local/etc/bacula-dir.conf:

@/usr/local/etc/bacula/bacula-dir.conf
@/usr/local/etc/bacula/storage.conf
@/usr/local/etc/bacula/clients.conf
@/usr/local/etc/bacula/messages.conf
@/usr/local/etc/bacula/schedules.conf
@/usr/local/etc/bacula/pools.conf

As you can see, I place everything in etc/bacula/.

Here is a beautiful output of tree(1):

bacula
|-- bacula-dir.conf
|-- bin
|   |-- create_client.sh
|   `-- package_list.sh
|-- clients.conf
|-- clients.d
|   |-- 10am
|   |-- 10pm
|   |-- 11pm
|   |-- 12am
|   |-- 1am
|   |-- 2am
|   |-- 3am
|   |-- 4am
|   |-- 4pm
|   |-- 5am
|   |-- 5pm
|   |-- 6am
|   |-- 6pm
|   |-- 7am
|   |-- 7pm
|   |-- 8am
|   |-- 8pm
|   |-- 9am
|   |-- 9pm
|   |-- TEMPLATE-mac
|   |-- TEMPLATE-unix
|   `-- TEMPLATE-win32
|-- excludes.d
|   |-- common.conf
|   |-- mac.conf
|   |-- unix.conf
|   `-- win32.conf
|-- messages.conf
|-- pools.conf
|-- schedules.conf
|-- storage.conf
`-- storage.d
    |-- write-01.conf
    |-- write-02.conf
    |-- write-03.conf
    |-- write-04.conf
    |-- write-05.conf
    `-- write-06.conf

Storage Nodes

All of the storage nodes are using ZFS as the filesystem/Volume manager.

write-06# zpool list
NAME         SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
filevol001  90.6T  33.3T  57.3T    36%  ONLINE  -

They all have one volume, /filevol001, and I created 512 “drives” within that volume. Effectivly, each storage node has 512 drives, and clients are randomly assigned a drive.

Since I have 6 storage nodes, I wrote a little shell script to handle the directory creation:

#!/bin/bash
i=1
 
while [ $i -le 512 ]
do
        install -d -o bacula -g bacula -m 770 /filevol001/drive$i
        ((i++))
done

Simple, right? I also wrote a script to generate the bacula-sd.conf file on a storage node as well:

#!/bin/bash
 
usage()
{
cat << EOF
    Usage $0 NUMBER > /usr/local/etc/bacula-sd.conf
 
    Where "NUMBER" is just a single digit indicating which storage node this is.
 
    Example, for write-07:
    $ make_sd.sh 7 > /usr/local/etc/bacula-sd.conf
EOF
}
 
i=1
 
if [[ -z $1 ]]
then
    usage
    exit
fi
 
printf "Storage {\n"
printf "\tName = write-0$1.llnl.gov-sd\n"
printf "\tSDAddress = write-0$1.llnl.gov\n"
printf "\tSDPort = 9103\n"
printf "\tWorkingDirectory = \"/var/db/bacula\"\n"
printf "\tPid Directory = \"/var/run\"\n"
printf "\tMaximum Concurrent Jobs = 516\n"
printf "}\n"
 
printf "#\n"
printf "# List Directors who are permitted to contact Storage daemon\n"
printf "#\n"
printf "Director {\n"
printf "\tName = write.llnl.gov-dir\n"
printf "\tPassword = \"ItsASecret\"\n"
printf "}\n"
 
printf "#\n"
printf "# Restricted Director, used by tray-monitor to get the\n"
printf "#   status of the storage daemon\n"
printf "#\n"
printf "Director {\n"
printf "\tName = write.llnl.gov-mon\n"
printf "\tPassword = \"ItsANotherSecret\"\n"
printf "\tMonitor = yes\n"
printf "}\n"
 
printf "Messages {\n"
printf "\tName = Standard\n"
printf "\tdirector = write.llnl.gov-dir = all\n"
printf "}\n"
 
 
printf "Device {\n"
printf "\tName = W0$1FileStorage\n"
printf "\tMedia Type = File\n"
printf "\tArchive Device = /filevol001\n"
printf "\tLabelMedia = yes;\n"
printf "\tRandom Access = Yes;\n"
printf "\tAutomaticMount = yes;\n"
printf "\tRemovableMedia = no;\n"
printf "\tAlwaysOpen = no;\n"
printf "\tMaximum Concurrent Jobs = 2\n"
printf "}\n"
 
while [ $i -le 512 ]
do
        printf "\n"
        printf "Device {\n"
        printf "\tName = W0$1FileStorageD$i\n"
        printf "\tMedia Type = File\n"
        printf "\tArchive Device = /filevol001/drive$i\n"
        printf "\tLabelMedia = yes;\n"
        printf "\tRandom Access = Yes;\n"
        printf "\tAutomaticMount = yes;\n"
        printf "\tRemovableMedia = no;\n"
        printf "\tAlwaysOpen = no;\n"
        printf "\tMaximum Concurrent Jobs = 2\n"
        printf "}\n"
        ((i++))
done

On the Directory, a storage node definition is saved in /usr/local/etc/bacula/storage.d/write-0{N}.conf, which is included in /usr/local/etc/bacula/storage.conf:

@/usr/local/etc/bacula/storage.d/write-01.conf
@/usr/local/etc/bacula/storage.d/write-02.conf
@/usr/local/etc/bacula/storage.d/write-03.conf
@/usr/local/etc/bacula/storage.d/write-04.conf
@/usr/local/etc/bacula/storage.d/write-05.conf
@/usr/local/etc/bacula/storage.d/write-06.conf

Client Generation

There are two components, the TEMPLATE file (there are three, TEMPLATE-unix, TEMPLATE-win32 and TEMPATE-mac) and the shell script.

The Client TEMPLATE File

Here is what one of the TEMPLATE files looks like:

#
# Client Definition, the Password here must match
#  the clients bacula-fd.conf Client definition.
#
# Using Vi/m, you can easily replaced HOSTNAME with
#  the short hostname of the client with:
#  %s/HOSTNAME/yourhostname/
#
#

Client {
    Name = HOSTNAME.llnl.gov
    Address = HOSTNAME.llnl.gov
    FDPort = 9102
    Catalog = Catalog001
    Password = "ItsASecret"
    File Retention = 40 days
    Job Retention = 1 months
    AutoPrune = yes
    Maximum Concurrent Jobs = 10
    Heartbeat Interval = 300
}

Console {
    Name = HOSTNAME.llnl.gov-acl
    Password = ItsASecret
    JobACL = "HOSTNAME.llnl.gov RestoreFiles", "HOSTNAME.llnl.gov"
    ScheduleACL = *all*
    ClientACL = HOSTNAME.llnl.gov
    FileSetACL = "HOSTNAME.llnl.gov FileSet"
    CatalogACL = Catalog001
    CommandACL = *all*
    StorageACL = *all*
    PoolACL = HOSTNAME.llnl.gov-File
}

Job {
    Name = "HOSTNAME.llnl.gov"
    Type = Backup
    Level = Incremental
    FileSet = "HOSTNAME.llnl.gov FileSet"
    Client = "HOSTNAME.llnl.gov"
    Storage = FileStorageD##
    Pool = HOSTNAME.llnl.gov-File
    Schedule = "@@"
    Messages = Standard
    Priority = 10
    Write Bootstrap = "/var/db/bacula/%c.bsr"
    Maximum Concurrent Jobs = 10
    Reschedule On Error = yes
    Reschedule Interval = 1 hour
    Reschedule Times = 1
    Max Wait Time = 30 minutes
    Cancel Lower Level Duplicates = yes
    Allow Duplicate Jobs = no
    RunScript {
        RunsWhen = Before
        FailJobOnError = no
        Command = "/etc/scripts/package_list.sh"
        RunsOnClient = yes
    }
}

Pool {
    Name = HOSTNAME.llnl.gov-File
    Pool Type = Backup
    Recycle = yes
    AutoPrune = yes
    Volume Retention = 1 months
    Maximum Volume Bytes = 10G
    Maximum Volumes = 100
    LabelFormat = "HOSTNAME.llnl.govFileVol"
    Maximum Volume Jobs = 5
}

Job {
    Name = "HOSTNAME.llnl.gov RestoreFiles"
    Type = Restore
    Client= HOSTNAME.llnl.gov
    FileSet="HOSTNAME.llnl.gov FileSet"
    Storage = FileStorageD##
    Pool = HOSTNAME.llnl.gov-File
    Messages = Standard
    #Where = /tmp/bacula-restores
}

FileSet {
    Name = "HOSTNAME.llnl.gov FileSet"
    Include {
        Options {
            signature = MD5
            compression = GZIP6
                        fstype = ext2
                        fstype = xfs
                        fstype = jfs
                        fstype = ufs
                        fstype = zfs
                        onefs = no
                        Exclude = yes
                        @/usr/local/etc/bacula/excludes.d/common.conf
        }
                File = /
                File = /usr/local
                Exclude Dir Containing = .excludeme
    }
    Exclude {
        @/usr/local/etc/bacula/excludes.d/unix.conf
    }
}

The Create Client Script

So here is what really makes creating clients easy for us, the create_client script.

I didn’t want to do it this way, really, so part of me is very ashamed of this tool. I would have preferred to re-write this in Python, or make a web page out of it, and let admins create clients from their desktop. Or, I would have loved to create a puppet module to handle this automagically (but that would exlcude everything that *isn’t* running Puppet, which is huge).

With that disclaimer, here is my create_client shell script:

#!/usr/bin/env bash
# usage: cclient -t unix -s 12am -h hostname
#
 
umask 022
 
# Variables
## Randomize Schedule
SCHEDULES="4pm 5pm 6pm 7pm 8pm 9pm 10pm 11pm 12am 1am 2am 3am 4am 5am 6am 7am 8am 9am 10am"
s=($SCHEDULES)
num_s=${#s[*]}
RAND_SCHED=${s[$((RANDOM%num_s))]}
# Randomize which storage node we use
NODES="write-06 write-01 write-06 write-01 write-02 write-03 write-04 write-05"
n=($NODES)
num_n=${#n[*]}
RAND_NODE=${n[$((RANDOM%num_n))]}
 
export DRIVE=`jot -r 1 1 512`
export BDIR="/usr/local/etc/bacula"
export TYPE="unix"
export SCHEDULE=$RAND_SCHED
export HOSTNAME=""
export STORAGE_NODE=$RAND_NODE
export GIT_DIR="/usr/local/etc/bacula/.git"
export CLASS="desktop"
 
if [ $(whoami) == "root" ]
then
cat << EOF
                Please do not run this as root. This script runs a
                git add/commit, which is how changes are managed and
                tracked. If you run this as root, then it shows up
                as carlson39 or root.
 
                If you encounter a problem with your normal OUN account,
                please contact Mike Carlson, or submit a bug here:
                https://st-scm.llnl.gov/redmine/snt/projects/bacula/issues/new
EOF
 
exit 1
fi
 
usage()
{
cat << EOF
 
        Usage: $0 [OPTION]... -h HOSTNAME
 
        This script will generate a bacula client definition.
 
        OPTIONS:
        -s      schedule, (4pm|5pm|6pm|7pm|8pm|9pm|10pm|11pm|12am|1am|2am|3am|4am|5am|6am|7am|8am|9am). The default schedule is random.
        -t      type, (unix|win32|mac), unix is the default
        -n      storage node (write-01|write-02|...), the default is random.
        -h      hostname (use the short hostname)
EOF
}
 
cd $BDIR
 
while getopts 'c:t:s:n:h:' OPTION
do
        case $OPTION in
                c)
                        CLASS=$OPTARG
                        ;;
                t)
                        TYPE=$OPTARG
                        ;;
                s)
                        SCHEDULE=$OPTARG
                        ;;
                h)
                        HOSTNAME=$OPTARG
                        echo $HOSTNAME | egrep -q "(llnl.gov|ucllnl.org)"
                        if [ $? -eq 0 ]
                        then
                        HOSTNAME=`echo $HOSTNAME|sed -e 's/.llnl.gov//' -e 's/.ucllnl.org//'`
                        fi
 
                        ;;
                n)
                        STORAGE_NODE=$OPTARG
                        ;;
                ?)
                        usage
                        exit
                        ;;
        esac
done
 
if [[ -z $CLASS ]] || [[ -z $TYPE ]] || [[ -z $SCHEDULE ]] || [[ -z $HOSTNAME ]] || [[ -z $STORAGE_NODE ]]
then
        usage
        exit 1
fi
 
grep -w $HOSTNAME $BDIR/clients.conf
if [ $? -eq 0 ]
then
        echo 'client '$HOSTNAME 'already exists...'
else
        export RETRY_COUNT="2"
 
        if [ $STORAGE_NODE == "write-01" ]
        then
                DRIVE=`jot -r 1 33 512`
                sed -e 's/HOSTNAME/'$HOSTNAME'/g' -e 's/FileStorageD##/FileStorageD'$DRIVE'/' -e 's/\@\@/'$SCHEDULE'/' -e 's/RETRY_COUNT/'$RETRY_COUNT'/g' $BDIR/clients.d/TEMPLATE-$TYPE > $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
                echo \@$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf >> $BDIR/clients.conf
        else
                export SN=`echo $STORAGE_NODE | cut -c 7-8`
                sed -e 's/HOSTNAME/'$HOSTNAME'/g' -e 's/FileStorageD##/W'$SN'FileStorageD'$DRIVE'/' -e 's/\@\@/'$SCHEDULE'/' -e 's/RETRY_COUNT/'$RETRY_COUNT'/g' $BDIR/clients.d/TEMPLATE-$TYPE > $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
                echo \@$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf >> $BDIR/clients.conf
        fi
 
        chgrp st-bacula-admins $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
        git add $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf $BDIR/clients.conf
        git commit
        echo 'created client definition: '$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
        echo 'for '$HOSTNAME'.llnl.gov'
fi

This is always a work in progress, but at the core, it is a simple sed wrapper with a lot of randomization and a git commit.

Why all the randomization?

Because I had to add around 1000 clients in a VERY short amount of time. We didn’t have a problem pushing the Bacula client to all of the platforms, nor the bacula-fd.conf file either. What I could not do was spend the time to create and manage all of the resources for each client. That is why I have so many devices/drives, so I can attempt to have a 1:1 without having to actually think about it.

So, I wrote ANOTHER script to wrap around this one when I need to do bulk client creations. I’m not going to post that, it just loops through the above command.

Pre-Job command – Package List

I only do this on the Unix/Linux clients, and I thought it was a cool idea.

Yeah, I will pat myself on the back a little bit for that :)

I exclude the Operating System from backups for two reasons, 1) to reduce backing up duplicate and reproducible data and 2) Our build/Imaging process is so quick and clean it is just faster to rebuild than restore everything.

Still, I needed a way to keep the state of installed packages/software.

This is where the pre-job command comes in handy. This part right here:

    RunScript {
        RunsWhen = Before
        FailJobOnError = no
        Command = "/etc/scripts/package_list.sh"
        RunsOnClient = yes
    }

That package_list.sh file looks like this:

 
#!/usr/bin/env bash
 
export PLIST="/root/plist.txt"
 
case "`uname -s`" in
Linux)
                if [ -x /usr/bin/lsb_release ]; then
                        DIST=`lsb_release -d`
                fi
 
                # RHEL
                if [ -x /usr/bin/up2date ]; then
                        rpm -qa > $PLIST
                fi
 
                # RHEL 5
                if [ -x /usr/bin/yum ]; then
                        if [ -f /var/run/yum.pid ]; then
                                echo "Yum currently in use, exiting gracefully..."
                                exit 0
                        else
                        /usr/bin/yum list installed | awk '{print $1}' > $PLIST
                        fi
                fi
 
                # Ubuntu
                if [ -x /usr/bin/dpkg ]; then
                        /usr/bin/dpkg --get-selections | awk '{print $1}' > $PLIST
                fi
                ;;
 
FreeBSD)
                pkg_info|awk '{print $1}' > $PLIST
                ;;
SunOS)
                pkginfo |awk '{print $1}' > $PLIST
                ;;
esac

That file, /root/plist.txt, gets backed up.

Now we have a record of what was installed on our Unix platforms :)

That is it for now, see you at Part 3