Well, after about 9 and a half years with Lawrence Livermore National Laboratory 11/09 was my last day. Now, I wasn’t the only admin besides Jenny (thats the aquilino1@llnl.gov email you see there), but she was my closest friend and peer while I was there. There was a little poetic license there, but it was accurate.

It is hard to quantify the emotions about leaving LLNL. I do not know if I have the appropriate words to describe the feeling.

How about this:

Remember when you left the comfortable confines of high school? That microcosm was your entire world. You had spend the majority of your adolescent life being a part of that world; your friends, you responsibilities, and learning that environment. When you left high school, it was sort of scary, and it was hard to imagine how things would be after leaving high school. The only certainty was the regiment of “work” that you learned in school would continue on (hopefully to college).

Its a big scary world out there, and when you are “institutionalized” you may not want to leave even if it is time.

This is where I am at. It was time for me to forge a new path in my career, and I was starting to feel a little confined by the Lab’s philosophy of what I.T. is. I was also feeling a bit over-specialized. Not in what I specifically do, which is *NIX Administration, but things that only a national lab has to deal with. That is a bit more complicated, so despite the fact that I could have made a lifetime out of LLNL, I was not getting the fulfillment I require. I look for technical challenges, not inter-personally communication challenges.

Like high school, I will keep some of the friends I had made there. Also like high school, I will soon find myself in a different environment where my vantage point and perspective will be force to change and adapt to the new culture. There is nothing wrong with that as long as I continue to have a healthy respect for the past, it is after-all what brought me here.

Most importantly, I will have my anchor: the “work”. The core of my skill set may be re-balanced for the new job, but it will not fundamentally change and I am happy to accept that.

So far, after two full days have gone by, I’m pretty excited about the growing list of projects and tasks that we are facing. Renting in a shared house with two other room mates isn’t so awesome though, I’ll be a full fledged germaphobe (mysophobia) in no time. I clean my dishes but I sort of freak out of my sponge touches anything else but my dish.

Here is to new adventures!

As mentioned many times, this is a FreeBSD based environment. Some good sysinfo output below:

Operating system release: FreeBSD 8.2-RELEASE
OS architecture: amd64
Kernel build dir location: /usr/obj/usr/src/sys/GENERIC
Currently booted kernel: /boot/kernel/kernel

Currently loaded kernel modules (kldstat(8)):
zfs.ko
opensolaris.ko

Bootloader settings for the Director/Database node:

The /boot/loader.conf has the following contents:
kern.ipc.semmni=1024
kern.ipc.semmns=2048
kern.ipc.semmnu=1024

All of the storage nodes and the director are running a GENERIC kernel with very few system tweaking. One of the storage nodes has a Chelsio 10Gb controller, but that hasn’t had a high enough load to crack the 1Gb/sec barrier.

I’m using Bacula from the ports tree, and the directory has a special Make flag to build with gcc’s debugging symbols. Jenny worked on getting that setup when we were having some stability issues.

The Bacula configuration one the director node is backed by a git repository. It adds a little bit of complexity for a systems administrator, when they want to add a client, but the benefit is clear. This backup project actually enforces change control and tracks all of the commits by who.

I’ve also setup Redmine as a project front-end, and I’ve begun to file tickets and reference what commit fixed what. This not only tracks my progress, but it is the first time I’ve had a backup server that was clearly documented and had some type of accountability.

A snippet of the Redmine site

The Structure

I’ve compared projects like bacula to a large box of LegosTM. It doesn’t enforce a structure by any means, and I’ve taken it upon myself to add meaning to the otherwise flat and incomprehensible bacula-dir.conf

The Bacula Port on FreeBSD installs all configuration files in /usr/local/etc.

Write, the Director, only contains the following in /usr/local/etc/bacula-dir.conf:

@/usr/local/etc/bacula/bacula-dir.conf
@/usr/local/etc/bacula/storage.conf
@/usr/local/etc/bacula/clients.conf
@/usr/local/etc/bacula/messages.conf
@/usr/local/etc/bacula/schedules.conf
@/usr/local/etc/bacula/pools.conf

As you can see, I place everything in etc/bacula/.

Here is a beautiful output of tree(1):

bacula
|-- bacula-dir.conf
|-- bin
|   |-- create_client.sh
|   `-- package_list.sh
|-- clients.conf
|-- clients.d
|   |-- 10am
|   |-- 10pm
|   |-- 11pm
|   |-- 12am
|   |-- 1am
|   |-- 2am
|   |-- 3am
|   |-- 4am
|   |-- 4pm
|   |-- 5am
|   |-- 5pm
|   |-- 6am
|   |-- 6pm
|   |-- 7am
|   |-- 7pm
|   |-- 8am
|   |-- 8pm
|   |-- 9am
|   |-- 9pm
|   |-- TEMPLATE-mac
|   |-- TEMPLATE-unix
|   `-- TEMPLATE-win32
|-- excludes.d
|   |-- common.conf
|   |-- mac.conf
|   |-- unix.conf
|   `-- win32.conf
|-- messages.conf
|-- pools.conf
|-- schedules.conf
|-- storage.conf
`-- storage.d
    |-- write-01.conf
    |-- write-02.conf
    |-- write-03.conf
    |-- write-04.conf
    |-- write-05.conf
    `-- write-06.conf

Storage Nodes

All of the storage nodes are using ZFS as the filesystem/Volume manager.

write-06# zpool list
NAME         SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
filevol001  90.6T  33.3T  57.3T    36%  ONLINE  -

They all have one volume, /filevol001, and I created 512 “drives” within that volume. Effectivly, each storage node has 512 drives, and clients are randomly assigned a drive.

Since I have 6 storage nodes, I wrote a little shell script to handle the directory creation:

#!/bin/bash
i=1
 
while [ $i -le 512 ]
do
        install -d -o bacula -g bacula -m 770 /filevol001/drive$i
        ((i++))
done

Simple, right? I also wrote a script to generate the bacula-sd.conf file on a storage node as well:

#!/bin/bash
 
usage()
{
cat << EOF
    Usage $0 NUMBER > /usr/local/etc/bacula-sd.conf
 
    Where "NUMBER" is just a single digit indicating which storage node this is.
 
    Example, for write-07:
    $ make_sd.sh 7 > /usr/local/etc/bacula-sd.conf
EOF
}
 
i=1
 
if [[ -z $1 ]]
then
    usage
    exit
fi
 
printf "Storage {\n"
printf "\tName = write-0$1.llnl.gov-sd\n"
printf "\tSDAddress = write-0$1.llnl.gov\n"
printf "\tSDPort = 9103\n"
printf "\tWorkingDirectory = \"/var/db/bacula\"\n"
printf "\tPid Directory = \"/var/run\"\n"
printf "\tMaximum Concurrent Jobs = 516\n"
printf "}\n"
 
printf "#\n"
printf "# List Directors who are permitted to contact Storage daemon\n"
printf "#\n"
printf "Director {\n"
printf "\tName = write.llnl.gov-dir\n"
printf "\tPassword = \"ItsASecret\"\n"
printf "}\n"
 
printf "#\n"
printf "# Restricted Director, used by tray-monitor to get the\n"
printf "#   status of the storage daemon\n"
printf "#\n"
printf "Director {\n"
printf "\tName = write.llnl.gov-mon\n"
printf "\tPassword = \"ItsANotherSecret\"\n"
printf "\tMonitor = yes\n"
printf "}\n"
 
printf "Messages {\n"
printf "\tName = Standard\n"
printf "\tdirector = write.llnl.gov-dir = all\n"
printf "}\n"
 
 
printf "Device {\n"
printf "\tName = W0$1FileStorage\n"
printf "\tMedia Type = File\n"
printf "\tArchive Device = /filevol001\n"
printf "\tLabelMedia = yes;\n"
printf "\tRandom Access = Yes;\n"
printf "\tAutomaticMount = yes;\n"
printf "\tRemovableMedia = no;\n"
printf "\tAlwaysOpen = no;\n"
printf "\tMaximum Concurrent Jobs = 2\n"
printf "}\n"
 
while [ $i -le 512 ]
do
        printf "\n"
        printf "Device {\n"
        printf "\tName = W0$1FileStorageD$i\n"
        printf "\tMedia Type = File\n"
        printf "\tArchive Device = /filevol001/drive$i\n"
        printf "\tLabelMedia = yes;\n"
        printf "\tRandom Access = Yes;\n"
        printf "\tAutomaticMount = yes;\n"
        printf "\tRemovableMedia = no;\n"
        printf "\tAlwaysOpen = no;\n"
        printf "\tMaximum Concurrent Jobs = 2\n"
        printf "}\n"
        ((i++))
done

On the Directory, a storage node definition is saved in /usr/local/etc/bacula/storage.d/write-0{N}.conf, which is included in /usr/local/etc/bacula/storage.conf:

@/usr/local/etc/bacula/storage.d/write-01.conf
@/usr/local/etc/bacula/storage.d/write-02.conf
@/usr/local/etc/bacula/storage.d/write-03.conf
@/usr/local/etc/bacula/storage.d/write-04.conf
@/usr/local/etc/bacula/storage.d/write-05.conf
@/usr/local/etc/bacula/storage.d/write-06.conf

Client Generation

There are two components, the TEMPLATE file (there are three, TEMPLATE-unix, TEMPLATE-win32 and TEMPATE-mac) and the shell script.

The Client TEMPLATE File

Here is what one of the TEMPLATE files looks like:

#
# Client Definition, the Password here must match
#  the clients bacula-fd.conf Client definition.
#
# Using Vi/m, you can easily replaced HOSTNAME with
#  the short hostname of the client with:
#  %s/HOSTNAME/yourhostname/
#
#

Client {
    Name = HOSTNAME.llnl.gov
    Address = HOSTNAME.llnl.gov
    FDPort = 9102
    Catalog = Catalog001
    Password = "ItsASecret"
    File Retention = 40 days
    Job Retention = 1 months
    AutoPrune = yes
    Maximum Concurrent Jobs = 10
    Heartbeat Interval = 300
}

Console {
    Name = HOSTNAME.llnl.gov-acl
    Password = ItsASecret
    JobACL = "HOSTNAME.llnl.gov RestoreFiles", "HOSTNAME.llnl.gov"
    ScheduleACL = *all*
    ClientACL = HOSTNAME.llnl.gov
    FileSetACL = "HOSTNAME.llnl.gov FileSet"
    CatalogACL = Catalog001
    CommandACL = *all*
    StorageACL = *all*
    PoolACL = HOSTNAME.llnl.gov-File
}

Job {
    Name = "HOSTNAME.llnl.gov"
    Type = Backup
    Level = Incremental
    FileSet = "HOSTNAME.llnl.gov FileSet"
    Client = "HOSTNAME.llnl.gov"
    Storage = FileStorageD##
    Pool = HOSTNAME.llnl.gov-File
    Schedule = "@@"
    Messages = Standard
    Priority = 10
    Write Bootstrap = "/var/db/bacula/%c.bsr"
    Maximum Concurrent Jobs = 10
    Reschedule On Error = yes
    Reschedule Interval = 1 hour
    Reschedule Times = 1
    Max Wait Time = 30 minutes
    Cancel Lower Level Duplicates = yes
    Allow Duplicate Jobs = no
    RunScript {
        RunsWhen = Before
        FailJobOnError = no
        Command = "/etc/scripts/package_list.sh"
        RunsOnClient = yes
    }
}

Pool {
    Name = HOSTNAME.llnl.gov-File
    Pool Type = Backup
    Recycle = yes
    AutoPrune = yes
    Volume Retention = 1 months
    Maximum Volume Bytes = 10G
    Maximum Volumes = 100
    LabelFormat = "HOSTNAME.llnl.govFileVol"
    Maximum Volume Jobs = 5
}

Job {
    Name = "HOSTNAME.llnl.gov RestoreFiles"
    Type = Restore
    Client= HOSTNAME.llnl.gov
    FileSet="HOSTNAME.llnl.gov FileSet"
    Storage = FileStorageD##
    Pool = HOSTNAME.llnl.gov-File
    Messages = Standard
    #Where = /tmp/bacula-restores
}

FileSet {
    Name = "HOSTNAME.llnl.gov FileSet"
    Include {
        Options {
            signature = MD5
            compression = GZIP6
                        fstype = ext2
                        fstype = xfs
                        fstype = jfs
                        fstype = ufs
                        fstype = zfs
                        onefs = no
                        Exclude = yes
                        @/usr/local/etc/bacula/excludes.d/common.conf
        }
                File = /
                File = /usr/local
                Exclude Dir Containing = .excludeme
    }
    Exclude {
        @/usr/local/etc/bacula/excludes.d/unix.conf
    }
}

The Create Client Script

So here is what really makes creating clients easy for us, the create_client script.

I didn’t want to do it this way, really, so part of me is very ashamed of this tool. I would have preferred to re-write this in Python, or make a web page out of it, and let admins create clients from their desktop. Or, I would have loved to create a puppet module to handle this automagically (but that would exlcude everything that *isn’t* running Puppet, which is huge).

With that disclaimer, here is my create_client shell script:

#!/usr/bin/env bash
# usage: cclient -t unix -s 12am -h hostname
#
 
umask 022
 
# Variables
## Randomize Schedule
SCHEDULES="4pm 5pm 6pm 7pm 8pm 9pm 10pm 11pm 12am 1am 2am 3am 4am 5am 6am 7am 8am 9am 10am"
s=($SCHEDULES)
num_s=${#s[*]}
RAND_SCHED=${s[$((RANDOM%num_s))]}
# Randomize which storage node we use
NODES="write-06 write-01 write-06 write-01 write-02 write-03 write-04 write-05"
n=($NODES)
num_n=${#n[*]}
RAND_NODE=${n[$((RANDOM%num_n))]}
 
export DRIVE=`jot -r 1 1 512`
export BDIR="/usr/local/etc/bacula"
export TYPE="unix"
export SCHEDULE=$RAND_SCHED
export HOSTNAME=""
export STORAGE_NODE=$RAND_NODE
export GIT_DIR="/usr/local/etc/bacula/.git"
export CLASS="desktop"
 
if [ $(whoami) == "root" ]
then
cat << EOF
                Please do not run this as root. This script runs a
                git add/commit, which is how changes are managed and
                tracked. If you run this as root, then it shows up
                as carlson39 or root.
 
                If you encounter a problem with your normal OUN account,
                please contact Mike Carlson, or submit a bug here:
                https://st-scm.llnl.gov/redmine/snt/projects/bacula/issues/new
EOF
 
exit 1
fi
 
usage()
{
cat << EOF
 
        Usage: $0 [OPTION]... -h HOSTNAME
 
        This script will generate a bacula client definition.
 
        OPTIONS:
        -s      schedule, (4pm|5pm|6pm|7pm|8pm|9pm|10pm|11pm|12am|1am|2am|3am|4am|5am|6am|7am|8am|9am). The default schedule is random.
        -t      type, (unix|win32|mac), unix is the default
        -n      storage node (write-01|write-02|...), the default is random.
        -h      hostname (use the short hostname)
EOF
}
 
cd $BDIR
 
while getopts 'c:t:s:n:h:' OPTION
do
        case $OPTION in
                c)
                        CLASS=$OPTARG
                        ;;
                t)
                        TYPE=$OPTARG
                        ;;
                s)
                        SCHEDULE=$OPTARG
                        ;;
                h)
                        HOSTNAME=$OPTARG
                        echo $HOSTNAME | egrep -q "(llnl.gov|ucllnl.org)"
                        if [ $? -eq 0 ]
                        then
                        HOSTNAME=`echo $HOSTNAME|sed -e 's/.llnl.gov//' -e 's/.ucllnl.org//'`
                        fi
 
                        ;;
                n)
                        STORAGE_NODE=$OPTARG
                        ;;
                ?)
                        usage
                        exit
                        ;;
        esac
done
 
if [[ -z $CLASS ]] || [[ -z $TYPE ]] || [[ -z $SCHEDULE ]] || [[ -z $HOSTNAME ]] || [[ -z $STORAGE_NODE ]]
then
        usage
        exit 1
fi
 
grep -w $HOSTNAME $BDIR/clients.conf
if [ $? -eq 0 ]
then
        echo 'client '$HOSTNAME 'already exists...'
else
        export RETRY_COUNT="2"
 
        if [ $STORAGE_NODE == "write-01" ]
        then
                DRIVE=`jot -r 1 33 512`
                sed -e 's/HOSTNAME/'$HOSTNAME'/g' -e 's/FileStorageD##/FileStorageD'$DRIVE'/' -e 's/\@\@/'$SCHEDULE'/' -e 's/RETRY_COUNT/'$RETRY_COUNT'/g' $BDIR/clients.d/TEMPLATE-$TYPE > $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
                echo \@$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf >> $BDIR/clients.conf
        else
                export SN=`echo $STORAGE_NODE | cut -c 7-8`
                sed -e 's/HOSTNAME/'$HOSTNAME'/g' -e 's/FileStorageD##/W'$SN'FileStorageD'$DRIVE'/' -e 's/\@\@/'$SCHEDULE'/' -e 's/RETRY_COUNT/'$RETRY_COUNT'/g' $BDIR/clients.d/TEMPLATE-$TYPE > $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
                echo \@$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf >> $BDIR/clients.conf
        fi
 
        chgrp st-bacula-admins $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
        git add $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf $BDIR/clients.conf
        git commit
        echo 'created client definition: '$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
        echo 'for '$HOSTNAME'.llnl.gov'
fi

This is always a work in progress, but at the core, it is a simple sed wrapper with a lot of randomization and a git commit.

Why all the randomization?

Because I had to add around 1000 clients in a VERY short amount of time. We didn’t have a problem pushing the Bacula client to all of the platforms, nor the bacula-fd.conf file either. What I could not do was spend the time to create and manage all of the resources for each client. That is why I have so many devices/drives, so I can attempt to have a 1:1 without having to actually think about it.

So, I wrote ANOTHER script to wrap around this one when I need to do bulk client creations. I’m not going to post that, it just loops through the above command.

Pre-Job command – Package List

I only do this on the Unix/Linux clients, and I thought it was a cool idea.

Yeah, I will pat myself on the back a little bit for that :)

I exclude the Operating System from backups for two reasons, 1) to reduce backing up duplicate and reproducible data and 2) Our build/Imaging process is so quick and clean it is just faster to rebuild than restore everything.

Still, I needed a way to keep the state of installed packages/software.

This is where the pre-job command comes in handy. This part right here:

    RunScript {
        RunsWhen = Before
        FailJobOnError = no
        Command = "/etc/scripts/package_list.sh"
        RunsOnClient = yes
    }

That package_list.sh file looks like this:

 
#!/usr/bin/env bash
 
export PLIST="/root/plist.txt"
 
case "`uname -s`" in
Linux)
                if [ -x /usr/bin/lsb_release ]; then
                        DIST=`lsb_release -d`
                fi
 
                # RHEL
                if [ -x /usr/bin/up2date ]; then
                        rpm -qa > $PLIST
                fi
 
                # RHEL 5
                if [ -x /usr/bin/yum ]; then
                        if [ -f /var/run/yum.pid ]; then
                                echo "Yum currently in use, exiting gracefully..."
                                exit 0
                        else
                        /usr/bin/yum list installed | awk '{print $1}' > $PLIST
                        fi
                fi
 
                # Ubuntu
                if [ -x /usr/bin/dpkg ]; then
                        /usr/bin/dpkg --get-selections | awk '{print $1}' > $PLIST
                fi
                ;;
 
FreeBSD)
                pkg_info|awk '{print $1}' > $PLIST
                ;;
SunOS)
                pkginfo |awk '{print $1}' > $PLIST
                ;;
esac

That file, /root/plist.txt, gets backed up.

Now we have a record of what was installed on our Unix platforms :)

That is it for now, see you at Part 3

I’ll hold you in suspense no longer, I made it to the top, and back down, without any physical issues like cramping. The hike was in fact, very pleasent, and it rained on us but it was nice.

Last year’s experience was a really good learning experience, and it made me a lot more aware of the warning signs of fatigue. I know now that I have to properly prepare for extraneousness activities, and of course, be conscious of the fact that my ill-preparedness will effect those around me. I think Jenny learned a lot too, like how to make sure Mike doesn’t break down… So, along with plenty of extra food, she also brought her Husbands rain-gear and hiking boots (thanks, to both Jenny and Marshall!). I had a moment of reservation about borrowing it, because I hate the feeling of depending on someone else… but, then I came to my senses and realized how awesome being comfortable and dry was going to be!

So, while this hiking trip will lack some of the great details the last one did, it was a much more rewarding trip. The group we went with was cool, especially Mark. He retired from the Lab in 2008, and he was such a nice and complementary person. He is also a “sweat hog” like me, and even though he quickly conceeded to my stunning shirt-drenching prowess, it was great to meet another person that perspired (and even suffered from muscle cramping!). It was also great to hear him and Jenny chat it up, those two were pretty cute on the trail swapping stories :)

Here is the first half of the trip, my phones battery was at 60%, I didn;t charge it the night before, but it was nice to track the “big burn”:
View Track 6 in a larger map

And here are some photo’s that Jenny took, both from this year and last year:

This was me last year, already dying...

Me this year, imitating Jenny's UCD chant that she does..

Probably the funnest picture the three of us took last year. It was Velimir's idea to pretend we were looking at something in the distance.

So here, Jenny and I tried to re-create the photo at the same spot... if you look, it is nearly identical.

Last years group photo

This years group photo reenactment. I somehow managed to get rabbit ears again too...

Jenny petting the dead Coyote... there there dead coyote...

This is the log we had lunch at last year. This was when I first saw my calf muscles cramping up. I re-lived the moment, making sure they were not cramping up.

Small group photo, Mark, Me, Teresa and Jenny

The top of Rose Peak! Adding to the rock pile.

Jenny adding her, what is it now, 4th rock to the peak :) what a pro

Jenny and her new friend

Cow

I’ll link Jenny’s gallery back here when I grab the URL

The setup is this:
The OS was installed on a mirrored hardware raid device (using the mpt(4) driver), and then I had a large RAID6 array attached via a FC controller (using the isp(4) driver). When I installed the OS, the mpt device was showing up as da0. So I went ahead with the install and rebooted the system, so far so good.

What I didn’t realize was the FC device was not seen yet, so after some fiddling, Jenny and I got the large RAID6 array to show up… unfortunately, the isp card was before the mpt card on the PCI bus:

isp0@pci0:2:1:0: class=0x0c0400 card=0x01321077 chip=0x63221077 rev=0x03 hdr=0x00
vendor = 'QLogic Corporation'
device = 'QLA6322 Fibre Channel Adapter'
class = serial bus
subclass = Fibre Channel
mpt0@pci0:2:3:0: class=0x010000 card=0x30601000 chip=0x00501000 rev=0x02 hdr=0x00
vendor = 'LSI Logic (Was: Symbios Logic, NCR)'
device = 'SAS 3000 series, 4-port with 1064 -StorPort'
class = mass storage
subclass = SCSI

and the RAID6 now became da0, and the OS device now became da1.

Doh!

The system prompted for the / drive, so I had to call out the correct device at the mount> prompt:

mount> ufs:/dev/da1s1a

After that, the system continue to boot into mult-user mode, which cause some very strange console behavior (it acted like the return key was being held down), and my only option was to SSH in as local user, su to root, and then fix /etc/fstab.

This was not devastating, however, it show the importance of using disk labels instead of device handles in certain use cases. I haven’t fixed the / mount, but to get a comfort level with using GEOM labels I added another drive to the system and called it EXPORT.

You can assign a permanent label in two ways (that I know of). When you newfs the device, you can specify the L flag (BTW, -O2 means to use UFS2, and -U will use Soft-Updates):
[root@paper ~]> newfs -O2 -U -L EXPORT /dev/da2s1a
OR using glabel (which is what you would have to do for a non UFS filesystem.
[root@paper ~]> glabel create EXPORT da2s1a
Now we can see our newly labeled device in action:
[root@paper ~]> ls /dev/label
. .. EXPORT
[root@paper ~]> glabel status
Name Status Components
label/EXPORT N/A da2s1a

To add it to /etc/fstab, you can either edit the file, or append the correctly tab-delimited line like so:

[root@paper ~]> echo "/dev/label/EXPORT\t/export\tufs\trw\t2\t2" >> /etc/fstab
[root@paper ~]> mkdir /export
[root@paper ~]> mount export

Hurray!

[root@paper ~]> df
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/da1s1a 60931274 4754540 51302234 8% /
devfs 1 1 0 100% /dev
tank 10569645824 107237376 10462408448 1% /tank
/dev/label/EXPORT 132022788 4 121460962 0% /export

[root@paper ~]> mount
/dev/da1s1a on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
tank on /tank (zfs, NFS exported, local)
/dev/label/EXPORT on /export (ufs, local)


This is now a persistent label. To be safe, I’ll have to boot off of a CD/USB drive and modify the root device.

What also made this conference unique, is how honest the Puppet team and community were about the projects strength and weaknesses. Those that have deployed Puppet on a larger scale (MessageOne and Google) seemed to go through the same iterations in attempting to scale out their Puppetmaster’s. From WEBrick (which is what I’m currently running Puppet with :) ), which is hated by all since its a single process/thread web server that can only handle one request at a time. To Mongrel, which you have to manage a mongrel cluster script, feed it lots of memory, and then throw an apache proxy server in front of them. Now, people are starting to settle on using Passenger/mod_rack, which is what I spent most of yesterday looking into and setting up. This allows apache to mount a rails instance, and then you don’t actually have to run puppetmasterd. This still requires some decent hardware, and I’m currently running my puppetmaster on a VM with 2GB or memory, so I’ll have to watch out for that. Chris, the one who introduced me to Puppet, said he still uses WEBrick for all of his DB, Tomcat, and Apache servers (I think he said something like 200 systems) and it has been working out nicely. He, like the guys at Google, also doesn’t run puppet as a daemon.

Anyway, the point is, we learned a lot about the project, way more than if a sales person had come to us and just told us the things puppet does well, or how it operates on paper (*cough* LANDesk *cough*). It was really awesome to talk with Andrew Pollock and Nigel Kersten from Google. See, I was a little unsure about Puppet in our environment, where we have multi-purpose servers, computer servers, and desktops that we have to manage. It seemed, at a first glance, that most of the Puppet users out there have a homogeneous environment, and Andrew (Shafer) had stressed the concept of single role servers. After talking with them, I felt a lot more comfortable pursuing Puppet across our servers and desktops. Did I mention they were super cool and friendly?

We also learned a lot about the Puppet developers, which had its own interesting advantage. I have a lot of respect for what Luke Kanies has been able to do, and by the end of the conference, he showed significant mastery in what he has done, as well as some humility and admitting what he has not been able to do and why. I was a little put off the first day though, when both him and Andrew came off a little arrogant and crass. It did make me step back and think, “Is this project going to be well managed in the future with personalities like this in charge? Is their answer of ‘don’t do that!’ tongue in cheek, or are they not supportive of a diverse environment?”. In the end, I have more respect for the project than ever, and with it still being a young project, I hope they listened to some of the feedback, and I also can’t wait to see where it ends up in the next year.

Andrew, the Puppet Andrew, came up to us a lot during the conference, and he was fun to talk too, and he’s very academic and he had a lot of abstract concepts to talk about. Also, he said this was the first conference he has arranged, and I think he did a fantastic job. Jenny had commented that this was the first conference she had lasted the entire duration, so that says a lot about the pacing and content of PuppetCamp. I felt the same way, every session was incredibly engaging, and how Andrew had setup the democratic and chaotic Open Sessions was very impressive. Lets put it this way, I even got up there and pitched a topic, which is something I would have never done. Hurray for me stepping outside of my comfort zone!

Warning: side topic!

Now that I’ve had the weekend to google all the cool technologies I was exposed too, I’m also reminded why I really like having a FreeBSD server at my disposal. They had talked about CouchDB, so on a whim I did a

~> cd /usr/ports
/usr/ports> make search name=couchdb
Port: couchdb-0.9.0_1,1
Path: /usr/ports/databases/couchdb
Info: A document database server, accessible via a RESTful JSON API
Maint: till@php.net
B-deps: ca_root_nss-3.11.9_2 curl-7.19.6_1 erlang-lite-r13b01_6,1 gettext-0.17_1 gmake-3.81_3 icu-3.8.1_2 libiconv-1.13.1 libtool-2.2.6a nspr-4.8 perl-5.8.9_3 spidermonkey-1.7.0
R-deps: ca_root_nss-3.11.9_2 curl-7.19.6_1 erlang-lite-r13b01_6,1 gettext-0.17_1 gmake-3.81_3 icu-3.8.1_2 libiconv-1.13.1 libtool-2.2.6a nspr-4.8 perl-5.8.9_3 spidermonkey-1.7.0
WWW: http://couchdb.org/

Port: py26-simplecouchdb-0.9.26
Path: /usr/ports/databases/py-simplecouchdb
Info: Simple Librairy to Allow Python Applicationto Use CouchDB
Maint: wenheping@gmail.com
B-deps: py26-httplib2-0.5.0 py26-py-restclient-1.3.2 py26-setuptools-0.6c9 python26-2.6.2_3
R-deps: py26-httplib2-0.5.0 py26-py-restclient-1.3.2 py26-setuptools-0.6c9 python26-2.6.2_3
WWW: http://code.google.com/p/py-simplecouchdb/

I did a ‘make install’, and I had a cool little couchdb up and running. What is also cool is FreeBSD likes to give you very helpful information when you install something. For example, this is what is printed out when you install the CouchDB port:

===> COMPATIBILITY NOTE:
CouchDB is still pre-stable; between 0.8 and 0.9 the database format
changed which breaks BC. In current trunk, the format changed again, so
please double-check in case you are updating an existing installation.

More info:
* http://wiki.apache.org/couchdb/Breaking_changes?action=show&redirect=BreakingChanges
* http://wiki.apache.org/couchdb/BreakingChangesUpdateTrunkTo0Dot9

See, isn’t that helpful? Best of all, I didn’t have to enable additional repositories, or fetch the src manually, and its dependencies and then figure out how to run the right configure script flags… FreeBSD makes it easy, and since it automatically uses what you already have with what is required, its an incredibly stable build. Removing it is pretty simple as well, just:

> pkg_deinstall -R couchdb
---> Deinstalling 'couchdb-0.9.0_1,1'
---> Deinstalling 'erlang-lite-r13b02,1'
[Updating the pkgdb
in /var/db/pkg ... - 118 packages found (-1 +0) (...) done]
---> Deinstalling 'curl-7.19.6_1'
[Updating the pkgdb
in /var/db/pkg ... - 117 packages found (-1 +0) (...) done]
---> Deinstalling 'ca_root_nss-3.11.9_2'
---> Deinstalling 'spidermonkey-1.7.0'
---> Deinstalling 'nspr-4.8'
[Updating the pkgdb
in /var/db/pkg ... - 116 packages found (-1 +0) (...) done]
---> Deinstalling 'gmake-3.81_3'
[Updating the pkgdb
in /var/db/pkg ... - 115 packages found (-1 +0) (...) done]
---> Deinstalling 'perl-threaded-5.8.9_3'
[Updating the pkgdb
in /var/db/pkg ... - 114 packages found (-1 +0) (...) done]
---> Deinstalling 'gettext-0.17_1'
---> Deinstalling 'libiconv-1.13.1'
---> Deinstalling 'icu-3.8.1_2'
---> Deinstalling 'libtool-2.2.6a'
** Listing the failed packages (-:ignored / *:skipped / !:failed)
! curl-7.19.6_1 (pkg_delete failed)
! ca_root_nss-3.11.9_2 (pkg_delete failed)
! perl-threaded-5.8.9_3 (pkg_delete failed)
! gettext-0.17_1 (pkg_delete failed)
! libiconv-1.13.1 (pkg_delete failed)

This does a upwards recursive dependency removal. Also, if one dependency is relied on by another, it wont get removed. Like, if Perl58 was a dependency of a package, it wouldn’t be removed if perl58 is used by many other packages. This is smart. So, above, the packages that failed to deinstall where ones that are required dependencies of other installed packages.

Speaking of package management; have you ever installed something that ended up having a few dozen dependencies, then you want to uninstall that package with a “rpm -e cba8″, or something equivalent, but what about all the other cruft that came along with it? You would have to keep track of each dependency, and specify all of them and hope you don’t break another program. FreeBSD has a few tools to do this, one in particular, portmaster can remove all ports that were once a dependency but no longer used:

> portmaster -s
Information for neon28-0.28.4:
Comment:
An HTTP and WebDAV client library for Unix systems
===>>> neon28-0.28.4 is no longer depended on, delete? [n] y
===>>> Delete old and new distfiles for www/neon28
without prompting? [n] y
===>>> Running pkg_delete -f neon28-0.28.4
Information for rubygem-actionwebservice-1.2.6:
...

I ended up removing 4 packages that were no longer used.

CentOS and RHEL are the larger Puppet consumers, I’m still a big proponent for FreeBSD, and at work, it has allowed me to quickly build an Apache + Puppet + RubyPassenger/mod_rack stack with the minimal dependencies installed. So, the puppet server is still pretty lean, which means updates are smaller and faster. It still surprises me that its relatively unknown, even though Netcraft always has it listed in the top domains with the best uptime and consistently growing over the years. Why do I feel like an AmigaOS fan sometimes?

Hmm, it is sort of weird that this turned into a FreeBSD ports management entry :)

Okay, final word: PuppetCamp09 was Freaking awesome. There were a lot of smart developers and sysadmins there. We even got a very cool git howto, which I found useful. It was very diverse, which is strange for a conference based on one project in particular.

I went with Jenny (and Velimir) and the hiking group she goes with, the Society of Outdoor Cardinals, lead by “V”. I went with Jenny and “V” in January on a 14 mile hike up Mount Diablo. That trip was a good deal of fun, and I didn’t die so I naturally assumed that a 21 mile hike would be difficult, but not impossible…

The weather could have been better, it was supposed to be around 102, so I brought plenty of water (6L) and the first half of the trip was good. I was powering up the steep incline without a problem, and it wasn’t until right before lunch that I started to feel the convulsion of my leg muscles. I’ve had cramps here and there, usually after a strenuous class at Wushu West, but I can deal with it. However, while we sat down for lunch, I noticed (and even joked about it) my calf muscles were twitching. It was like they were dancing to some unknown rhythm. “V” said it was because I was low on salt ( and then he told me that every muscle contraction required both Potassium and Salt), and the simple evidence of my salt encrusted backpack was proof enough that I needed to do something. So, I had what I felt to be a salty lunch, and then got back on the trail for the last 3 miles to Rose Peak.

That last stretch of the trail was very intense for me, and soon lead up to a series of the most painful moments in my life.

Jenny, Velimir and I were sticking together, and maybe a mile in I wasn’t feeling so hot. I had to rest on a log because my right foot was twisted outwards to the far right, because my shin muscle was completely contracted from a cramp. This was uncomfortable, but still tolerable. I really hated holding them up, so I did my best to flex it out and get back on my feet. I always feel like I can tolerate my own pain pretty well, and I’d rather do that than feel like I’m being a nuisance to everyone else by resting so much. Resting is for sissies!

I don’t know how much further I had lasted, maybe another mile or so. My shins, calves, quads, and ham strings decided they would all get in on the cramping game, and they seized up all at once. I think I pointed out to Jenny that my quads were hard as a rock. I said this between my teeth though, as the pain was like nothing I have every felt before, and as much as I try to keep my cool, I did make some awful hissing and breathing sounds. I felt that standing was easier, for all but a second, and then I sort of plopped down, hopped up again because that didn’t feel any better, then sat down again. The thing was, when I had my legs straight, my quads would cramp up. If I bent my knees at all, my calves would join in, if I tried really anything, both legs would take turns locking up.

While I rested yet again, Jenny and Velimir offered my food, water, and it was unfortunate that I could hardly think or talk clearly due to the pain. What was cool, is two horses and three cows came around the corner and came right up to us. I’ve grown up around horses, so while they are not new to me, its very unnerving to have them roaming free without an owner.I also though, if anything happened, like they got spooked, I was in no shape to get out of the way. I tried to envision what it would be like to get trampled to death in my current state, and I actually thought that would be better :)

After some more time had passed, and I ate some salty jerky, I decided it was time to make rose peak my bitch and get back on my feet. I had come this far, and I did NOT want to give up for anything. So, Jenny and I hiked a bit further, and we came across Magda. She said we were almost there, about 15 more minutes… shortly after that, I had another cramp attack, and this was way more intense than the previous ones. After this, I started to get the suspicion that this was it, my hike was done for. Jenny ran off to the peak to see if anyone else was there (check that out, I’m freaking dying, and shes JOGGING up the trail and back :) ). When she got back, she helped me get comfortable, filled me up with more salty food, a banana, and handed me water. Jenny has terrific bedside manners, and I felt terrible for putting her through the trauma of my pain.

Once I was laid out on the trail, and Jenny was off to get someone else, I let out a nice howl, and dare I say a F-Bomb. I also managed to shove some water bottles under my knees. This help my legs stay in a neutral state, where my quads or calves were not working at all (even in this position, they were still 100% camped up). “V” came around, and then at that point I know it would have been REALLY stupid to tell “V” and Jenny that I still wanted to reach the peak. I knew that would have been pretty insulting, especially since my injury forced him to say behind with us. He was very nice, and both him and Jenny waited with my while my legs at least stopped locking up. Seriously, I felt completely betrayed by my own body, it was incredible what was going on. This is what I get for pushing my legs around all these years, making them run, kick and jump all the time :)

The only problem with that hike is I was only 1/2 way done. I had another 9 or so miles to go, and it was all up and down terrain. “V” had told me that once my cramping had subsided, and I got the right nutrients in me, I would be sore, but I would eventually get the blood flowing again and be okay.

Unfortunately, that was not the case at all. I’ve got some freaky body chemistry or sumthin’ cause there was no recovering from this. I was only slightly uncomfortable if we were on flat even ground, anything outside of that was either exerting force on my quads and shins (which would cramp up), or my calves and hamstrings (which would cramp up). So for the next few miles it was a slow process of hiking for a bit, then stopping to either nurse a new series of cramps away or try and prevent another wave.

It was also getting late, and thankfully both “V” and Jenny brought head lamps, so our nice day hike turned into a nice night hike :) “V” lead the four of us with his lamp, Magda and Velimir trailed him while I used Jenny’s headlamp and Jenny trailed behind me. We did catch a cool view of the sun going down, and there was some relief to hiking back in a cooler climate. I was also now using “V”s hiking poles and training myself to use my arms and the poles as much as my legs to move along. I was getting a pretty good workout, now I had to use my entire body to move myself forward as well as fighting my own muscles from cramping.

Let me relate this to the Wushu folks; You know when your doing a form and you are tense? I’ve heard it a lot, Mark or Patti will say “relax! your shoulders are too tight, loosen up.”, because it saps all your energy to try and move around with those muscles activated. That is was this was like, every step required twice the amount of effort.

The rest of the trip was pretty monotonous, I still had to take a break from time to time. There was one part where I had to take a large step UP, and it really sucked because by this time I had a very good idea as to what actions would incur the wrath of my legs. Basically, anytime I had to flex my muscles in a quick and major way, I would completely cramp up. So, I looked at the large step up, took a deep breath and did it, hobbled out of harms way (like, away from the ledge so I wouldn’t fall back) and stop immediately to rest and wait for it to pass.

Around 9:15pm, a pair of park rangers found us and asked if we were the droids they were looking for :) Velimir, who has a strange sense of humor said “Hey, we should hide from them !!” and then laughed. By this time, I was completely out of it, my body was shot, my brain was fried, and I was still dehydrated. Seeing these rangers was the best thing that could have happened. Sure, I felt a little robbed that I didn’t get to make it to the peak, OR finish off the last 2 miles of the hike, but really, it was for the best. We had also exceeded the park hours, so there wasn’t a choice, we were escorted out :)

We said out goodbyes, Jenny Drove those two back to Stanford, and I made my way home. I stopped by the gas station to pick up a few necessities:

• Gatorade, to continue and replenish my ph balance
• more water, as I had ran out
• New Castle… hey, “V” said it would help :)

The commute home was an adventure in itself, because Vasco Road is reduced to one lane in the evenings now. Cal Trans has decided the two year old asphalt needs replacing, and since that entire organization hates me, they feel the need to close down almost all of Vasco to one lane while they perform the same work they did two years ago. Normally this just results in a lot of my time waisted, sitting there in traffic. Tonight though, that was extra special. Driving in stop and go traffic with both legs cramping up is an interesting situation to be in. I spent most of my commute in tears.

When I finally go home, I had two more steps to face, those leading to my front door. That kicked off one of the last cramping sessions of the day, and it was quit the grand finally. I recounted the entire day to Michele as I winced in pain, and maybe 30 minutes later I got up and took a hot shower. That felt GREAT, I was already on the road to a good recovery after that. The only thing left was to have a salty dinner, a beer, and ice my legs down.  I ended going to bed around 1:30am, and as exhausted as I was, I hardly slept a wink. The pain was still bad enough to keep me awake all night, but at least I didn’t cramp up any more. The next day was pretty rough as well, I felt woozy and sick all day, be resting has become easier since then.

So, that was my hike. It is now Tuesday, and my legs are still very very sore. I still did weights Monday though :) Ha, I’m back bitches!

Freddy, what I am saying is: know your limitations. You are a moron.

This movie was complete shit… Ha, just kidding Jenny :)

No, really, it was hilarious. There is a lot of quick come backs and one liners that you really have to pay attention to catch. At the risk of sounding (even more) like an old man, they don’t make comedies like they used to. Really, you don’t see this kind of a movie now, this is the kind of comedy I like, way more than what Judd Apatow and the like pump out. It’s classy, yet its about coning people which isn’t classy at all.

Now, back to chasing kids off my lawn….

Over a month ago (Jan 31st), Jenny invited me to hike Mount Diablo with the Stanford Outing Group, lead by “V”. I’ve posted her photos to the gallery here, and you should read her post on our trip. Its more entertaining than mine :)

http://jettagirl.wordpress.com/2009/02/01/mount-diablo-hike-to-the-summit/

This was a 14 mile hike, took about 6-7 hours (I think) and made my legs feel like jelly. That’s definitely my kind of workout, now only if it was 20 miles instead…

The start of our hike, at Mitchell Canyon Road, is about 15 minutes away from my house. Thats right, I have a freaking mountain to hike in my backyard and I’ve never actually done it. For that matter, its been about 10 – 12 years since I was last at Mt. Diablo. I really do need a group of people to get me out of the house on a weekend. The entire group ( Jenny is excluded from this statement, because she’s always cool) was cool, and the leader, V, was hilarious. I loved how he would mention ways we could die, or how to check for ticks (or you’ll die!), or if you trip… you get the idea. He comes off as overly- cautious, but he was great to hike with and talk too.

This was my first hike, and I was glad I could ask Jenny before hand what I should bring or wear. I brought a couple of bottles of water, and I ended up re-filling one on the way back down. The three of you that check out this blog already know how much I sweat, so the minimum water recommendation for a normal human being should be doubled for me :)

The trip up was strenuous, I would zone out and put one foot in front of the othe during the real up-hill parts. We took a few breaks to get some water, have a snack, but the real enjoyment was when we reached the peak. After all the “hard” work was done, I felt I could look around more, check out the scenery, and even talk a little. On the way down, “V” and Jenny told me about the different rock types (“scree” is difficult to walk on), saw a little waterfall and a cave on the side of a cliff, and then we found our own little cave on the side of the path. It was cool.

Overall it was a awesome trip and I look forward to doing one again. I don’t think I’ll get lucky with it being 15 minutes away from my house any time soon though. Eventually I’ll have to start learning to get up before 8am :)

So thanks Jenny and “V” for a memorable trip.