Outside the GAMH

Since I say this after almost every concert I go to, I’ll not forget to repeat it again:

This was the BEST show I’ve ever seen.

There are many aspects why: smaller venue, and our seats were on the balcony. This gave us a great view of the show, and we were pretty darn close to everything. Oh, and you know, it is one of my favorite artist doing what he does best. Not to mentioned good food and beverages at a incredible price. The sound was great as well, not too loud, but enough to rumble you chest.

Even more triumphant, was the fact that Michele not only liked it, she LOVED it and requested that I load up her music devices with his music. His loud and insanely complex music, not the sweet stuff. There was only two mellow songs in that set, the rest were hair raising shred-fests. This is such an accomplishment, because leading up to this, she would ask me to play it in the car just so she could bitterly acclimate to it. Like it was cough medicine, she would sort of grimace, and roll her eyes, or take a jab at me and play the air guitar with a goofy face like “this is what you two look like!”

In the end, great music conquers all, and there is something powerful about live music. Living, breathing music in the air that you can almost taste.

Before I dive into that, lets get to the opening two man band, Wolff. A drummer and a tuba, and that tuba is a multi-purpose device. The musician uses it as a percussion instrument, a mic, and a sample device all at once. The sheer amount of skill he displayed creating and mixing samples of the sounds he made from it was incredible. He was mixing and looping effects on the fly for the entire set.

I also like watching drummers play. I told Michele that every time I see an opening band, I always like the drums. It makes me want to take lessons, or live vicariously though my kids and make one (or both!) of them pick up the drums. I think I’m also envious because I struggle with timing and rhythm. Total white-boy stereotype shining through.

So Wolff was cool, even though when they announced that there were only two more songs until Buckethead came out, I literally screamed inside my own head, “YAY!!!”

There is something about his music that resonates with me on a very deep level, more than any other artists. Despite what is observed by most, I do enjoy a lot of different music, but from the the ~140 artists currently on my “shuffle all” selection, not a one comes close.

Think about that, for yourself. Think of all the music you have heard, and narrow down the artists that give you goosebumps, or make your heart race, and make you smile really big.

I dunno, it seems to map well with my brain and how I see music. So I sat very still and very focused the entire set, except during Jordan, when I had to use the restroom.

That was funny because there was a line outside, and the guy behind me peeked in the Ladies room, looked back at me and said “What? *laughs* There are no ladies here, its all guys…”

Totally true. There were maybe 5 or 6 women there, and I would have guessed that most of them were like Michele (dates or spouses). Guitar players attract guys, period.

Anyway, the point is, I was mesmerized. A few times I would just shake my head in awe. Michele laughed because during the song Siege Engine, there is a part where Buckethead goes from the low E open note, hammers on the 3rd fret (G), hammers on the 8th fret (C). Then does the reverse ( pulls off the 8th fret, to the 3rd, and pulls off the 3rd back to open). This is all done in one very smooth rhythm, and if you have never laid your fingers across a guitar fret board, that is a LOOONG stretch. It always puts a smile on my face, I’ve worked at it many times, and it is demanding.

As far as his guitar work and accuracy, my goodness… the command he has over that instrument is ridiculous. I’m also amazed that he plays along with a MP3 player. That means he doesn’t get to take a break, or improvise a bit longer, he has to know his set inside and out without missing the next song. Sprinkle in the sweet robot dance moves, dispensing toys to the crowd (and getting a few back from his fans), and he wears a MASK. I actually don’t know how he can see much of anything with that on. It was so impressive that the both of us reminisced on the drive home. Michele even mentioned not only looking forward to seeing him again, but joked that we should go the next night. I would have, except it was sold out :)

The venue staff asked that no photographs (especially flash) or videos be taken, so I compromised… I took one photo without the flash:

The lighting just happened to be perfect for this shot. My phone isn't the best at taking low-light pictures

Otherwise, it was me scrunched up against the balcony so I could get as close as possible.

To finally wrap this up, I enjoyed it. I even heard a few new songs (Final Wars, pretty flippin’ awesome), so mre music to enjoy! Also the minimum age at GAMH is 6, guess who I’ll be taking in the future?

Oh don’t be so judgmental! My first concert was the Highwaymen, a “country super-group”, on the grassy field at the Concord Pavilion. It was terrible, this couldn’t be any worse and I’m pretty sure there was less pot smoked at the Buckethead show than at The Highwaymen’s show.

For an artist who doesn’t use any words, he knows how to say a lot. To me at least.

Cool I tell you!

Chris, Corrigan and I went to Google in 2008 for MeetBSD. We had a great time, so when I saw the announcement earlier this year for MeetBSD 2010, I rallied the troops.

The event was Friday and Saturday, and it took place at the Hacker Dojo in Mountain View. The format of the conference was the same that Puppet Camp had, where the talks were proposed and arranged by what the attendees wanted to talk about. So, the conference started off with James from iX Systems putting the Google Document URL up on a board, and then he let everyone vote and propose topics

Our main MC was James Nixon. The flow was great, and I think he did a very nice job and getting people to participate and tease out details from everyone. He did that by having everyone stand up and gather around after a break-out session was over. Then, he had everyone talk about their break-out session. As much as people probably hate being put on the spot, it really tore down some social barriers. I probably would have been a more passive participant, but a format like this helped me/forced me to be more engaging. It was excellent.

So, that was the first day, and I got to hear about CLANG and LLVM and what it means to the BSD community, and how the new LGPL license effects the base system (for example, no GCC 4 in the base of FreeBSD, it will have to be a Port). I got to hear Matt Dillon talk about in details his HAMMER File System (B-Tree file system with sweet rollback support), SSD drives, Hybrid Drives (and why they are not a good idea), and why ZFS and HAMMERFS will not outperform UFS in the use-case of a Database. Oh, and why Log devices are so critical for ZFS and HAMMER FS. I also got to tackle Pawel, the ZFS developer, and ask him about the roadmap for v28.

A few other really cool talks:

• David Maxwell – The State of The NetBSD Foundation. This was very fun and informative, he presented some of the details of NetBSD’s development and funding, while asking FreeBSD, OpenBSD and DragonflyBSD members to talk about their own. It was a great way to learn about the BSD’s and their difference. It was also a well done presentation because it was reciprocal and respectful off the variants. Really good.
• John Kanen Flowers – Kane|Box. John has a strong presense, and I am interested in seeing where Kane|Box goes. Since it is both a commercial hardware device, and an open source project, I may run it on my server at home.

I also spent some time randomly going up to people like Dru Lavigne, Pawel Dawidek, and Will Backman (of BSDTalk, one of my regular and favorite podcasts), just to tell them I appreciate what they do. I’ve always taken such a passive role in conferences like these, yet I’m so enthusiastic about the BSD’s, that I have to recognize that I AM a part of the community. I don’t develop, but I do blog, document, and talk as much as I can about them. Lets also not forget, I’m a happy end-user, being a SysAdmin of FreeBSD systems is pretty fun. The point is, I’m trying to put myself out there more, be more involved and participate in the discussion.

With FreeBSD reporting their largest status report this quarter, and the excitement everyone here seem to have, I think it confirms the solid and steady progress of the BSD’s. One person did comment to Chris, Corrigan and I about the age group at the conference. He didn’t say that he thought it was good or bad, but he said that the people in the BSD community tend to be “more mature”. I think he would like to see younger people more involved, and I agree. I’ve always felt that the BSD’s are driven from an engineering perspective. They don’t have marketing department telling them they need to include support for the latest buzz-word. This is good to me, otherwise, you see ridiculous things like clustering packages that are completely virtualized. However, the negative aspect to that is the lack of media attention. I sometimes thing it would slightly benefit the projects if people in the academic world saw it as ‘sexy’.

    > netstat -I em0 -w 1
                input          (em0)           output
       packets  errs      bytes    packets  errs      bytes colls
         90166     0   98762637      95363     0    5332847     0
         18131     0   24713156      20042     0    1123684     0
             4     0        310          1     0        178     0
             8     0        518          1     0        178     0
         10153     0   10952920      10696     0     598129     0
         92990     0  102837002      98476     0    5514994     0
         92025     0  102680574      97277     0    5439496     0
         92080     0  101799874      97403     0    5448637     0
         75348     0   90861608      80972     0    4537737     0
         90895     0  100323946      95781     0    5360948     0
         89313     0   97371154      94364     0    5278618     0
         81363     0   89229738      85861     0    4803589     0
             2     0        126          3     0        286     0

I was so shocked that I had to use gstat and zpool iostat to verify the information:

    dT: 1.002s  w: 1.000s  filter: da0
     L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
       35   1476      0      0    0.0   1476 188421   23.7  100.0| da0

    > zpool iostat  1
                   capacity     operations    bandwidth
    pool         used  avail   read  write   read  write
    ----------  -----  -----  -----  -----  -----  -----
    tank        5.68T  4.32T      1     81   250K  10.1M
    tank        5.68T  4.32T      0  1.37K      0   175M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M

This is all through Samba (3.3.9), There was no local work being done. I unfortunately didn’t configure MRTG correctly, so it had built a malformed graph while all this happened. Having a picture from all of this would have been nice.

The underlying storage is a SATABoy2 RAID6 array, with a simple “flat” ZFS filesystem (version 13). As cheap as the SATABoy’s are (and come on, they have a terrible IIS web interface), they can at least keep up with the current load.

I have felt that if you are going to use ZFS, you should let it manage the RAID, and not bother with a hardware RAID controller. While the hardware RAID may be faster, ZFS’s ability to self-correct bad blocks is a great feature despite the performance set back. However, RAID6 is pretty good in itself, and having dual parity would ideally reduce the risk of a bad block being detrimental.

One thing I noticed with Samba is it doesn’t seem to be a threaded daemon. When I do a top(1) -H, there are only 2-3 smbd processes, and one of them is running around 30%. Though I don’t really know how well Samba can scale out, this environment only has about 10 users. I would like to see how samba reacts if there are a couple hundred active users. Furthermore, how does a native Windows server handle a couple hundred users? It may handle it a little better, however, I don’t think I would enjoy watching NTFS handling a multi-terabyte volume… it would be like watching a stroke victim eat a bowl of soup. I do admit I am biased and I have no working experience with Windows as a large file server, most of them that I have worked on are horribly limited and underpowered, and no one seems to care if they perform well or not.

Hardware

CPU information

    Machine class:    amd64
    CPU Model:    Dual Core AMD Opteron(tm) Processor 285
    No. of Cores:    4
    Cores per CPU:

RAM information

    Memory information from dmidecode(8)
    Maximum Capacity: 8 GB
    Number Of Devices: 4
    Maximum Capacity: 8 GB
    Number Of Devices: 4

    INFO: Run `dmidecode -t memory` to see further information.

    System memory summary
    Total real memory available:    8048 MB
    Logically used memory:        2876 MB
    Logically available memory:    5172 MB

    Swap information
    Device          1K-blocks     Used    Avail Capacity
    /dev/da1s1b       8373844      28K     8.0G     0%

Storage information

    Available hard drives:
    cd0:  Removable CD-ROM SCSI-0 device
    cd0: 1.000MB/s transfers
    da2:  Fixed Direct Access SCSI-5 device
    da2: 300.000MB/s transfers
    da2: Command Queueing enabled
    da2: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C)
    da1:  Fixed Direct Access SCSI-2 device
    da1: 300.000MB/s transfers
    da1: Command Queueing enabled
    da1: 69618MB (142577664 512 byte sectors: 255H 63S/T 8875C)
    da0:  Fixed Direct Access SCSI-5 device
    da0: 200.000MB/s transfers
    da0: Command Queueing enabled
    da0: 10491861MB (21487333120 512 byte sectors: 255H 63S/T 1337524C)

    Raid controllers:
    umass-sim0:
    mpt0:
    vendor='LSI Logic (Was: Symbios Logic, NCR)'
    device='SAS 3000 series, 4-port with 1064 -StorPort'
    isp0:
    vendor='QLogic Corporation'
    device='QLA6322 Fibre Channel Adapter'

    Currently mounted filesystems:
    /dev/da1s1a on /
    devfs on /dev
    tank on /tank
    /dev/ufs/EXPORT on /export

    I/O statistics:
           tty             da0              da1              da2             cpu
     tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
       0    40 63.61 167 10.36  16.53   2  0.03  61.65   0  0.00   1  0  4  0 94
    INFO: Run iostat(8) or gstat(8) to see live statistics.

    Disk usage:
    Filesystem         Size    Used   Avail Capacity  Mounted on
    /dev/da1s1a         58G    3.4G     50G     6%    /
    devfs              1.0K    1.0K      0B   100%    /dev
    tank               9.8T    5.7T    4.1T    58%    /tank
    /dev/ufs/EXPORT    126G    148K    116G     0%    /export

Software

• FreeBSD 8.0-RELEASE-p1 FreeBSD 8.0-RELEASE-p1 amd64
• samba-3.3.9 A free SMB and CIFS client and server for UNIX

Samba 3.3.9 Compile-Time Config

> make showconfig
===> The following configuration options are available for samba-3.3.9:
     LDAP=on "With LDAP support"
     ADS=on "With Active Directory support"
     CUPS=off "With CUPS printing support"
     WINBIND=on "With WinBIND support"
     SWAT=off "With SWAT WebGUI"
     ACL_SUPPORT=on "With ACL support"
     AIO_SUPPORT=on "With Asyncronous IO support"
     FAM_SUPPORT=on "With File Alteration Monitor"
     SYSLOG=on "With Syslog support"
     QUOTAS=on "With Disk quota support"
     UTMP=off "With UTMP accounting support"
     PAM_SMBPASS=on "With PAM authentication vs passdb backends"
     DNSUPDATE=off "With dynamic DNS update(require ADS)"
     DNSSD=off "With DNS service discovery support"
     EXP_MODULES=on "With experimental modules"
     POPT=on "With system-wide POPT library"
     MAX_DEBUG=off "With maximum debugging"
     SMBTORTURE=off "With smbtorture"
===> Use 'make config' to modify these settings

System Tuning

The Kernel

I enabled device polling, and took out debugging in the kernel (Sanders, get it! Mmm, I’m hungry…)

diff /usr/src/sys/amd64/conf/GENERIC /usr/src/sys/amd64/conf/SANDERS
    33d32
    < makeoptions    DEBUG=-g        # Build kernel with gdb(1) debug symbols
    78c77
    <
    ---
    > options        DEVICE_POLLING

/boot/loader.conf

    ispfw_load="YES"
    kern.hz="2000"
    aio_load="YES"

/etc/sysctl.conf

    kern.coredump=0
    security.bsd.see_other_uids=0
    security.bsd.see_other_gids=0
    kern.ipc.maxsockbuf=16777216
    kern.ipc.nmbclusters=32768
    kern.ipc.somaxconn=32768
    kern.maxfiles=65536
    kern.maxfilesperproc=32768
    kern.maxvnodes=800000
    net.inet.tcp.delayed_ack=0
    net.inet.tcp.inflight.enable=0
    net.inet.tcp.path_mtu_discovery=0
    net.inet.tcp.recvbuf_auto=1
    net.inet.tcp.recvbuf_inc=524288
    net.inet.tcp.recvbuf_max=16777216
    net.inet.tcp.recvspace=65536
    net.inet.tcp.sendbuf_auto=1
    net.inet.tcp.sendbuf_inc=524288
    net.inet.tcp.sendspace=65536
    net.inet.udp.maxdgram=57344
    net.inet.udp.recvspace=65536
    net.local.stream.recvspace=65536
    net.inet.tcp.sendbuf_max=16777216
    net.inet.tcp.mssdflt=9142

rc.conf (em0 flags)

I want to thank Zilla (see post comments) for the sysctl.conf help.

    ifconfig_em0="inet xxx.xxx.xxx.xxx  netmask 255.255.255.0 polling tso mtu 9194"

smb.conf

        min receivefile size = 131072
        aio read size = 1
        aio write size = 1
        use sendfile = yes
        lock directory = /var/run/samba/
        keepalive = 300

I’m also using LDAP users and group. I wasn’t sure if there would be a noticible performance hit for local users or LDAP users. There doesn’t seem to be one.

We use Active Directory, and since Quest/Vintela still won’t make a FreeBSD client for the Quest Authentication Servers ( a sales rep once told me “There are just too many versions of BSD…”) , I have to use all the open source utilities like OpenSSL, OpenLDAP Client and Kerberos. I don’t mind having to do it, but it is always nice if you can maintain one standard process across ALL systems, and we have a lot more Linux and Solaris systems than FreeBSD. I’m the odd one.

That aside, I use the latest OpenSSL in FreeBSD 8.0, OpenLDAP 2.4.20, and the built-in version of Heimdal Kerberos.

I get similar performance form NFS, however, most desktop users have are either on a Windows or OS X, and CIFS seems to be the unifying network storage protocol.

One thing I have yet to really figure out is configuring Samba to use proper NT ACL’s. However, if you can live with UNIX style permissions, a setup like this is pretty good at serving out lots and lots of data. Maybe that will be next.

What also made this conference unique, is how honest the Puppet team and community were about the projects strength and weaknesses. Those that have deployed Puppet on a larger scale (MessageOne and Google) seemed to go through the same iterations in attempting to scale out their Puppetmaster’s. From WEBrick (which is what I’m currently running Puppet with :) ), which is hated by all since its a single process/thread web server that can only handle one request at a time. To Mongrel, which you have to manage a mongrel cluster script, feed it lots of memory, and then throw an apache proxy server in front of them. Now, people are starting to settle on using Passenger/mod_rack, which is what I spent most of yesterday looking into and setting up. This allows apache to mount a rails instance, and then you don’t actually have to run puppetmasterd. This still requires some decent hardware, and I’m currently running my puppetmaster on a VM with 2GB or memory, so I’ll have to watch out for that. Chris, the one who introduced me to Puppet, said he still uses WEBrick for all of his DB, Tomcat, and Apache servers (I think he said something like 200 systems) and it has been working out nicely. He, like the guys at Google, also doesn’t run puppet as a daemon.

Anyway, the point is, we learned a lot about the project, way more than if a sales person had come to us and just told us the things puppet does well, or how it operates on paper (*cough* LANDesk *cough*). It was really awesome to talk with Andrew Pollock and Nigel Kersten from Google. See, I was a little unsure about Puppet in our environment, where we have multi-purpose servers, computer servers, and desktops that we have to manage. It seemed, at a first glance, that most of the Puppet users out there have a homogeneous environment, and Andrew (Shafer) had stressed the concept of single role servers. After talking with them, I felt a lot more comfortable pursuing Puppet across our servers and desktops. Did I mention they were super cool and friendly?

We also learned a lot about the Puppet developers, which had its own interesting advantage. I have a lot of respect for what Luke Kanies has been able to do, and by the end of the conference, he showed significant mastery in what he has done, as well as some humility and admitting what he has not been able to do and why. I was a little put off the first day though, when both him and Andrew came off a little arrogant and crass. It did make me step back and think, “Is this project going to be well managed in the future with personalities like this in charge? Is their answer of ‘don’t do that!’ tongue in cheek, or are they not supportive of a diverse environment?”. In the end, I have more respect for the project than ever, and with it still being a young project, I hope they listened to some of the feedback, and I also can’t wait to see where it ends up in the next year.

Andrew, the Puppet Andrew, came up to us a lot during the conference, and he was fun to talk too, and he’s very academic and he had a lot of abstract concepts to talk about. Also, he said this was the first conference he has arranged, and I think he did a fantastic job. Jenny had commented that this was the first conference she had lasted the entire duration, so that says a lot about the pacing and content of PuppetCamp. I felt the same way, every session was incredibly engaging, and how Andrew had setup the democratic and chaotic Open Sessions was very impressive. Lets put it this way, I even got up there and pitched a topic, which is something I would have never done. Hurray for me stepping outside of my comfort zone!

Warning: side topic!

Now that I’ve had the weekend to google all the cool technologies I was exposed too, I’m also reminded why I really like having a FreeBSD server at my disposal. They had talked about CouchDB, so on a whim I did a

~> cd /usr/ports
/usr/ports> make search name=couchdb
Port: couchdb-0.9.0_1,1
Path: /usr/ports/databases/couchdb
Info: A document database server, accessible via a RESTful JSON API
Maint: till@php.net
B-deps: ca_root_nss-3.11.9_2 curl-7.19.6_1 erlang-lite-r13b01_6,1 gettext-0.17_1 gmake-3.81_3 icu-3.8.1_2 libiconv-1.13.1 libtool-2.2.6a nspr-4.8 perl-5.8.9_3 spidermonkey-1.7.0
R-deps: ca_root_nss-3.11.9_2 curl-7.19.6_1 erlang-lite-r13b01_6,1 gettext-0.17_1 gmake-3.81_3 icu-3.8.1_2 libiconv-1.13.1 libtool-2.2.6a nspr-4.8 perl-5.8.9_3 spidermonkey-1.7.0
WWW: http://couchdb.org/

Port: py26-simplecouchdb-0.9.26
Path: /usr/ports/databases/py-simplecouchdb
Info: Simple Librairy to Allow Python Applicationto Use CouchDB
Maint: wenheping@gmail.com
B-deps: py26-httplib2-0.5.0 py26-py-restclient-1.3.2 py26-setuptools-0.6c9 python26-2.6.2_3
R-deps: py26-httplib2-0.5.0 py26-py-restclient-1.3.2 py26-setuptools-0.6c9 python26-2.6.2_3
WWW: http://code.google.com/p/py-simplecouchdb/

I did a ‘make install’, and I had a cool little couchdb up and running. What is also cool is FreeBSD likes to give you very helpful information when you install something. For example, this is what is printed out when you install the CouchDB port:

===> COMPATIBILITY NOTE:
CouchDB is still pre-stable; between 0.8 and 0.9 the database format
changed which breaks BC. In current trunk, the format changed again, so
please double-check in case you are updating an existing installation.

More info:
* http://wiki.apache.org/couchdb/Breaking_changes?action=show&redirect=BreakingChanges
* http://wiki.apache.org/couchdb/BreakingChangesUpdateTrunkTo0Dot9

See, isn’t that helpful? Best of all, I didn’t have to enable additional repositories, or fetch the src manually, and its dependencies and then figure out how to run the right configure script flags… FreeBSD makes it easy, and since it automatically uses what you already have with what is required, its an incredibly stable build. Removing it is pretty simple as well, just:

> pkg_deinstall -R couchdb
---> Deinstalling 'couchdb-0.9.0_1,1'
---> Deinstalling 'erlang-lite-r13b02,1'
[Updating the pkgdb
in /var/db/pkg ... - 118 packages found (-1 +0) (...) done]
---> Deinstalling 'curl-7.19.6_1'
[Updating the pkgdb
in /var/db/pkg ... - 117 packages found (-1 +0) (...) done]
---> Deinstalling 'ca_root_nss-3.11.9_2'
---> Deinstalling 'spidermonkey-1.7.0'
---> Deinstalling 'nspr-4.8'
[Updating the pkgdb
in /var/db/pkg ... - 116 packages found (-1 +0) (...) done]
---> Deinstalling 'gmake-3.81_3'
[Updating the pkgdb
in /var/db/pkg ... - 115 packages found (-1 +0) (...) done]
---> Deinstalling 'perl-threaded-5.8.9_3'
[Updating the pkgdb
in /var/db/pkg ... - 114 packages found (-1 +0) (...) done]
---> Deinstalling 'gettext-0.17_1'
---> Deinstalling 'libiconv-1.13.1'
---> Deinstalling 'icu-3.8.1_2'
---> Deinstalling 'libtool-2.2.6a'
** Listing the failed packages (-:ignored / *:skipped / !:failed)
! curl-7.19.6_1 (pkg_delete failed)
! ca_root_nss-3.11.9_2 (pkg_delete failed)
! perl-threaded-5.8.9_3 (pkg_delete failed)
! gettext-0.17_1 (pkg_delete failed)
! libiconv-1.13.1 (pkg_delete failed)

This does a upwards recursive dependency removal. Also, if one dependency is relied on by another, it wont get removed. Like, if Perl58 was a dependency of a package, it wouldn’t be removed if perl58 is used by many other packages. This is smart. So, above, the packages that failed to deinstall where ones that are required dependencies of other installed packages.

Speaking of package management; have you ever installed something that ended up having a few dozen dependencies, then you want to uninstall that package with a “rpm -e cba8″, or something equivalent, but what about all the other cruft that came along with it? You would have to keep track of each dependency, and specify all of them and hope you don’t break another program. FreeBSD has a few tools to do this, one in particular, portmaster can remove all ports that were once a dependency but no longer used:

> portmaster -s
Information for neon28-0.28.4:
Comment:
An HTTP and WebDAV client library for Unix systems
===>>> neon28-0.28.4 is no longer depended on, delete? [n] y
===>>> Delete old and new distfiles for www/neon28
without prompting? [n] y
===>>> Running pkg_delete -f neon28-0.28.4
Information for rubygem-actionwebservice-1.2.6:
...

I ended up removing 4 packages that were no longer used.

CentOS and RHEL are the larger Puppet consumers, I’m still a big proponent for FreeBSD, and at work, it has allowed me to quickly build an Apache + Puppet + RubyPassenger/mod_rack stack with the minimal dependencies installed. So, the puppet server is still pretty lean, which means updates are smaller and faster. It still surprises me that its relatively unknown, even though Netcraft always has it listed in the top domains with the best uptime and consistently growing over the years. Why do I feel like an AmigaOS fan sometimes?

Hmm, it is sort of weird that this turned into a FreeBSD ports management entry :)

Okay, final word: PuppetCamp09 was Freaking awesome. There were a lot of smart developers and sysadmins there. We even got a very cool git howto, which I found useful. It was very diverse, which is strange for a conference based on one project in particular.

I went with Jenny (and Velimir) and the hiking group she goes with, the Society of Outdoor Cardinals, lead by “V”. I went with Jenny and “V” in January on a 14 mile hike up Mount Diablo. That trip was a good deal of fun, and I didn’t die so I naturally assumed that a 21 mile hike would be difficult, but not impossible…

The weather could have been better, it was supposed to be around 102, so I brought plenty of water (6L) and the first half of the trip was good. I was powering up the steep incline without a problem, and it wasn’t until right before lunch that I started to feel the convulsion of my leg muscles. I’ve had cramps here and there, usually after a strenuous class at Wushu West, but I can deal with it. However, while we sat down for lunch, I noticed (and even joked about it) my calf muscles were twitching. It was like they were dancing to some unknown rhythm. “V” said it was because I was low on salt ( and then he told me that every muscle contraction required both Potassium and Salt), and the simple evidence of my salt encrusted backpack was proof enough that I needed to do something. So, I had what I felt to be a salty lunch, and then got back on the trail for the last 3 miles to Rose Peak.

That last stretch of the trail was very intense for me, and soon lead up to a series of the most painful moments in my life.

Jenny, Velimir and I were sticking together, and maybe a mile in I wasn’t feeling so hot. I had to rest on a log because my right foot was twisted outwards to the far right, because my shin muscle was completely contracted from a cramp. This was uncomfortable, but still tolerable. I really hated holding them up, so I did my best to flex it out and get back on my feet. I always feel like I can tolerate my own pain pretty well, and I’d rather do that than feel like I’m being a nuisance to everyone else by resting so much. Resting is for sissies!

I don’t know how much further I had lasted, maybe another mile or so. My shins, calves, quads, and ham strings decided they would all get in on the cramping game, and they seized up all at once. I think I pointed out to Jenny that my quads were hard as a rock. I said this between my teeth though, as the pain was like nothing I have every felt before, and as much as I try to keep my cool, I did make some awful hissing and breathing sounds. I felt that standing was easier, for all but a second, and then I sort of plopped down, hopped up again because that didn’t feel any better, then sat down again. The thing was, when I had my legs straight, my quads would cramp up. If I bent my knees at all, my calves would join in, if I tried really anything, both legs would take turns locking up.

While I rested yet again, Jenny and Velimir offered my food, water, and it was unfortunate that I could hardly think or talk clearly due to the pain. What was cool, is two horses and three cows came around the corner and came right up to us. I’ve grown up around horses, so while they are not new to me, its very unnerving to have them roaming free without an owner.I also though, if anything happened, like they got spooked, I was in no shape to get out of the way. I tried to envision what it would be like to get trampled to death in my current state, and I actually thought that would be better :)

After some more time had passed, and I ate some salty jerky, I decided it was time to make rose peak my bitch and get back on my feet. I had come this far, and I did NOT want to give up for anything. So, Jenny and I hiked a bit further, and we came across Magda. She said we were almost there, about 15 more minutes… shortly after that, I had another cramp attack, and this was way more intense than the previous ones. After this, I started to get the suspicion that this was it, my hike was done for. Jenny ran off to the peak to see if anyone else was there (check that out, I’m freaking dying, and shes JOGGING up the trail and back :) ). When she got back, she helped me get comfortable, filled me up with more salty food, a banana, and handed me water. Jenny has terrific bedside manners, and I felt terrible for putting her through the trauma of my pain.

Once I was laid out on the trail, and Jenny was off to get someone else, I let out a nice howl, and dare I say a F-Bomb. I also managed to shove some water bottles under my knees. This help my legs stay in a neutral state, where my quads or calves were not working at all (even in this position, they were still 100% camped up). “V” came around, and then at that point I know it would have been REALLY stupid to tell “V” and Jenny that I still wanted to reach the peak. I knew that would have been pretty insulting, especially since my injury forced him to say behind with us. He was very nice, and both him and Jenny waited with my while my legs at least stopped locking up. Seriously, I felt completely betrayed by my own body, it was incredible what was going on. This is what I get for pushing my legs around all these years, making them run, kick and jump all the time :)

The only problem with that hike is I was only 1/2 way done. I had another 9 or so miles to go, and it was all up and down terrain. “V” had told me that once my cramping had subsided, and I got the right nutrients in me, I would be sore, but I would eventually get the blood flowing again and be okay.

Unfortunately, that was not the case at all. I’ve got some freaky body chemistry or sumthin’ cause there was no recovering from this. I was only slightly uncomfortable if we were on flat even ground, anything outside of that was either exerting force on my quads and shins (which would cramp up), or my calves and hamstrings (which would cramp up). So for the next few miles it was a slow process of hiking for a bit, then stopping to either nurse a new series of cramps away or try and prevent another wave.

It was also getting late, and thankfully both “V” and Jenny brought head lamps, so our nice day hike turned into a nice night hike :) “V” lead the four of us with his lamp, Magda and Velimir trailed him while I used Jenny’s headlamp and Jenny trailed behind me. We did catch a cool view of the sun going down, and there was some relief to hiking back in a cooler climate. I was also now using “V”s hiking poles and training myself to use my arms and the poles as much as my legs to move along. I was getting a pretty good workout, now I had to use my entire body to move myself forward as well as fighting my own muscles from cramping.

Let me relate this to the Wushu folks; You know when your doing a form and you are tense? I’ve heard it a lot, Mark or Patti will say “relax! your shoulders are too tight, loosen up.”, because it saps all your energy to try and move around with those muscles activated. That is was this was like, every step required twice the amount of effort.

The rest of the trip was pretty monotonous, I still had to take a break from time to time. There was one part where I had to take a large step UP, and it really sucked because by this time I had a very good idea as to what actions would incur the wrath of my legs. Basically, anytime I had to flex my muscles in a quick and major way, I would completely cramp up. So, I looked at the large step up, took a deep breath and did it, hobbled out of harms way (like, away from the ledge so I wouldn’t fall back) and stop immediately to rest and wait for it to pass.

Around 9:15pm, a pair of park rangers found us and asked if we were the droids they were looking for :) Velimir, who has a strange sense of humor said “Hey, we should hide from them !!” and then laughed. By this time, I was completely out of it, my body was shot, my brain was fried, and I was still dehydrated. Seeing these rangers was the best thing that could have happened. Sure, I felt a little robbed that I didn’t get to make it to the peak, OR finish off the last 2 miles of the hike, but really, it was for the best. We had also exceeded the park hours, so there wasn’t a choice, we were escorted out :)

We said out goodbyes, Jenny Drove those two back to Stanford, and I made my way home. I stopped by the gas station to pick up a few necessities:

• Gatorade, to continue and replenish my ph balance
• more water, as I had ran out
• New Castle… hey, “V” said it would help :)

The commute home was an adventure in itself, because Vasco Road is reduced to one lane in the evenings now. Cal Trans has decided the two year old asphalt needs replacing, and since that entire organization hates me, they feel the need to close down almost all of Vasco to one lane while they perform the same work they did two years ago. Normally this just results in a lot of my time waisted, sitting there in traffic. Tonight though, that was extra special. Driving in stop and go traffic with both legs cramping up is an interesting situation to be in. I spent most of my commute in tears.

When I finally go home, I had two more steps to face, those leading to my front door. That kicked off one of the last cramping sessions of the day, and it was quit the grand finally. I recounted the entire day to Michele as I winced in pain, and maybe 30 minutes later I got up and took a hot shower. That felt GREAT, I was already on the road to a good recovery after that. The only thing left was to have a salty dinner, a beer, and ice my legs down.  I ended going to bed around 1:30am, and as exhausted as I was, I hardly slept a wink. The pain was still bad enough to keep me awake all night, but at least I didn’t cramp up any more. The next day was pretty rough as well, I felt woozy and sick all day, be resting has become easier since then.

So, that was my hike. It is now Tuesday, and my legs are still very very sore. I still did weights Monday though :) Ha, I’m back bitches!

PC-BSD KDE Desktop

PC-BSD is a nice mesh between FreeBSD and a ready to use Desktop (which uses about 6GB of disk space). It is based on FreeBSD 7.2, so it has all the cool features of the latest release. Best of all, without ANY additional configuration, I was able to:

• Use the official FreeBSD nVidia driver for hardware acceleration
• Watch clips on YouTube (with flashplayer)
• Play back all sorts of media types like mp3′s, divx, mpegs,wmv,qt…
• use ZFS
• Create and edit documents with the latest OpenOffice 3.1
• Browse the web with Firefox 3.5
• Create VM’s with VirtualBox

Plus, if there wasn’t a PBI package for what I wanted, I could still use FreeBSD’s pkg_add or, cd to /usr/ports and make one. I would say, that’s pretty impressive for a commercially supported Unix platform.

Otherwise, it is very much like Fedora or Ubuntu, where it has an update manager (updates PBI’s and the system), network manager, helpful tutorials, and for once (for FreeBSD at least) a full blown X11/QT graphical installer. FreeBSD has always had a simple ncurses installer, which I like, but it tends to frighten a lot of people who are used to GUI installers.

Once strange thing it does is place all PC-BSD binaries in /usr/PCBSD. I guess this is to remain independent and out of the way of the base FreeBSD binaries, as well as /usr/local, which is the normal prefix for all Ports.

To wrap it up; my initial impression of PC-BSD is a positive one. I like how I could use the FreeBSD ports and package system and it did not conflict with the PC-BSD packages that were installed. I like the installer, and the storage options at install time (UFS2+SU, or UFS2+Journal, Encrypted swap…). With all OS’s, it normally takes a few weeks of using it to see its weaknesses, so I’m sure PC-BSD has some issues waiting to pop up. The only one I see right now is that KDE4 is the default GUI, and I prefer Gnome. I could install it, but it wasn’t an up front, as all of PC-BSD’s install tools are written in QT. The initial X setup tool was pretty slick, and it worked with my picky laptop.

I spent the weekend at Google for MeetBSD in celebration of FreeBSD’s 15 birthday. I drove 70.2×4 miles, on a weekend, and for Saturday I got up slightly earlier than I would have for work. All worth it, this was the coolest mini-conference I’ve been to. Of course, the last conference I went to was BSDCon in 2003, and that was nice as well. The point is, I don’t get out all that much when it comes to conferences, I just make an exception for my favorite UNIX OS, FreeBSD.

When I got there for the registration (9am) I stood in line next to two gentlemen from Sweden, Karl and Pontus (or Pontis, didn’t know for sure). I call them gentlemen because they were EXTREMELY polite; so polite I was audacious enough to assume they were not from the US :) So, I chatted with them for a bit, picked up my swag-bag and waited outside for Chris and Corrigan to arrive. The swag bag was the most impressive one I’ve seen so far (even more than what Splunk usually provides). It contained a shirt, shot glass, hip flask, coffee mug, mouse pad, cool FreeBSD propaganda, a mouse pad, and a 2GB flash drive! The shot glass and hip flask were little hints that those involved like to party a little, and while I didn’t attend the after party, people were talking about it the next day.

Corrigan arrived first, then Chris, and we sat down just in time for the first speaker.

FreeBSD Network Stack Performance – Optimizations for Modern Hardware

by Robert Watson

Probably the best talk of the weekend. Robert Watson is really good speaker, and his presentation was well laid out and was just at my level. He did start off saying this talk wasn’t for kernel hackers or developers. It was really cool to see how things like TCP Offloading in hardware effects the network stack, the kernel, and how it can cause some complications with certain kernel level utilities like TCPDump, firewalls and packet filters.

Isolating Cluster Jobs for Predictability and Performance

by Brooks Davis

Brooks had an interesting presentation, and I’ve always wanted to see FreeBSD in the cluster market even though I know Linux has won that one out by far. I did feel the presentation was a little unfocused, or maybe i misinterpreted the title to mean something else. It was still interesting to see how you could use various methods to allow engineers different tiers of a cluster. His FreeBSD cluster of 1400 cores is controlled by Sun Grid Engine.

Embedding FreeBSD

by M. Warner Losh

I liked this one, and M. Warner Losh was a good presenter. Having him walk through how he prototypes an embedded device, the tools used to strip it down, and then getting it to boot in 2 seconds was very impressive. He also had a very good sense of humor throughout the conference.

PC-BSD 7 from a Developers Perspective

by Kris Moore

PC-BSD is geared towards the Desktop/Laptop market, and has a corporate backing. I can easily compare it to Ubuntu or Redhat Enterprise Linux. Its still FreeBSD, no arguing that, and you can still do all the cool things on it like use the Ports system, but it has a new and efficient package system like YUM+RPM or Ubuntu’s Synaptic/APT system. This is crucial in my mind for a desktop environment, especially laptops, where you don’t want to build everything from the Ports tree (which is all source code), you want a standard and consistent system, where you won’t always be building a custom web server. It was a good talk, but one thing that sort of drives me crazy is the name. See, its PC-BSD, but really FreeBSD, so in my kind of environment at work where we tightly control what is on the network; how are we supposed to classify this system? As PC-BSD, or FreeBSD? If someone asks for a FreeBSD desktop, do was say they are using PC-BSD or FreeBSD? Thats just my thing with it. Aside from that, I really welcome a corporate aspect to FreeBSD.

BSD Certification

by Dru Lavigne

There is now a certification process for FreeBSD. It’s new, and right now there is the BSDA certificate which is aimed at Jr. System Administrators. It covers FreeBSD, NetBSD, and OpenBSD, and the explanation of what went into the test, the Psychometrics involved, and the awesome price point of $75 for the exam. This was so it would be widely affordable to everyone across the globe. I thought of taking the test, but I was having too much fun and I didn’t want to feel that pre-test anxiety.

Xen Virtualization on FreeBSD

by Kip Macy

I’ve had some exposure to Xen with Redhat, so its nice to see it on FreeBSD, and it was really a quick status update on the project. Kip then went into something completely different and I didn’t take any notes on it so it’s already lost in my brain :)

From here the PC-BSD team brought out “BSDGirl”, which was the equivalent of a booth babe. After a few minutes of her parading around I turned to Chris and Corrigan and talked about how I felt about it. See, I hate to judge her, I mean, she may really be into FreeBSD and likes to dress up (and she does seem to have a web presence pimping out PC-BSD). It was the fact that she seemed to isolate herself from everyone there, and didn’t seem to talk to anyone or network with the folks there. I just felt the event was cheapened with a marketing stunt like that and Chris and Corrigan felt exactly the same way. I don’t know, that’s the kind of crap I expect from big companies that don’t care, not the small BSD community that I take personally. It didn’t seem like anyone else there was all that interested in her, so it was even more out of place.

We hung out until the Google staff pretty much kicked us out, and Chris and Corrigan went to the after party for a little bit. I headed back home, watched a little TV and went to bed so I was ready for day 2.

Here is a quick rundown of it all:

• Intel Core 2 Duo E6750 (2.66Mhz 4MB cache)
• Intel S975XBX2 workstation motherboard
• AMCC 3Ware 9650SE 4 port SATA RAID controller (4x PCI-e)
  • Batter backup for the 3Ware so I can enable cached writes
• 2GB ECC Crucial Memory Kit
• 750 Watt PC Power & Cooling power supply
• ASUS EN6200 LE 16x PCI-e nVidia GFX card
• Plextor DVD+RW PX-810SA SATA
• 4 Western Digital 1TB Drives

All of this was of the highest quality, and Chris said since he got my into FreeBSD, he felt I should have a stable and rock solid system since my current “server” has died 6 times. So after getting it all put together and powering it up… it wouldn’t post. I swapped the cpu with an older Pentium 4D that I’ve had lying around until my htpc comes back up and that worked. It turns out, the motherboard doesn’t support the 1333Mhz bus speed of the E6750 Core 2 Duo. So I’ve done some testing with the p4 as the cpu, and most of the tests were I/O bound and not CPU.

First off was to test out ZFS. Since it is new to FreeBSD (new in general really) I followed a ZFS tuning guide for 7.0 and followed some pretty stock directions:

$ zpool create tank raidz da0 da1 da2 da3

Which automatically mounted a 2.7TB filesystem. This was a lot nicer than fooling around with partitioning and filesystem tools. I also like the feature set of ZFS compared to standard RAID’s like self healing and data checksums. Performance was a little slower. Doing a simple ‘dd’ with 1mb block size showed about 101MB/sec

 dd if=/dev/zero of=/tank/1gb.dat bs=1m count=1000

1000+0 records in

1000+0 records out

1048576000 bytes transferred in 10.420047 secs (100630640 bytes/sec)

Not bad, and since my GigE network cannot saturate that type of I/O I’m pretty satisfied with those results. I use this system as a network file server, among the other network related services like NAT, www, and mail, so my biggest concern was getting more than 80MB/sec (around gigabit ethernet’s limit).

BUUUT I am underutilizing my fancy 3Ware raid controller, so I can’t just leave that alone. I blew away the ZFS volume and created a full RAID5 with the controller. Everything was fine with the exception of fdisk, which isn’t usable for large volumes. I emailed 3Ware’s support wondering why my 2.78TB volume was only being partitioned at 722GB. They quickly responded with ‘use gpt’, which I did:

$ gpt create /dev/da0

$ gpt add -t ufs /dev/da0

/dev/da0p1 added

$ gpt show /dev/da0

	       start        size  index  contents

	           0           1         PMBR

	           1           1         Pri GPT header

	           2          32         Pri GPT table

	          34  5859311549      1  GPT part - FreeBSD UFS/UFS2

	  5859311583          32         Sec GPT table

	  5859311615           1         Sec GPT header

I did a newfs -U -O2 and mounted the new /SafeKeg. Another dd test showed higher numbers, about 150MB/sec, and with the tw_cli tool installed I can manage the 3Ware card from FreeBSD itself. Very cool. I did play with iSCSI, exporting a 50GB file to my Windows desktop, but the performance was incredibly slow, like 1-2MB/sec. Terrible! I’m not sure if Free/NetBSD’s iscsi-target is at fault, or Window’s iscs initiator, or if iscsi is just not all that up to snuff. I thought it would be nice to utilize, but I’d like to expierement with it more to see exactly how much performance I can squeeze out of it.

UPDATE (10/09/2008):

After getting a few comments on my poor iscsi performance, I have patched to the latest version and I have been MUCH happier with the results. On my little home network I’ve gotten around 40MB/sec for reads and writes.