My Wushu Blog

I need to share this. When I google for “Samba performance”, I never see real numbers, real configuration files, or real hardware environments. All I read are anecdotal recollections, and that is not good enough. I like numbers, and I’ll let the numbers speak for themselves:

    > netstat -I em0 -w 1
                input          (em0)           output
       packets  errs      bytes    packets  errs      bytes colls
         90166     0   98762637      95363     0    5332847     0
         18131     0   24713156      20042     0    1123684     0
             4     0        310          1     0        178     0
             8     0        518          1     0        178     0
         10153     0   10952920      10696     0     598129     0
         92990     0  102837002      98476     0    5514994     0
         92025     0  102680574      97277     0    5439496     0
         92080     0  101799874      97403     0    5448637     0
         75348     0   90861608      80972     0    4537737     0
         90895     0  100323946      95781     0    5360948     0
         89313     0   97371154      94364     0    5278618     0
         81363     0   89229738      85861     0    4803589     0
             2     0        126          3     0        286     0

I was so shocked that I had to use gstat and zpool iostat to verify the information:

    dT: 1.002s  w: 1.000s  filter: da0
     L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
       35   1476      0      0    0.0   1476 188421   23.7  100.0| da0

    > zpool iostat  1
                   capacity     operations    bandwidth
    pool         used  avail   read  write   read  write
    ----------  -----  -----  -----  -----  -----  -----
    tank        5.68T  4.32T      1     81   250K  10.1M
    tank        5.68T  4.32T      0  1.37K      0   175M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M

This is all through Samba (3.3.9), There was no local work being done. I unfortunately didn’t configure MRTG correctly, so it had built a malformed graph while all this happened. Having a picture from all of this would have been nice.

The underlying storage is a SATABoy2 RAID6 array, with a simple “flat” ZFS filesystem (version 13). As cheap as the SATABoy’s are (and come on, they have a terrible IIS web interface), they can at least keep up with the current load.

I have felt that if you are going to use ZFS, you should let it manage the RAID, and not bother with a hardware RAID controller. While the hardware RAID may be faster, ZFS’s ability to self-correct bad blocks is a great feature despite the performance set back. However, RAID6 is pretty good in itself, and having dual parity would ideally reduce the risk of a bad block being detrimental.

One thing I noticed with Samba is it doesn’t seem to be a threaded daemon. When I do a top(1) -H, there are only 2-3 smbd processes, and one of them is running around 30%. Though I don’t really know how well Samba can scale out, this environment only has about 10 users. I would like to see how samba reacts if there are a couple hundred active users. Furthermore, how does a native Windows server handle a couple hundred users? It may handle it a little better, however, I don’t think I would enjoy watching NTFS handling a multi-terabyte volume… it would be like watching a stroke victim eat a bowl of soup. I do admit I am biased and I have no working experience with Windows as a large file server, most of them that I have worked on are horribly limited and underpowered, and no one seems to care if they perform well or not.

Hardware

CPU information

    Machine class:    amd64
    CPU Model:    Dual Core AMD Opteron(tm) Processor 285
    No. of Cores:    4
    Cores per CPU:

RAM information

    Memory information from dmidecode(8)
    Maximum Capacity: 8 GB
    Number Of Devices: 4
    Maximum Capacity: 8 GB
    Number Of Devices: 4

    INFO: Run `dmidecode -t memory` to see further information.

    System memory summary
    Total real memory available:    8048 MB
    Logically used memory:        2876 MB
    Logically available memory:    5172 MB

    Swap information
    Device          1K-blocks     Used    Avail Capacity
    /dev/da1s1b       8373844      28K     8.0G     0%

Storage information

    Available hard drives:
    cd0:  Removable CD-ROM SCSI-0 device
    cd0: 1.000MB/s transfers
    da2:  Fixed Direct Access SCSI-5 device
    da2: 300.000MB/s transfers
    da2: Command Queueing enabled
    da2: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C)
    da1:  Fixed Direct Access SCSI-2 device
    da1: 300.000MB/s transfers
    da1: Command Queueing enabled
    da1: 69618MB (142577664 512 byte sectors: 255H 63S/T 8875C)
    da0:  Fixed Direct Access SCSI-5 device
    da0: 200.000MB/s transfers
    da0: Command Queueing enabled
    da0: 10491861MB (21487333120 512 byte sectors: 255H 63S/T 1337524C)

    Raid controllers:
    umass-sim0:
    mpt0:
    vendor='LSI Logic (Was: Symbios Logic, NCR)'
    device='SAS 3000 series, 4-port with 1064 -StorPort'
    isp0:
    vendor='QLogic Corporation'
    device='QLA6322 Fibre Channel Adapter'

    Currently mounted filesystems:
    /dev/da1s1a on /
    devfs on /dev
    tank on /tank
    /dev/ufs/EXPORT on /export

    I/O statistics:
           tty             da0              da1              da2             cpu
     tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
       0    40 63.61 167 10.36  16.53   2  0.03  61.65   0  0.00   1  0  4  0 94
    INFO: Run iostat(8) or gstat(8) to see live statistics.

    Disk usage:
    Filesystem         Size    Used   Avail Capacity  Mounted on
    /dev/da1s1a         58G    3.4G     50G     6%    /
    devfs              1.0K    1.0K      0B   100%    /dev
    tank               9.8T    5.7T    4.1T    58%    /tank
    /dev/ufs/EXPORT    126G    148K    116G     0%    /export

Software

• FreeBSD 8.0-RELEASE-p1 FreeBSD 8.0-RELEASE-p1 amd64
• samba-3.3.9 A free SMB and CIFS client and server for UNIX

Samba 3.3.9 Compile-Time Config

> make showconfig
===> The following configuration options are available for samba-3.3.9:
     LDAP=on "With LDAP support"
     ADS=on "With Active Directory support"
     CUPS=off "With CUPS printing support"
     WINBIND=on "With WinBIND support"
     SWAT=off "With SWAT WebGUI"
     ACL_SUPPORT=on "With ACL support"
     AIO_SUPPORT=on "With Asyncronous IO support"
     FAM_SUPPORT=on "With File Alteration Monitor"
     SYSLOG=on "With Syslog support"
     QUOTAS=on "With Disk quota support"
     UTMP=off "With UTMP accounting support"
     PAM_SMBPASS=on "With PAM authentication vs passdb backends"
     DNSUPDATE=off "With dynamic DNS update(require ADS)"
     DNSSD=off "With DNS service discovery support"
     EXP_MODULES=on "With experimental modules"
     POPT=on "With system-wide POPT library"
     MAX_DEBUG=off "With maximum debugging"
     SMBTORTURE=off "With smbtorture"
===> Use 'make config' to modify these settings

System Tuning

The Kernel

I enabled device polling, and took out debugging in the kernel (Sanders, get it! Mmm, I’m hungry…)

diff /usr/src/sys/amd64/conf/GENERIC /usr/src/sys/amd64/conf/SANDERS
    33d32
    < makeoptions    DEBUG=-g        # Build kernel with gdb(1) debug symbols
    78c77
    <
    ---
    > options        DEVICE_POLLING

/boot/loader.conf

    ispfw_load="YES"
    kern.hz="2000"
    aio_load="YES"

/etc/sysctl.conf

    kern.coredump=0
    security.bsd.see_other_uids=0
    security.bsd.see_other_gids=0
    kern.ipc.maxsockbuf=16777216
    kern.ipc.nmbclusters=32768
    kern.ipc.somaxconn=32768
    kern.maxfiles=65536
    kern.maxfilesperproc=32768
    kern.maxvnodes=800000
    net.inet.tcp.delayed_ack=0
    net.inet.tcp.inflight.enable=0
    net.inet.tcp.path_mtu_discovery=0
    net.inet.tcp.recvbuf_auto=1
    net.inet.tcp.recvbuf_inc=524288
    net.inet.tcp.recvbuf_max=16777216
    net.inet.tcp.recvspace=65536
    net.inet.tcp.rfc1323=1
    net.inet.tcp.sendbuf_auto=1
    net.inet.tcp.sendbuf_inc=524288
    net.inet.tcp.sendspace=65536
    net.inet.udp.maxdgram=57344
    net.inet.udp.recvspace=65536
    net.local.stream.recvspace=65536
    net.inet.tcp.sendbuf_max=16777216
    net.inet.tcp.mssdflt=1460

rc.conf (em0 flags)

    ifconfig_em0="inet xxx.xxx.xxx.xxx  netmask 255.255.255.0 polling tso mtu 9194"

smb.conf

        min receivefile size = 131072
        aio read size = 1
        aio write size = 1
        use sendfile = yes
        lock directory = /var/run/samba/
        keepalive = 300

I’m also using LDAP users and group. I wasn’t sure if there would be a noticible performance hit for local users or LDAP users. There doesn’t seem to be one.

We use Active Directory, and since Quest/Vintela still won’t make a FreeBSD client for the Quest Authentication Servers ( a sales rep once told me “There are just too many versions of BSD…”) , I have to use all the open source utilities like OpenSSL, OpenLDAP Client and Kerberos. I don’t mind having to do it, but it is always nice if you can maintain one standard process across ALL systems, and we have a lot more Linux and Solaris systems than FreeBSD. I’m the odd one.

That aside, I use the latest OpenSSL in FreeBSD 8.0, OpenLDAP 2.4.20, and the built-in version of Heimdal Kerberos.

I get similar performance form NFS, however, most desktop users have are either on a Windows or OS X, and CIFS seems to be the unifying network storage protocol.

One thing I have yet to really figure out is configuring Samba to use proper NT ACL’s. However, if you can live with UNIX style permissions, a setup like this is pretty good at serving out lots and lots of data. Maybe that will be next.

mike Geekyness FreeBSD, kickassery, Samba, ZFS

I recently had a little problem with a new FreeBSD install, and it is one of those times were I sort of appreciate how FreeBSD assigns device handles, yet at the same time hate it

The setup is this:
The OS was installed on a mirrored hardware raid device (using the mpt(4) driver), and then I had a large RAID6 array attached via a FC controller (using the isp(4) driver). When I installed the OS, the mpt device was showing up as da0. So I went ahead with the install and rebooted the system, so far so good.

What I didn’t realize was the FC device was not seen yet, so after some fiddling, Jenny and I got the large RAID6 array to show up… unfortunately, the isp card was before the mpt card on the PCI bus:

isp0@pci0:2:1:0: class=0x0c0400 card=0x01321077 chip=0x63221077 rev=0x03 hdr=0x00
vendor = 'QLogic Corporation'
device = 'QLA6322 Fibre Channel Adapter'
class = serial bus
subclass = Fibre Channel
mpt0@pci0:2:3:0: class=0x010000 card=0x30601000 chip=0x00501000 rev=0x02 hdr=0x00
vendor = 'LSI Logic (Was: Symbios Logic, NCR)'
device = 'SAS 3000 series, 4-port with 1064 -StorPort'
class = mass storage
subclass = SCSI

and the RAID6 now became da0, and the OS device now became da1.

Doh!

The system prompted for the / drive, so I had to call out the correct device at the mount> prompt:

mount> ufs:/dev/da1s1a

After that, the system continue to boot into mult-user mode, which cause some very strange console behavior (it acted like the return key was being held down), and my only option was to SSH in as local user, su to root, and then fix /etc/fstab.

This was not devastating, however, it show the importance of using disk labels instead of device handles in certain use cases. I haven’t fixed the / mount, but to get a comfort level with using GEOM labels I added another drive to the system and called it EXPORT.

You can assign a permanent label in two ways (that I know of). When you newfs the device, you can specify the L flag (BTW, -O2 means to use UFS2, and -U will use Soft-Updates):
[root@paper ~]> newfs -O2 -U -L EXPORT /dev/da2s1a
OR using glabel (which is what you would have to do for a non UFS filesystem.
[root@paper ~]> glabel create EXPORT da2s1a
Now we can see our newly labeled device in action:
[root@paper ~]> ls /dev/label
. .. EXPORT
[root@paper ~]> glabel status
Name Status Components
label/EXPORT N/A da2s1a

To add it to /etc/fstab, you can either edit the file, or append the correctly tab-delimited line like so:

[root@paper ~]> echo "/dev/label/EXPORT\t/export\tufs\trw\t2\t2" >> /etc/fstab
[root@paper ~]> mkdir /export
[root@paper ~]> mount export

Hurray!

[root@paper ~]> df
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/da1s1a 60931274 4754540 51302234 8% /
devfs 1 1 0 100% /dev
tank 10569645824 107237376 10462408448 1% /tank
/dev/label/EXPORT 132022788 4 121460962 0% /export

[root@paper ~]> mount
/dev/da1s1a on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
tank on /tank (zfs, NFS exported, local)
/dev/label/EXPORT on /export (ufs, local)


This is now a persistent label. To be safe, I’ll have to boot off of a CD/USB drive and modify the root device.

mike Geekyness FreeBSD, GEOM, Jenny, Labels

I’ve always had a vested interest in the entire nvidia display driver for FreeBSD project, and I’ve pretty attached to the project. So much, that back in 2001 I started a little petition, got enough attention (and more importantly, a large list of people who signed my petition), and ever since 2002 FreeBSD users have been able to use high quality nvidia drivers. It wasn’t all me, whoever ran nvidia.netexplorer.org asked me to combine efforts, and I gave them my list, and they continued to market it and work with some folks at nvidia.

It is really nice to see that both the FreeBSD team and nvidia have worked together to do the necessary kernel development and get a 64bit driver. I used to use FreeBSD as my primary desktop at work, and it was great to use the hardware drivers for my displays. What is also nice is people in the nvidia forums are also asking for CUDA drivers on FreeBSD, that would be slick as well.

Digg the story if you want too:
http://digg.com/linux_unix/Official_64bit_NVIDIA_drivers_for_FreeBSD

mike Geekyness FreeBSD, nvidia

So, it looks like FreeBSD 8.0 has been pre-released; the official date is going to be 11/25, as noted in src/UPDATING:

Updating Information for FreeBSD current users

This file is maintained and copyrighted by M. Warner Losh
.  See end of file for further details.  For commonly
done items, please see the COMMON ITEMS: section later in the file.

Items affecting the ports and packages system can be found in
/usr/ports/UPDATING.  Please read that file before running
portupgrade.

NOTE TO PEOPLE WHO THINK THAT FreeBSD 8.x IS SLOW ON IA64 OR SUN4V:
        For ia64 the INVARIANTS and INVARIANT_SUPPORT kernel options
        were left in the GENERIC kernel because the kernel does not
        work properly without them.  For sun4v all of the normal kernel
        debugging tools present in HEAD were left in place because
        sun4v support still needs work to become production ready.

20091125:
        8.0-RELEASE.
...

Thanks for the warning, and I don’t feel that 8.0 is slow in any way

You can now update to FreeBSD 8.0 with either syncing your source with csup:

*default host=cvsup.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default delete use-rel-suffix
*default compress
src-all release=cvs tag=RELENG_8_0

Or with freebsd-update(8):

# freebsd-update -r 8.0-RELEASE upgrade

then

# freebsd-update install

and after the reboot, possibly another round of ‘freebsd-update install” to finish things up. You can actually upgrade from 7.2 to 8.0, which is pretty impressive since they are considered major releases (and minor release upgrades work just fine as well).

Why would you upgrade to 8.0 over 7.2? Well, Ivan Voras already has a very nice page on the notable features in 8:
http://ivoras.sharanet.org/freebsd/freebsd8.html
In case you want my short list version of that, here are the big highlights for me:

• Kernel Stuff
  • Kernel limit on amd64 increased (this greatly benefits ZFS)
  • Superpages
  • Network stack virtualization, equal cost multipath routing and other really cool network improvements
  • NGROUPS has been increased from 16 to 1024
  • Other kernel improvements like light weight threads, the new ULE 3.0 Scheduler
  • NFS Locking
  • Qlogic 8GB HBA support
  • New AHCI driver
• Userland Stuff
  • Parallel port builds
  • Jails v2
  • Dtrace
  • CLANG/LLVM Compiler

One of the cool things about FreeBSD is its focus on improving what is there. There have been some really big additions to FreeBSD from time to time, but overall, the goal has been to constantly refine and improve the performance. That is what I’m mostly excited about, the continual refinement of an already robust OS.

There are other features, like CLANG and LLVM or Dtrace, where I’m excited about them, but only because I can’t wait to see how others use them. I myself cannot obtain a lot of useful information from Dtrace, however, a kernel developer who knows what they are doing probably can, and that helps them out (which sometimes helps me out).

I’ve used the BETA and RC versions of 8.0, so not only was I pleased with the experience, I’m also excited to see its adoption with the new improvements. I’ve seen some PostgreSQL and MySQL benchmarks and there was a clear performance gain between 7 and 8.

Now is also a good time to mention that the FreeBSD Foundation is rounding up this years donations.

It’s pretty amazing that FreeBSD is a non-profit group; they do not have a CEO, a marketing department, or a horde of full-time developers… and yet they put out a extremely well engineered OS ( that is the boon of not having a marketing department all decisions are driven by the community demand and the developers, and not buzz-words like “the cloud”) with a killer network stack, and over 22,000 available ports.

mike Geekyness FreeBSD

This was a very cool conference. I picked up a lot of useful information on both the open source tool, Puppet, and some ideas on infrastructure.

What also made this conference unique, is how honest the Puppet team and community were about the projects strength and weaknesses. Those that have deployed Puppet on a larger scale (MessageOne and Google) seemed to go through the same iterations in attempting to scale out their Puppetmaster’s. From WEBrick (which is what I’m currently running Puppet with ), which is hated by all since its a single process/thread web server that can only handle one request at a time. To Mongrel, which you have to manage a mongrel cluster script, feed it lots of memory, and then throw an apache proxy server in front of them. Now, people are starting to settle on using Passenger/mod_rack, which is what I spent most of yesterday looking into and setting up. This allows apache to mount a rails instance, and then you don’t actually have to run puppetmasterd. This still requires some decent hardware, and I’m currently running my puppetmaster on a VM with 2GB or memory, so I’ll have to watch out for that. Chris, the one who introduced me to Puppet, said he still uses WEBrick for all of his DB, Tomcat, and Apache servers (I think he said something like 200 systems) and it has been working out nicely. He, like the guys at Google, also doesn’t run puppet as a daemon.

Anyway, the point is, we learned a lot about the project, way more than if a sales person had come to us and just told us the things puppet does well, or how it operates on paper (*cough* LANDesk *cough*). It was really awesome to talk with Andrew Pollock and Nigel Kersten from Google. See, I was a little unsure about Puppet in our environment, where we have multi-purpose servers, computer servers, and desktops that we have to manage. It seemed, at a first glance, that most of the Puppet users out there have a homogeneous environment, and Andrew (Shafer) had stressed the concept of single role servers. After talking with them, I felt a lot more comfortable pursuing Puppet across our servers and desktops. Did I mention they were super cool and friendly?

We also learned a lot about the Puppet developers, which had its own interesting advantage. I have a lot of respect for what Luke Kanies has been able to do, and by the end of the conference, he showed significant mastery in what he has done, as well as some humility and admitting what he has not been able to do and why. I was a little put off the first day though, when both him and Andrew came off a little arrogant and crass. It did make me step back and think, “Is this project going to be well managed in the future with personalities like this in charge? Is their answer of ‘don’t do that!’ tongue in cheek, or are they not supportive of a diverse environment?”. In the end, I have more respect for the project than ever, and with it still being a young project, I hope they listened to some of the feedback, and I also can’t wait to see where it ends up in the next year.

Andrew, the Puppet Andrew, came up to us a lot during the conference, and he was fun to talk too, and he’s very academic and he had a lot of abstract concepts to talk about. Also, he said this was the first conference he has arranged, and I think he did a fantastic job. Jenny had commented that this was the first conference she had lasted the entire duration, so that says a lot about the pacing and content of PuppetCamp. I felt the same way, every session was incredibly engaging, and how Andrew had setup the democratic and chaotic Open Sessions was very impressive. Lets put it this way, I even got up there and pitched a topic, which is something I would have never done. Hurray for me stepping outside of my comfort zone!

Warning: side topic!

Now that I’ve had the weekend to google all the cool technologies I was exposed too, I’m also reminded why I really like having a FreeBSD server at my disposal. They had talked about CouchDB, so on a whim I did a

~> cd /usr/ports
/usr/ports> make search name=couchdb
Port: couchdb-0.9.0_1,1
Path: /usr/ports/databases/couchdb
Info: A document database server, accessible via a RESTful JSON API
Maint: till@php.net
B-deps: ca_root_nss-3.11.9_2 curl-7.19.6_1 erlang-lite-r13b01_6,1 gettext-0.17_1 gmake-3.81_3 icu-3.8.1_2 libiconv-1.13.1 libtool-2.2.6a nspr-4.8 perl-5.8.9_3 spidermonkey-1.7.0
R-deps: ca_root_nss-3.11.9_2 curl-7.19.6_1 erlang-lite-r13b01_6,1 gettext-0.17_1 gmake-3.81_3 icu-3.8.1_2 libiconv-1.13.1 libtool-2.2.6a nspr-4.8 perl-5.8.9_3 spidermonkey-1.7.0
WWW: http://couchdb.org/

Port: py26-simplecouchdb-0.9.26
Path: /usr/ports/databases/py-simplecouchdb
Info: Simple Librairy to Allow Python Applicationto Use CouchDB
Maint: wenheping@gmail.com
B-deps: py26-httplib2-0.5.0 py26-py-restclient-1.3.2 py26-setuptools-0.6c9 python26-2.6.2_3
R-deps: py26-httplib2-0.5.0 py26-py-restclient-1.3.2 py26-setuptools-0.6c9 python26-2.6.2_3
WWW: http://code.google.com/p/py-simplecouchdb/

I did a ‘make install’, and I had a cool little couchdb up and running. What is also cool is FreeBSD likes to give you very helpful information when you install something. For example, this is what is printed out when you install the CouchDB port:

===> COMPATIBILITY NOTE:
CouchDB is still pre-stable; between 0.8 and 0.9 the database format
changed which breaks BC. In current trunk, the format changed again, so
please double-check in case you are updating an existing installation.

More info:
* http://wiki.apache.org/couchdb/Breaking_changes?action=show&redirect=BreakingChanges
* http://wiki.apache.org/couchdb/BreakingChangesUpdateTrunkTo0Dot9

See, isn’t that helpful? Best of all, I didn’t have to enable additional repositories, or fetch the src manually, and its dependencies and then figure out how to run the right configure script flags… FreeBSD makes it easy, and since it automatically uses what you already have with what is required, its an incredibly stable build. Removing it is pretty simple as well, just:

> pkg_deinstall -R couchdb
---> Deinstalling 'couchdb-0.9.0_1,1'
---> Deinstalling 'erlang-lite-r13b02,1'
[Updating the pkgdb
in /var/db/pkg ... - 118 packages found (-1 +0) (...) done]
---> Deinstalling 'curl-7.19.6_1'
[Updating the pkgdb
in /var/db/pkg ... - 117 packages found (-1 +0) (...) done]
---> Deinstalling 'ca_root_nss-3.11.9_2'
---> Deinstalling 'spidermonkey-1.7.0'
---> Deinstalling 'nspr-4.8'
[Updating the pkgdb
in /var/db/pkg ... - 116 packages found (-1 +0) (...) done]
---> Deinstalling 'gmake-3.81_3'
[Updating the pkgdb
in /var/db/pkg ... - 115 packages found (-1 +0) (...) done]
---> Deinstalling 'perl-threaded-5.8.9_3'
[Updating the pkgdb
in /var/db/pkg ... - 114 packages found (-1 +0) (...) done]
---> Deinstalling 'gettext-0.17_1'
---> Deinstalling 'libiconv-1.13.1'
---> Deinstalling 'icu-3.8.1_2'
---> Deinstalling 'libtool-2.2.6a'
** Listing the failed packages (-:ignored / *:skipped / !:failed)
! curl-7.19.6_1 (pkg_delete failed)
! ca_root_nss-3.11.9_2 (pkg_delete failed)
! perl-threaded-5.8.9_3 (pkg_delete failed)
! gettext-0.17_1 (pkg_delete failed)
! libiconv-1.13.1 (pkg_delete failed)

This does a upwards recursive dependency removal. Also, if one dependency is relied on by another, it wont get removed. Like, if Perl58 was a dependency of a package, it wouldn’t be removed if perl58 is used by many other packages. This is smart. So, above, the packages that failed to deinstall where ones that are required dependencies of other installed packages.

Speaking of package management; have you ever installed something that ended up having a few dozen dependencies, then you want to uninstall that package with a “rpm -e cba8″, or something equivalent, but what about all the other cruft that came along with it? You would have to keep track of each dependency, and specify all of them and hope you don’t break another program. FreeBSD has a few tools to do this, one in particular, portmaster can remove all ports that were once a dependency but no longer used:

> portmaster -s
Information for neon28-0.28.4:
Comment:
An HTTP and WebDAV client library for Unix systems
===>>> neon28-0.28.4 is no longer depended on, delete? [n] y
===>>> Delete old and new distfiles for www/neon28
without prompting? [n] y
===>>> Running pkg_delete -f neon28-0.28.4
Information for rubygem-actionwebservice-1.2.6:
...

I ended up removing 4 packages that were no longer used.

CentOS and RHEL are the larger Puppet consumers, I’m still a big proponent for FreeBSD, and at work, it has allowed me to quickly build an Apache + Puppet + RubyPassenger/mod_rack stack with the minimal dependencies installed. So, the puppet server is still pretty lean, which means updates are smaller and faster. It still surprises me that its relatively unknown, even though Netcraft always has it listed in the top domains with the best uptime and consistently growing over the years. Why do I feel like an AmigaOS fan sometimes?

Hmm, it is sort of weird that this turned into a FreeBSD ports management entry

Okay, final word: PuppetCamp09 was Freaking awesome. There were a lot of smart developers and sysadmins there. We even got a very cool git howto, which I found useful. It was very diverse, which is strange for a conference based on one project in particular.

mike Geekyness FreeBSD, Jenny, kickassery, port management, Puppet, Ruby

PC-BSD KDE Desktop

PC-BSD is a nice mesh between FreeBSD and a ready to use Desktop (which uses about 6GB of disk space). It is based on FreeBSD 7.2, so it has all the cool features of the latest release. Best of all, without ANY additional configuration, I was able to:

• Use the official FreeBSD nVidia driver for hardware acceleration
• Watch clips on YouTube (with flashplayer)
• Play back all sorts of media types like mp3’s, divx, mpegs,wmv,qt…
• use ZFS
• Create and edit documents with the latest OpenOffice 3.1
• Browse the web with Firefox 3.5
• Create VM’s with VirtualBox

Plus, if there wasn’t a PBI package for what I wanted, I could still use FreeBSD’s pkg_add or, cd to /usr/ports and make one. I would say, that’s pretty impressive for a commercially supported Unix platform.

Otherwise, it is very much like Fedora or Ubuntu, where it has an update manager (updates PBI’s and the system), network manager, helpful tutorials, and for once (for FreeBSD at least) a full blown X11/QT graphical installer. FreeBSD has always had a simple ncurses installer, which I like, but it tends to frighten a lot of people who are used to GUI installers.

Once strange thing it does is place all PC-BSD binaries in /usr/PCBSD. I guess this is to remain independent and out of the way of the base FreeBSD binaries, as well as /usr/local, which is the normal prefix for all Ports.

To wrap it up; my initial impression of PC-BSD is a positive one. I like how I could use the FreeBSD ports and package system and it did not conflict with the PC-BSD packages that were installed. I like the installer, and the storage options at install time (UFS2+SU, or UFS2+Journal, Encrypted swap…). With all OS’s, it normally takes a few weeks of using it to see its weaknesses, so I’m sure PC-BSD has some issues waiting to pop up. The only one I see right now is that KDE4 is the default GUI, and I prefer Gnome. I could install it, but it wasn’t an up front, as all of PC-BSD’s install tools are written in QT. The initial X setup tool was pretty slick, and it worked with my picky laptop.

mike Geekyness Flash, FreeBSD, kickassery, OS Review, PC-BSD

FreeBSD 7.x has been using version 6 of ZFS, and originally only 8.0 was going to have the newly updated ZFS version: 13.

Last week the core team MFC’d (Merge From Current) the ZFS updates to 7.2, so I cvsup’d and re-build my server’s kernel and world ( with a simple “make buildworld && make buildkernel && make installworld && make installkernel ), rebooted, and now I have the latest ZFS version running:

[root@server ~]> zpool upgrade -v
This system is currently running ZFS pool version 13.

The following versions are supported:

VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS version
 2   Ditto blocks (replicated metadata)
 3   Hot spares and double parity RAID-Z
 4   zpool history
 5   Compression using the gzip algorithm
 6   bootfs pool property
 7   Separate intent log devices
 8   Delegated administration
 9   refquota and refreservation properties
 10  Cache devices
 11  Improved scrub performance
 12  Snapshot properties
 13  snapused property
For more information on a particular version, including supported releases, see:

http://www.opensolaris.org/os/community/zfs/version/N

Where 'N' is the version number.

mike Geekyness FreeBSD, ZFS

I don’t have a backup system for home (which is where this site, and others are located), and I have generally relied on duplicating enough of my important stuff between friends and other computers. That, and I have a RAID5 setup for my large storage, and then home directories and website stuff is on a RAID1 ZFS volume. This doesn’t prevent accidental “oh-no”s, but it does protect me from some hardware failures.

Last year when I upgraded to the new server, I lost a lot of data because I forgot to backup all of my MySQL databases. I like to think I can learn from my mistakes, so a full year later I finally did something about it and signed up for Amazon’s S3 service.

The pricing is pretty nice, and I don’t have all that much data to backup. I figure, I’ll use up a few GB in total, and keep the monthly price around $1 – $2. That seems worth the price for off-site backup’s.

Now, I have 3 main websites that I need to backup, and one test site that I like to play around with:

• http://www.m87-blackhole.org/ <- This is the first domain that I owned, and its the site where my family checks out new photos
• http://www.willowoakboarding.com/ <- My parent’s site for their boarding ranch. I’m glad they have no concept of a SLA, or that things need to be backed up
• http://www.mywushublog.com/ <- This site of course, where I claim my own identity on the internet
• http://www.evil-genius-network.com/ <- a test domain, but now I run a little OpenID service, For one…

After a quick “FreeBSD s3 backup” Google search, I found Gary Dalton’s blog post: http://dvector.com/oracle/2008/10/18/backing-up-to-amazon-s3/. After reading this post, I formulated my plan of attack:

• Sign up for S3, create a “bucket” for each site
• Use something to interface with S3 ( duplicity )
• Automate MySQL and PostgreSQL backups
• Create a service account to run both s3 and db backup scripts as
• Set up a cron job for backups

So, after I signed up for S3, I had to create the buckets. I couldn’t find a way to do this though my Amazon account settings, so I created a little ruby script.

$ sudo gem install aws-s3
$ vim make-bucket.rb


#!/usr/local/bin/ruby
 
require 'aws/s3'
 
AWS::S3::Base.establish_connection!(
:access_key_id     => 'my-s3-key-id',
:secret_access_key => 'my-s3-secret-access-key'
)
AWS::S3::Bucket.create('mywushublog')
AWS::S3::Bucket.create('willowoak')
AWS::S3::Bucket.create('m87-blackhole')
AWS::S3::Bucket.create('evil-genius-network')

$ ./make-bucket.rb
Next, I had to install duplicity and py-boto
[root@server ~] cd /usr/ports/sysutils/duplicity
[root@server duplicity] make install
...
[root@server duplicity] cd ../../devel/py-boto
[root@server py-boto] make install clean
...
[root@server py-boto]
Next step, create a user (with access to shared data, and website data) to run the backups with the adduser command…
[root@server py-boto] adduser -g shared-data -G www -s /bin/tcsh -w random s3backupuser
...
[roott@server py-boto] su - s3backupuser
In tcsh, you can `set autolist' to have the shell automatically show
all the possible matches when doing filename/directory expansion.
%
I’ll have to set my Access ID and Access Key in the s3backupuser’s environment, as well as a GnuPG passphrase so the backups are encrypted (and compressed). I mean, I trust Amazon, but not THAT much
% vim .cshrc
setenv AWS_ACCESS_KEY_ID my-s3-key-id
setenv AWS_SECRET_ACCESS_KEY my-s3-secrect-access-key
setenv PASSPRASE AVeryRandonPassphraseForGnuPG
Next, I copied the very useful automysqlbackup.sh script into a separate script for each website. I could have just dumped every database that was running, but I wanted to segregate each site’s databases into a different directory. So, I’m complicating my cron job by running multiple backup scripts, but I really want to make the end result easily readable and identifiable by me. So for each site, I create a directoy under /u01/backups:
%ll /u01/backups/
total 8
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:46 evil-genius-network
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:47 m87-blackhole
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:46 mywushublog
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:47 willowoak
Next was the s3-backups.sh script, which is very crude and simple. If I’m really motivated, I’ll make it nicer but I’m lazy and if I don’t need anymore functionality then I’ll just leave it. One thing I initially forgot was that I set my Amazon S3 variables in the users .cshrc profile. This is not a good place to have those things, it was just handy as I was running the duplicity commands manually. So I had to add those in, otherwise the cron job would fail.

~/bin/s3-backups.sh:

#!/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/s3/bin
 
# Amazon S3 keys, and GnuPG keys
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
PASSPHRASE=
export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY
export PASSPHRASE
 
echo "*************************************************"
echo "*   Backing up Website content....              *"
echo "*                                               *"
echo "*     www.willowoakboarding.com...              *"
duplicity /www/www.willowoakboarding.com s3+http://s3.amazon.com/willowoak/www
echo "*     www.mywushublog.com...                    *"
duplicity /www/www.mywushublog.com s3+http://s3.amazon.com/mywushublog/www
echo "*     www.m87-blackhole.org...                  *"
duplicity /www/www.m87-blackhole.org s3+http://s3.amazon.com/m87-blackhole/www
echo "*************************************************"
echo "*   Backing up databases....                    *"
echo "*                                               *"
echo "*     www.willowoakboard.com...                 *"
duplicity /u01/backups/willowoak s3+http://s3.amazon.com/willowoak/db
echo "*     www.mywushublog.com...                    *"
duplicity /u01/backups/mywushublog s3+http://s3.amazon.com/mywushublog/db
echo "*     www.m87-blackhole.org...                  *"
duplicity /u01/backups/m87-blackhole s3+http://s3.amazon.com/m87-blackhole/db
echo "*************************************************"

And last but not least, a cronjob to tie it all together:
% crontab -e
@weekly ~/bin/s3-backups.sh
@weekly ~/bin/mywushublog-mysql-backup.sh
@weekly ~/bin/willowoak-mysql-backup.sh
@weekly ~/bin/m87-blackhole-mysql-backup.sh
@weekly ~/bin/evil-genius-network-mysql-backup.sh
I can check the status of a backup by running duplicity with the ‘collection-status‘ flag:
%duplicity collection-status s3+http://s3.amazon.com/mywushublog/db
Last full backup date: Sat Apr 25 15:08:02 2009
Collection Status
-----------------
Connecting with backend: BotoBackend
Archive dir: None
Found 0 backup chains without signatures.
Found a complete backup chain with matching signature chain:
-------------------------
Chain start time: Sat Apr 25 15:08:02 2009
Chain end time: Sat Apr 25 15:08:02 2009
Number of contained backup sets: 1
Total number of contained volumes: 1
Type of backup set:                            Time:      Num volumes:
Full         Sat Apr 25 15:08:02 2009                 1
-------------------------
No orphaned or incomplete backup sets found.
I can also list the files:
%duplicity list-current-files s3+http://s3.amazon.com/mywushublog/db
Last full backup date: Sat Apr 25 15:08:02 2009
Sat Apr 25 15:05:11 2009 .
Sat Apr 25 15:05:10 2009 daily
Sat Apr 25 15:05:10 2009 daily/mywushublog
Sat Apr 25 15:05:10 2009 monthly
Sat Apr 25 15:05:10 2009 weekly
Sat Apr 25 15:05:11 2009 weekly/mywushublog
Sat Apr 25 15:05:11 2009 weekly/mywushublog/mywushublog_week.17.2009-04-25_15h05m.sql.gz
Pretty sweet automated backup process. It is a lot cheaper than tapes or additional disk storage. With S3, I also don’t have to worry about buying additional hardware, the maintenance of a library or tape drive (which is what I had a few years ago, what a headache).

mike Geekyness Backups, FreeBSD, MySQL, PostgreSQL, S3

!!! UPDATE on 12/29/2009!!!
Since this blog post seems to get a good amount or hits from google, if you are reading this, please see my updated post: http://www.mywushublog.com/2009/12/freebsd-8-0-a-great-nas-server/ which has some additional information about FreeBSD 8.0
EOF

Lately at work, I’ve been involved with a very large file system that is being export from Solaris 10/ZFS to windows and OS X users via Samba. Even with a very large Sun server (T5220) a lot of users are complaining about the slow performance of the system. I’m not going to go into details, but what it has prompted me to do is to look into what I use at home (FreeBSD + Samba) for my network storage needs, and see if I can improve the performance of it.

Well, I was checking out the very useful Whats cooking for FreeBSD 8, unrelated to my Samba needs, when I noticed this post on Ivan Voras’ blog. Ivan has some really cool information there, and with this new knowledge I began my update from running Samba 3.0.28a,2 to 3.3.3,0 (the comma represents the Ports version).

Hvala Ivan!

I’ve discussed my new FreeBSD environment before here, with that, here are some quick details:

• OS: FreeBSD 7.1
• Intel Core 2 Duo E6750 (2.66Mhz 4MB cache)
• Intel S975XBX2 workstation motherboard
• AMCC 3Ware 9650SE 4 port SATA RAID controller (4x PCI-e)
  • BatterY backup for the 3Ware so I can enable cached writes
• 2GB ECC Crucial Memory Kit
• ASUS EN6200 LE 16x PCI-e nVidia GFX card
• 4 Western Digital 1TB Drives

The 4 1TB drives create a nice 2.6TB (RAID5) array which I used Samba to share out to my 4 other systems in the house (which is a mix of Windows XP and Vista, sorry, no OS X). I do a lot with this array, any work that Michele and I do like photo editing, word documents, media files, etc.. all gets saved to this volume. Needless to say, this volume is accessed A LOT, and if the house ever caught on fire, I’d save the server before the family (Hey, I could still look at their pictures and videos…).

The Old Configuration

samba 3.0.28a:

[root@server samba3]> make showconfig
===> The following configuration options are available for samba-3.0.34,1:
     LDAP=on "With LDAP support"
     ADS=on "With Active Directory support"
     CUPS=on "With CUPS printing support"
     WINBIND=on "With WinBIND support"
     ACL_SUPPORT=on "With ACL support
     AIO_SUPPORT=on "With Asyncronous IO support"
     FAM_SUPPORT=on "With File Alteration Monitor"
     SYSLOG=on "With Syslog support"
     QUOTAS=off "With Disk quota support"
     UTMP=on "With UTMP accounting support"
     PAM_SMBPASS=off "With PAM authentication vs passdb backends"
     CLUSTER=off "With experimental cluster support"
     DNSUPDATE=off "With dynamic DNS update(require ADS)"
     EXP_MODULES=off "With experimental modules"
     POPT=on "With system-wide POPT library"
     PCH=on "With precompiled headers optimization"
     MAX_DEBUG=off "With maximum debugging"
     SMBTORTURE=off "With smbtorture"
===> Use 'make config' to modify these settings

The New Configuration

samba 3.3 config:

[root@server samba33]> make showconfig
===> The following configuration options are available for samba-3.3.3:
     LDAP=on "With LDAP support"
     ADS=off "With Active Directory support"
     CUPS=on "With CUPS printing support"
     WINBIND=on "With WinBIND support"
     SWAT=off "With SWAT WebGUI"
     ACL_SUPPORT=on "With ACL support"
     AIO_SUPPORT=on "With Asyncronous IO support"
     FAM_SUPPORT=off "With File Alteration Monitor"
     SYSLOG=off "With Syslog support"
     QUOTAS=off "With Disk quota support"
     UTMP=off "With UTMP accounting support"
     PAM_SMBPASS=on "With PAM authentication vs passdb backends"
     DNSUPDATE=off "With dynamic DNS update(require ADS)"
     DNSSD=off "With DNS service discovery support"
     EXP_MODULES=off "With experimental modules"
     SHARED_LIBS=off "With shared libraries"
     POPT=on "With system-wide POPT library"
     MAX_DEBUG=off "With maximum debugging"
     SMBTORTURE=off "With smbtorture"
===> Use 'make config' to modify these settings

If you have never dealt with FreeBSD’s Ports system, setting these compile time options is a breeze with ‘make config’. These options are presented in a nurses interface (which is great for ssh or other terminal based sessions) like this:

FreeBSD's make config screen

The noticable options that I chose are AIO=yes and ADS=off. I’ve always compiled ADS support thinking that I would also get around to configuring Samba as an Active Directory-like server. But you know, there are really only two active users here, myself and Michele, and I don’t see the benefit right now. It could also slow down samba with the extra system calls so again, I’m leaving it out. AIO, Asyncronous IO, is new, and it is reported to increase size (oh wait, wrong advertisement) I mean, network IO.

smb.conf options

The performance related options in smb.conf are here:

        socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY IPTOS_LOWDELAY
        min receivefile size=16384
        aio read size = 16384
        aio write size = 16384
        aio write behind = true

Again, I got these from Ivan’s post, plus what I’ve used for the past 8 years.

Benchmarking with IOzone

I ran a simple

iozone -Ra -b samba3.0.28.xls

for both versions of samba. I’d create some nice 3d charts but I really don’t know excel well enough. So, I’ll just link these here and you can see for yourself what the substantial differences are. Some quick glances while the tests ran, I saw around 50MB/sec for Samba 3.0.28 here and there, topping out at ~60MB/sec. This was only with 256MB and above files. Smaller files always stayed around 5-10MB/sec.

Samba 3.3.3 – with AIO enabled started up fine, but iozone crashed after wrting a few small 64K bytes. This was a little disappointing, however, I did continue the benchmark with the new send and recieve sizes.

UPDATE:
After doing a little more reading, I found out I was supposed to load the aio kernel module. After running:

$ kldload aio

I restarted Samba with the AIO options enabled, re-ran iozone, and it all worked.

ufs2-3ware-raid5-freebsd71

samba3028

samba333

mike Geekyness Benchmarking, FreeBSD, Samba

I’ve configured this blog to use my OpenID accounts. I have two (which totally goes against the single identity mindset of OpenID )

• http://mcarlson.myopenid.com/
• http://openid.evil-genius-network.com/mike

The second one I just stood up today. I’m always concerned with who has my information, and if I can, I try to keep it all within the realm of my control. Also, the evil genius domain has absolutely no purpose besides a testing ground that I have no problems destroying

Using my own OpenID service is attractive, most of all its a fun exercise. Lets go through what I did (so one day I can remember).

The easiest part was finding an OpenID server. A quick google search brought me here:

http://wiki.openid.net/Run_your_own_identity_server

The hard part was deciding which one I should use. I actually tried out 4 of them, phpMyID, Masquerade, DjangoID, and finally, Java OpenID Server. I got three of them running, and in the end I simply settled on JOS. For now. I had a lot of fun building a MCV app in both Ruby on Rails and Django. I’ve been on a MCV kick, as a month ago I got pretty excited about Ruby on Rails. The big part where I shy away from Django or RoR is integrating things into Apache. With Java, I have Tomcat, and I’ve used it before so I have an immediate comfort level with it. I did have to ask Chris for a little bit of help when it came to the mod_jk stuff.

First thing was to go over the JOS documentation. I knew I would need the following:

• Java App Server – I decided to use Apache’s Tomcat 6.0
• A database – PostgreSQL 8.3
• JCalendar, this was simple, as the readme pointed me to one

Over the years, I’ve always used MySQL. It’s simple, light, and all the new fancy “Web 2.0″ site use it. I’m considering making the switch to PostgreSQL for two reasons. 1) Sun seems to be mishandling the QA and release engineering of MySQL . 2) Recent benchmarks with FreeBSD 7.1 and PostgreSQL have been phenomenally good, and even though I’m not running a big site with millions of visitors I do like to keep up with whats current and performs well.

Second, I built the required applications and enabled both tomcat and PostgreSQL in /etc/rc.conf. Think of rc.conf as a simple text-based chkconfig, except with rc.conf, you can specify additional command arguments, profile environment, and anything else the application might support. I like the ease of the chkconfig/service system works in Linux, but FreeBSD’s run command (rc) system is very flexible and easier to tune.

> sudo su -
$ cd /usr/ports/databases/postgresql83-server
$ make install
$ echo 'postgresql_enable="YES"' >> /etc/rc.conf
$ cd /usr/ports/www/tomcat6
$ make install
$ echo 'tomcat60_enable="YES"' >> /etc/rc.conf
$ cd /usr/ports/databases/postgresql-jdbc
$ make install

I could have simply added pre-built packages with “pkg_add -r tomcat6 postgresql83-server postgresql-jdbc” but I like seeing what compile time options are available, and then setting those. Hurray for the flexibility of FreeBSD!

One thing that you have to do with PostgreSQL (that you don’t have to do with MySQL) is initialize the database/config:

$ initdb /usr/local/pgsql/data
$ su - pgsql
> createdb jos-openid
> makepasswd --chars=13
 a nice 13 character random string
> createuser josuser -P
> psql jos-openid 

Welcome to psql 8.3.6, the PostgreSQL interactive terminal. Type:
        \copyright for distribution terms
        \h for help with SQL commands
        \? for help with psql commands
        \g or terminate with semicolon to execute query
        \q to quit

jos-openid=# select * from pg_user;
  usename | usesysid | usecreatedb | usesuper | usecatupd |  passwd  | valuntil | useconfig
 ---------+----------+-------------+----------+-----------+----------+----------+-----------
  pgsql   |       10 | t           | t        | t         | ******** |          |
  josuser |    16386 | t           | f        | f         | ******** |          |
(2 rows)
jos-openid=#

Next, I had to unpack the war file and modify the jdbc.properties to use PostgreSQL

jar -xvf jos-webapp-1.2.0.war .
...
jar -cvf /usr/local/tomcat6/webapps/ROOT.war .

Yeah, after configuring the app and zipping it back up, I called it ROOT, it was a lot easier this way. I didn’t want to manage multiple java apps at this point. I can be a very lazy admin

After starting both Tomcat and PostgreSQL up, I now had a working web app running on my server at port 8180. The last part is to mount the java application inside of apache. For that, I needed to install mod_jk:

$ cd /usr/ports/www/mod_jk
$ make install

Thats the easy part of installing mod_jk, the next parts are the worker.properties file, modifying httpd.conf, and then modifying my virtualhost configuration for the domain evil-genius-network.com. I also added a record for openid.evil-genius-network.com. So, in that order, this is what I did:

/usr/local/etc/apache2/worker.properties:

workers.tomcat_home=/usr/local/apache-tomcat6.0
workers.java_home=/usr/local/jdk1.6.0
ps=/
worker.list=localhost
worker.tomcat.type=lb
#worker.tomcat.balanced_workers=localhost
#worker.loadbalancer.local_worker_only=0
worker.localhost.port=8009
worker.localhost.host=localhost
worker.localhost.type=ajp13
worker.localhost.lbfactor=1

/usr/local/etc/apache2/httpd.conf:

LoadModule jk_module libexec/apache22/mod_jk.so
# mod_jk
JkWorkersFile /usr/local/etc/apache22/workers.properties
JkLogFile  /var/log/jk.log
JkShmFile  /var/log/jk-runtime-status
JkLogLevel error

/usr/local/etc/apache2/virtualhosts/evil-genius-network.com (in the openid.evil-genius-network.com VirtualHost section):

JkMount /* localhost

Then, I restarted apache:

$ /usr/local/etc/rc.d/apache2 restart

Now, have my own little OpenID server running at http://openid.evil-genius-network.com/

BTW, I had to re-edit EVERY pre section of this page about 6 times, that was the least-fun part of all of this.

mike Geekyness Apache, FreeBSD, Java, OpenID, PostgreSQL, Tomcat 6