My Wushu Blog

I’ve had a good run with my Zune MP3 player, excellent in fact, and I would highly recommend that player to anyone who was looking for an iTunes alternative. Truthfully, I’d recommend one no matter what, but it is impossible to beat the ubiquitous nature of iTunes and the iPod, like it or not. I’ve had mine for over 2 years, and I only one had a problem where it locked up. For a Microsoft device, it is the most un-Microsoft like product out there. It was incredibly clean, stable (2+ years of constant running and only one lock up, most servers struggle for that kind of uptime), and the Zune software itself was probably the best media manager I’ve ever used. I can’t say enough pleasant things about the product, it really is well made. Here is a shot of the main interface:

Here is a shot of my profile:

BTW, I’ve received a “Silver” Badge for Pink Floyd, Buckethead and Smog. That means I’ve listened to their music at least 1000 times…

So, Sunday morning, when I asked Michele if she had seen my MP3 player, I almost leaked out a few tears when she opened up the Washing machine lid after the load was done and fished out my poor little 8GB Zune Player. There it was, a sad little thing, I felt like I found the family cat under the house or something. It was cold, a little damp, and completely devoid of life. There was a glimmer of hope, because I still kept the Extended Warranty papers… and it was for two years, which expired on 05/26/2010. The timing of this event was pretty comical.

Faced with a crisis of having to listen to the dreaded morning radio, I came up with two options. One, was to use what I had left in Best Buy gift cards (about $50), and get a new Zune HD (~$179). The Second option was to join my geek friends and get a smart phone.

I talked with my friend Steve for a bit, primarily to ask how well an Android based phone would perform as a MP3 player and Podcast manager. He said that syncing music is probably more laborious compared to what I was used to, but he said managing podcast’s with Google Reader is where it really shines. He mentioned that he will play podcasts at work in his desktop, and then continue it on his phone as if it were the same device.

In the end, I went with the phone, the Droid. I’m a bit sad about giving up my Zune, and I even softly whistled “Taps” as I put it back in its original box. However, I would like to take a little more advantage of what a “Smart Phone” can offer, and consolidate two devices into one. I guess it can make phone calls too, but I doubt I’ll do that very often.

I didn’t find any blog posts of discussions on FreeBSD and multipath (for storage) that wasn’t a man page.

That means it is up to me to write about it

Hardware

CPU

Machine class:	amd64
CPU Model:	Intel(R) Xeon(R) CPU           E5530  @ 2.40GHz
No. of Cores:	16

Memory

Total real memory available:	65511 MB
Logically used memory:		3945 MB
Logically available memory:	61565 MB

Storage

The storage is a large ~90TB Enterprise class Fibre Channel array, a Data Direct Networks S2A9900. Connected to that are two, dual port QLogic 2532 8Gb HBA’s. We also have two SSD drives (configured as a RAID1 device) for the ZFS Intent Log.

The storage array was configured from 120 1TB, 7200RPM Hitachi drives. It has 12 volumes in total, composed of 10 of the SATA drives (1 parity, 1 Spare), or ~7TB.

The S2N9900 has two controllers, one controller is responsible for LUN’s 1-6, the other controller is responsible for LUN’s 7-12. every LUN is presented to all four Fibre Channel ports. This got a little messy, trying to sort out 48 raw disk devices takes some patience and a decent attention span…

yeah, I did make a few typo’s here and there, thankfully creating and clearing disk labels is easy.

# camcontrol devlist|grep lun\ 0
                at scbus0 target 0 lun 0 (pass0,da0)
                at scbus1 target 0 lun 0 (pass6,da6)
                at scbus4 target 0 lun 0 (pass24,da24)
                at scbus5 target 0 lun 0 (pass30,da30)

# camcontrol inquiry da0 -S
108EA1B10001
# camcontrol inquiry da6 -S
108EA1B10001
# camcontrol inquiry da24 -S
108EA1B10001
# camcontrol inquiry da30 -S
108EA1B10001
# gmultipath label -v DDN-v00 /dev/da0 /dev/da6 /dev/da24 /dev/da30
Done.

# gmultipath status
             Name  Status  Components
multipath/DDN-v00     N/A  da0
                           da6
                           da24
                           da30

Now, to do that 12 more times…

Whew, hard work!

Now, to create a simple ZFS volume across all 12 luns:

# zpool create zfs multipath/DDN-v00 multipath/DDN-v01 multipath/DDN-v02 multipath/DDN-v03 multipath/DDN-v04 multipath/DDN-v05 multipath/DDN-v06 multipath/DDN-v07 multipath/DDN-v08 multipath/DDN-v09 multipath/DDN-v10 multipath/DDN-v11 log mfid1

# zpool status
  pool: zfs
 state: ONLINE
 scrub: none requested
config:

	NAME                 STATE     READ WRITE CKSUM
	zfs                   ONLINE       0     0     0
	  multipath/DDN-v00  ONLINE       0     0     0
	  multipath/DDN-v01  ONLINE       0     0     0
	  multipath/DDN-v02  ONLINE       0     0     0
	  multipath/DDN-v03  ONLINE       0     0     0
	  multipath/DDN-v04  ONLINE       0     0     0
	  multipath/DDN-v05  ONLINE       0     0     0
	  multipath/DDN-v06  ONLINE       0     0     0
	  multipath/DDN-v07  ONLINE       0     0     0
	  multipath/DDN-v08  ONLINE       0     0     0
	  multipath/DDN-v09  ONLINE       0     0     0
	  multipath/DDN-v10  ONLINE       0     0     0
	  multipath/DDN-v11  ONLINE       0     0     0
	logs                 ONLINE       0     0     0
	  mfid1              ONLINE       0     0     0

errors: No known data errors

Results

These results wre obtained from two similar servers. The other server is using a Winchester Systems Storage array, and has 24GB of system memory. The Winchester Storage is ~40TB of 2TB SATA disks:

I used IOZone for the test (iozone -a). The default iozone test is using 64k files to 512MB files, and since I’m trying to see how the server might actually react to the real worl, I’m okay with this (ie, I fully understand that a LOT of caching is taking place, and I want that for right now).

The S2N9900 is a pretty nice device. Although you have to use TELNET (yeesh, couldn’t they spend a few more bucks on a small ARM processor and use ssh?), the controllers have a decent command line environment with HELP pages. What is also nice is the company provides the documentation for their products for free, and no registration is required. Good Job!

As far as raw read and write speeds, that is hard to nail down. I’ve been using IOZone, and when I run that, and take a look at ‘zpool iostat 1′, the ZFS Pool stays at a constant 200MB/sec for writes. I’ve seen in pop up higher, like 250MB to 500MB, but 200 seems to be the ceiling. I’ve done with and without a dedicated log device, with and without gmultipath, and finally, using the SSD RAID1 as a L2ARC cache device. All results are nearly identical. Reads are pretty crazy though, with 64GB of system memory, reading a file is nearly 1GB/sec.

I don’t have a central theme with this post, but I wanted to at least do something (it has been a while).

Packet Filter

Based on Chris’s “Falling in love with pf(4)” google status, I decided to take the plunge and move off of ipfw(4) to pf(4). I’m not at the point where I could write my one filter, however, I do feel I at least understand what is happening here. I also took the time to update all my ports, so I’m even running PHP 5.3.2, the latest WP release and about 600 other installed ports (Yikes, I’ve got a LOT of stuff on this server!).

I’ve always built my own kernel, at least on my home server, so the first thing to do is sync my /usr/src tree:

$ sudo su -
root# csup ~/bin/src-supfile
...
root# vim /usr/src/sys/amd64/conf/BLACKHOLE
# pf
device          pf
device          pflog
# pf's QoS - ALTQ
options         ALTQ
options         ALTQ_CBQ        # Class Bases Queuing (CBQ)
options         ALTQ_RED        # Random Early Detection (RED)
options         ALTQ_RIO        # RED In/Out
options         ALTQ_HFSC       # Hierarchical Packet Scheduler (HFSC)
options         ALTQ_PRIQ       # Priority Queuing (PRIQ)
options         ALTQ_NOPCC      # Required for SMP build

root# cd /usr/src ; make -j8 buildkernel && make installkernel && reboot

I use tcsh, a C Shell variant, and I find the AND (&&) operator really useful to chain commands together but I require that they succeed. This way, if my build fails, it will abort and not proceed with the install and reboot.

Now that I have a updated kernel with PF enabled, I had to steal Chris’s configuration:

/etc/rc.conf

root# vim /etc/rc.conf
#
# Packet Filter
#
pf_enable="YES"
pf_rules="/etc/pf.conf"
pflog_enable="YES"

#
# Unused, pf replaces all of this
#
#natd_program="/sbin/natd"       # path to natd, if you want a different one.
#natd_enable="YES"                # Enable natd (if firewall_enable == YES).
#natd_interface="em0"               # Public interface or IPaddress to use.
#natd_flags="-u -s -m"                   # Additional flags for natd.
#firewall_enable="YES"
#firewall_script="/usr/local/etc/rc.firewall"
#firewall_logging="YES"

/etc/pf.conf

root# vim /etc/pf.conf
# ----------------------------------------------------------------------------
# "THE BEER-WARE LICENSE" (Revision 42):
# cshumway@titan-project.org wrote this file. As long as you retain this notice you
# can do whatever you want with this stuff. If we meet some day, and you think
# this stuff is worth it, you can buy me a beer in return Christopher Shumway
# ----------------------------------------------------------------------------
#
# pf.conf
ext_if="em0"
int_if="em1"
lan_net="192.168.2.0/24"
open_ports="{ domain, ssh, http, https }"

# options
set skip on lo0
set skip on $int_if
set limit states 25000
set loginterface $ext_if
set state-policy if-bound

# scrub traffic
scrub in all

# NAT
nat on $ext_if from $lan_net to any -> ($ext_if)

# upnp redirection
rdr-anchor "miniupnpd"
anchor "miniupnpd"

# antispoofing
antispoof for $ext_if

# rules start here
block in
pass out on $ext_if keep state
pass in on $ext_if inet proto { tcp, udp } from any to ($ext_if) port $open_ports flags S/SA keep state
pass in on $ext_if inet proto icmp

Pretty simple, and after a reboot my top process is java, and not natd(8). I can almost feel the internet becoming faster

Rock n Roll Owen

We’ve made up, and did a publicity photo-op together

Caralyne’s Garden

Dogs

Last month when I helped my Friend Glenn,who now works at An Olde Town Tattoo Parlour in Monterey, move he was cool enough to lighten his load by giving me a very cool cream and black Gibson SG. The original heavy metal guitar made popular by Toni Iommi and Angus Young.

The guitar’s neck was broken at the body, and thankfully someone who knew what they were doing successfully glued it back on. Glenn put some heavy .11 gauge strings on it to keep a good amount of tension to help maintain structural integrity. I normally play with .10′s, so its really fun playing with the heavy strings and then switching back to my strat. I’ve decided to keep the heavy strings, and even down tune to D and C# just for fun. Keeping that in mind, I went ahead and picked up two Dimarzio X2N humbuckers, and since most of the brass parts on the SG went through some heavy usage, I also picked up a new tail-piece, bridge, and a new all black faceplate.

Replacing pickups is pretty easy, it required a little (de)soldering but that is the hardest part. What probably took me the most time was figuring out HOW I was supposed to re-wire the new 4 wire X2N pickups in a setup that only had 1 wire…

It seems modern humbuckers are REALLY fancy, and if you have a 5-way selector you can activate them in different combinations. Whereas the SG pickups had only 1 loop for the humbucker, the DiMarzio’s had a north loop and a south loop, and you could either wire them in series or parallel. There is no right way, its a matter of preference I guess, but since I’ve never hear either one, I chose a guide I found on Semour Duncan’s site.

I should point this out, even though I bought DiMarzio Pickups, I actually found more useful documentation on Seymour Duncan’s site. Not only did they have a really nice support section (http://www.seymourduncan.com/support/wiring-diagrams/), they also have color-code diagrams for each pickup vendor. So a big kudo’s for SD, maybe I’ll get those next time.

Okay, onto the work, let’s take a look at what we are working with:

Here are the new parts:

The real work begins:

Let me just say, getting the pickups screwed on to the face plate was HARD. There are two tension springs that keep the pickup stationary, and getting those compressed so the screw could reach the faceplate took a lot of effort. I almost lost two of the four twice since they would shoot off while I was struggling to screw the pickup on.

I had a problem with the two screws that are used to adjust the bridge, the were slightly bigger than the old screws. I actually had to re-drill the holes for them, and this makes me a little uncomfortable. If it ever needs to be replaced, there is now a good chance the holes will have to be filled and re-drilled. Oh well.

This was a great way to spend a Saturday afternoon, working on a guitar while listening to a Buckethead/Praxis playlist on Pandora. I spent most of the evening playing the guitar, those new pickups are very hot! They are going to require special attention so they don’t overload the amp. The sound is killer though, nice and low, a stark contrast to the Stratocaster’s sound.

Reference Material:
2h_2v_2t_3w
color_codes

I recently configured a NFS/Samba server with FreeBSD’s Link Aggregation protocol. Here is how I set it up.

FreeBSD Configuration

/boot/loader.conf

I recommend adding the if_lagg_load=YES and kern.hz=”2000″ to /boot/loader.conf.

The OS will automatically load the lagg kernel module when your network configuration loads, I prefer to explicitly set it to load.

ispfw_load="YES"
kern.hz="2000"
aio_load="YES"
hw.igb.rxd=4096
hw.igb.txd=4096
if_lagg_load="YES"

Since I am using the igb ethernet device (Intel 82575 and 82576 chipsets), I also set the max number of send and receive descriptors from the default from 256 to the maximum 4096. Give some thought to this step, increasing this will allocate more memory per interface. Since there are four in use in this setup, that is an order of magnitude higher than the stock configuration.

/etc/rc.conf

ifconfig_igb0="UP polling"
ifconfig_igb1="UP polling"
ifconfig_igb2="UP polling"
ifconfig_igb3="UP polling"
ifconfig_lagg0="create laggproto lacp laggport igb0 laggport igb1 laggport igb2 laggport igb3 128.115.132.165 netmask 255.255.255.0"

Jumbo Frames

Normally, I set a MTU of 9194 for my igb/82575 ethernet controllers.

Setting an arbitrary MTU size above the default 1500 can cause the unexpected, and decrease the stability in your environment. This configuration is using the Intel 82575 Quad Port 1000 VT adapter, which has the maximum MTU of 9194. Not all ethernet controllers support the same MTU sizes, for instance, Broadcom chipsets have a max MTU of 9022. Also, verify your switch can support Jumbo Frames, and have the ports in use set to the appropriate MTU.

Some Notes from Intel’s 8257[5-6] README:

- Only enable Jumbo Frames if your network infrastructure supports them.
- To enable Jumbo Frames, increase the MTU size on the interface beyond
1500.
- The Jumbo Frames setting on the switch must be set to at least 22 bytes
larger than that of the MTU.
- The maximum MTU setting for Jumbo Frames is 9216. This value coincides
with the maximum Jumbo Frames size of 9234 bytes.
- Using Jumbo Frames at 10 or 100 Mbps may result in poor performance or
loss of link.

Since the Cisco hardware used can be set to a max MTU of 9216, our igb interfaces can be set to 9194.

Of course, the lagg interface wont let me set a mtu higher than 1500. If anyone has additional information on this, that would be great to know.

Results

[root@laggy ~]> ifconfig lagg0

lagg0: flags=8843 metric 0 mtu 9194
	options=1bb
	ether 00:26:b9:62:ae:c8
	inet 128.115.132.165 netmask 0xffffff00 broadcast 128.115.132.255
	media: Ethernet autoselect
	status: active
	laggproto lacp
	laggport: igb3 flags=1c
	laggport: igb2 flags=1c
	laggport: igb1 flags=1c
	laggport: igb0 flags=1c

[root@laggy ~]> netstat -I lagg0 -w 1

            input        (lagg0)           output
   packets  errs      bytes    packets  errs      bytes colls
     87581     0  127400293          0     0          0     0
     85126     0  123891200          0     0          0     0
     84926     0  124237023          0     0          0     0

[root@laggy ~]> netstat -I igb0 -w 1

            input         (igb0)           output
   packets  errs      bytes    packets  errs      bytes colls
     38118     0   54749173          0     0          0     0
     35833     0   51498282          0     0          0     0

[root@laggy ~]> netstat -I igb1 -w 1

            input         (igb1)           output
   packets  errs      bytes    packets  errs      bytes colls
     12889     0   18315538          0     0          0     0
     16303     0   23159114          0     0          0     0
     27672     0   39275792          0     0          0     0

[root@laggy ~]> netstat -I igb2 -w 1

            input         (igb2)           output
   packets  errs      bytes    packets  errs      bytes colls
     23709     0   35778378          0     0          0     0
     24445     0   36901194          0     0          0     0

[root@laggy ~]> netstat -I igb3 -w 1

            input         (igb3)           output
   packets  errs      bytes    packets  errs      bytes colls
        11     0       1535          0     0          0     0
         1     0         60          0     0          0     0

Cisco Configuration

interface GigabitEthernet 8/41
 description (L5D13) laggy
 switchport                           # Required for L2 Etherchannel
 switchport access vlan 132           # VLAN assignment (optional)
 spanning-tree portfast               # Recommended
 channel-group 4 mode active          # Required assign channel # and mode
                                         see table below
 channel-protocol lacp                # Required assign lacp or pagp
 no shutdown

Verify the interface port-channel is not shutdown and add a description

interface Port-channel 4
 description Test Channel 2/22/10 laggy
 switchport
 switchport access vlan 101
 switchport trunk encapsulation dot1q
 no shutdown

This command will cause load balancing to occur for the source and destination IP addresses and applied to all Etherchannels on the switch. Other load balancing options exists.

port-channel load-balance src-dst-ip

Use the following show commands for verifying the condition of the Etherchannel

show interfaces port-channel ‘channel number’
show etherchannel port-channel
show etherchannel summary
show Etherchannel load-balance

Switch# show interfaces port-channel 4
Port-channel4 is up, line protocol is up (connected)
  Hardware is EtherChannel, address is 0017.9499.94ac (bia 0017.9499.94ae)
  Description: TEST CHANNEL 2/22/10 laggy
  MTU 9214 bytes, BW 4000000 Kbit, DLY 10 usec,
     reliability 255/255, txload 1/255, rxload 1/255
  Encapsulation ARPA, loopback not set
  Keepalive set (10 sec)
  Full-duplex, 1000Mb/s
  input flow-control is off, output flow-control is on
  Members in this channel: Gi8/41 Gi8/43 Gi8/45 Gi8/47

Switch#show etherchannel port-channel
                Channel-group listing:
                -----------------------
Group: 4
----------
                Port-channels in the group:
                ----------------------
Port-channel: Po4    (Primary Aggregator)

------------
Age of the Port-channel   = 0d:05h:25m:33s
Logical slot/port   = 14/1          Number of ports = 4
Port state          = Port-channel Ag-Inuse
Protocol            =   LACP

Ports in the Port-channel: 

Index   Load   Port     EC state        No of bits
------+------+------+------------------+-----------
  1     11     Gi8/41   Active    2
  2     22     Gi8/43   Active    2
  3     44     Gi8/45   Active    2
  0     88     Gi8/47   Active    2

Time since last port bundled:    0d:04h:47m:26s    Gi8/41
Time since last port Un-bundled: 0d:05h:22m:14s    Gi8/47

Switch#show etherchannel summary
Flags:  D - down        P - bundled in port-channel
        I - stand-alone s - suspended
        H - Hot-standby (LACP only)
        R - Layer3      S - Layer2
        U - in use      f - failed to allocate aggregator

        M - not in use, minimum links not met
        u - unsuitable for bundling
        w - waiting to be aggregated
Number of channel-groups in use: 1
Number of aggregators:           1

Group  Port-channel  Protocol    Ports
------+-------------+-----------+-----------------------------------------------
4      Po4(SU)         LACP      Gi8/41(P)  Gi8/43(P)  Gi8/45(P)  Gi8/47(P)
Switch#show Etherchannel load-balance
EtherChannel Load-Balancing Configuration:
        src-dst-ip
        mpls label-ip

EtherChannel Load-Balancing Addresses Used Per-Protocol:
Non-IP: Source XOR Destination MAC address
  IPv4: Source XOR Destination IP address
  IPv6: Source XOR Destination IP address
  MPLS: Label or IP

I need to share this. When I google for “Samba performance”, I never see real numbers, real configuration files, or real hardware environments. All I read are anecdotal recollections, and that is not good enough. I like numbers, and I’ll let the numbers speak for themselves:

    > netstat -I em0 -w 1
                input          (em0)           output
       packets  errs      bytes    packets  errs      bytes colls
         90166     0   98762637      95363     0    5332847     0
         18131     0   24713156      20042     0    1123684     0
             4     0        310          1     0        178     0
             8     0        518          1     0        178     0
         10153     0   10952920      10696     0     598129     0
         92990     0  102837002      98476     0    5514994     0
         92025     0  102680574      97277     0    5439496     0
         92080     0  101799874      97403     0    5448637     0
         75348     0   90861608      80972     0    4537737     0
         90895     0  100323946      95781     0    5360948     0
         89313     0   97371154      94364     0    5278618     0
         81363     0   89229738      85861     0    4803589     0
             2     0        126          3     0        286     0

I was so shocked that I had to use gstat and zpool iostat to verify the information:

    dT: 1.002s  w: 1.000s  filter: da0
     L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
       35   1476      0      0    0.0   1476 188421   23.7  100.0| da0

    > zpool iostat  1
                   capacity     operations    bandwidth
    pool         used  avail   read  write   read  write
    ----------  -----  -----  -----  -----  -----  -----
    tank        5.68T  4.32T      1     81   250K  10.1M
    tank        5.68T  4.32T      0  1.37K      0   175M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M

This is all through Samba (3.3.9), There was no local work being done. I unfortunately didn’t configure MRTG correctly, so it had built a malformed graph while all this happened. Having a picture from all of this would have been nice.

The underlying storage is a SATABoy2 RAID6 array, with a simple “flat” ZFS filesystem (version 13). As cheap as the SATABoy’s are (and come on, they have a terrible IIS web interface), they can at least keep up with the current load.

I have felt that if you are going to use ZFS, you should let it manage the RAID, and not bother with a hardware RAID controller. While the hardware RAID may be faster, ZFS’s ability to self-correct bad blocks is a great feature despite the performance set back. However, RAID6 is pretty good in itself, and having dual parity would ideally reduce the risk of a bad block being detrimental.

One thing I noticed with Samba is it doesn’t seem to be a threaded daemon. When I do a top(1) -H, there are only 2-3 smbd processes, and one of them is running around 30%. Though I don’t really know how well Samba can scale out, this environment only has about 10 users. I would like to see how samba reacts if there are a couple hundred active users. Furthermore, how does a native Windows server handle a couple hundred users? It may handle it a little better, however, I don’t think I would enjoy watching NTFS handling a multi-terabyte volume… it would be like watching a stroke victim eat a bowl of soup. I do admit I am biased and I have no working experience with Windows as a large file server, most of them that I have worked on are horribly limited and underpowered, and no one seems to care if they perform well or not.

Hardware

CPU information

    Machine class:    amd64
    CPU Model:    Dual Core AMD Opteron(tm) Processor 285
    No. of Cores:    4
    Cores per CPU:

RAM information

    Memory information from dmidecode(8)
    Maximum Capacity: 8 GB
    Number Of Devices: 4
    Maximum Capacity: 8 GB
    Number Of Devices: 4

    INFO: Run `dmidecode -t memory` to see further information.

    System memory summary
    Total real memory available:    8048 MB
    Logically used memory:        2876 MB
    Logically available memory:    5172 MB

    Swap information
    Device          1K-blocks     Used    Avail Capacity
    /dev/da1s1b       8373844      28K     8.0G     0%

Storage information

    Available hard drives:
    cd0:  Removable CD-ROM SCSI-0 device
    cd0: 1.000MB/s transfers
    da2:  Fixed Direct Access SCSI-5 device
    da2: 300.000MB/s transfers
    da2: Command Queueing enabled
    da2: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C)
    da1:  Fixed Direct Access SCSI-2 device
    da1: 300.000MB/s transfers
    da1: Command Queueing enabled
    da1: 69618MB (142577664 512 byte sectors: 255H 63S/T 8875C)
    da0:  Fixed Direct Access SCSI-5 device
    da0: 200.000MB/s transfers
    da0: Command Queueing enabled
    da0: 10491861MB (21487333120 512 byte sectors: 255H 63S/T 1337524C)

    Raid controllers:
    umass-sim0:
    mpt0:
    vendor='LSI Logic (Was: Symbios Logic, NCR)'
    device='SAS 3000 series, 4-port with 1064 -StorPort'
    isp0:
    vendor='QLogic Corporation'
    device='QLA6322 Fibre Channel Adapter'

    Currently mounted filesystems:
    /dev/da1s1a on /
    devfs on /dev
    tank on /tank
    /dev/ufs/EXPORT on /export

    I/O statistics:
           tty             da0              da1              da2             cpu
     tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
       0    40 63.61 167 10.36  16.53   2  0.03  61.65   0  0.00   1  0  4  0 94
    INFO: Run iostat(8) or gstat(8) to see live statistics.

    Disk usage:
    Filesystem         Size    Used   Avail Capacity  Mounted on
    /dev/da1s1a         58G    3.4G     50G     6%    /
    devfs              1.0K    1.0K      0B   100%    /dev
    tank               9.8T    5.7T    4.1T    58%    /tank
    /dev/ufs/EXPORT    126G    148K    116G     0%    /export

Software

• FreeBSD 8.0-RELEASE-p1 FreeBSD 8.0-RELEASE-p1 amd64
• samba-3.3.9 A free SMB and CIFS client and server for UNIX

Samba 3.3.9 Compile-Time Config

> make showconfig
===> The following configuration options are available for samba-3.3.9:
     LDAP=on "With LDAP support"
     ADS=on "With Active Directory support"
     CUPS=off "With CUPS printing support"
     WINBIND=on "With WinBIND support"
     SWAT=off "With SWAT WebGUI"
     ACL_SUPPORT=on "With ACL support"
     AIO_SUPPORT=on "With Asyncronous IO support"
     FAM_SUPPORT=on "With File Alteration Monitor"
     SYSLOG=on "With Syslog support"
     QUOTAS=on "With Disk quota support"
     UTMP=off "With UTMP accounting support"
     PAM_SMBPASS=on "With PAM authentication vs passdb backends"
     DNSUPDATE=off "With dynamic DNS update(require ADS)"
     DNSSD=off "With DNS service discovery support"
     EXP_MODULES=on "With experimental modules"
     POPT=on "With system-wide POPT library"
     MAX_DEBUG=off "With maximum debugging"
     SMBTORTURE=off "With smbtorture"
===> Use 'make config' to modify these settings

System Tuning

The Kernel

I enabled device polling, and took out debugging in the kernel (Sanders, get it! Mmm, I’m hungry…)

diff /usr/src/sys/amd64/conf/GENERIC /usr/src/sys/amd64/conf/SANDERS
    33d32
    < makeoptions    DEBUG=-g        # Build kernel with gdb(1) debug symbols
    78c77
    <
    ---
    > options        DEVICE_POLLING

/boot/loader.conf

    ispfw_load="YES"
    kern.hz="2000"
    aio_load="YES"

/etc/sysctl.conf

    kern.coredump=0
    security.bsd.see_other_uids=0
    security.bsd.see_other_gids=0
    kern.ipc.maxsockbuf=16777216
    kern.ipc.nmbclusters=32768
    kern.ipc.somaxconn=32768
    kern.maxfiles=65536
    kern.maxfilesperproc=32768
    kern.maxvnodes=800000
    net.inet.tcp.delayed_ack=0
    net.inet.tcp.inflight.enable=0
    net.inet.tcp.path_mtu_discovery=0
    net.inet.tcp.recvbuf_auto=1
    net.inet.tcp.recvbuf_inc=524288
    net.inet.tcp.recvbuf_max=16777216
    net.inet.tcp.recvspace=65536
    net.inet.tcp.sendbuf_auto=1
    net.inet.tcp.sendbuf_inc=524288
    net.inet.tcp.sendspace=65536
    net.inet.udp.maxdgram=57344
    net.inet.udp.recvspace=65536
    net.local.stream.recvspace=65536
    net.inet.tcp.sendbuf_max=16777216
    net.inet.tcp.mssdflt=9142

rc.conf (em0 flags)

I want to thank Zilla (see post comments) for the sysctl.conf help.

    ifconfig_em0="inet xxx.xxx.xxx.xxx  netmask 255.255.255.0 polling tso mtu 9194"

smb.conf

        min receivefile size = 131072
        aio read size = 1
        aio write size = 1
        use sendfile = yes
        lock directory = /var/run/samba/
        keepalive = 300

I’m also using LDAP users and group. I wasn’t sure if there would be a noticible performance hit for local users or LDAP users. There doesn’t seem to be one.

We use Active Directory, and since Quest/Vintela still won’t make a FreeBSD client for the Quest Authentication Servers ( a sales rep once told me “There are just too many versions of BSD…”) , I have to use all the open source utilities like OpenSSL, OpenLDAP Client and Kerberos. I don’t mind having to do it, but it is always nice if you can maintain one standard process across ALL systems, and we have a lot more Linux and Solaris systems than FreeBSD. I’m the odd one.

That aside, I use the latest OpenSSL in FreeBSD 8.0, OpenLDAP 2.4.20, and the built-in version of Heimdal Kerberos.

I get similar performance form NFS, however, most desktop users have are either on a Windows or OS X, and CIFS seems to be the unifying network storage protocol.

One thing I have yet to really figure out is configuring Samba to use proper NT ACL’s. However, if you can live with UNIX style permissions, a setup like this is pretty good at serving out lots and lots of data. Maybe that will be next.

I recently had a little problem with a new FreeBSD install, and it is one of those times were I sort of appreciate how FreeBSD assigns device handles, yet at the same time hate it

The setup is this:
The OS was installed on a mirrored hardware raid device (using the mpt(4) driver), and then I had a large RAID6 array attached via a FC controller (using the isp(4) driver). When I installed the OS, the mpt device was showing up as da0. So I went ahead with the install and rebooted the system, so far so good.

What I didn’t realize was the FC device was not seen yet, so after some fiddling, Jenny and I got the large RAID6 array to show up… unfortunately, the isp card was before the mpt card on the PCI bus:

isp0@pci0:2:1:0: class=0x0c0400 card=0x01321077 chip=0x63221077 rev=0x03 hdr=0x00
vendor = 'QLogic Corporation'
device = 'QLA6322 Fibre Channel Adapter'
class = serial bus
subclass = Fibre Channel
mpt0@pci0:2:3:0: class=0x010000 card=0x30601000 chip=0x00501000 rev=0x02 hdr=0x00
vendor = 'LSI Logic (Was: Symbios Logic, NCR)'
device = 'SAS 3000 series, 4-port with 1064 -StorPort'
class = mass storage
subclass = SCSI

and the RAID6 now became da0, and the OS device now became da1.

Doh!

The system prompted for the / drive, so I had to call out the correct device at the mount> prompt:

mount> ufs:/dev/da1s1a

After that, the system continue to boot into mult-user mode, which cause some very strange console behavior (it acted like the return key was being held down), and my only option was to SSH in as local user, su to root, and then fix /etc/fstab.

This was not devastating, however, it show the importance of using disk labels instead of device handles in certain use cases. I haven’t fixed the / mount, but to get a comfort level with using GEOM labels I added another drive to the system and called it EXPORT.

You can assign a permanent label in two ways (that I know of). When you newfs the device, you can specify the L flag (BTW, -O2 means to use UFS2, and -U will use Soft-Updates):
[root@paper ~]> newfs -O2 -U -L EXPORT /dev/da2s1a
OR using glabel (which is what you would have to do for a non UFS filesystem.
[root@paper ~]> glabel create EXPORT da2s1a
Now we can see our newly labeled device in action:
[root@paper ~]> ls /dev/label
. .. EXPORT
[root@paper ~]> glabel status
Name Status Components
label/EXPORT N/A da2s1a

To add it to /etc/fstab, you can either edit the file, or append the correctly tab-delimited line like so:

[root@paper ~]> echo "/dev/label/EXPORT\t/export\tufs\trw\t2\t2" >> /etc/fstab
[root@paper ~]> mkdir /export
[root@paper ~]> mount export

Hurray!

[root@paper ~]> df
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/da1s1a 60931274 4754540 51302234 8% /
devfs 1 1 0 100% /dev
tank 10569645824 107237376 10462408448 1% /tank
/dev/label/EXPORT 132022788 4 121460962 0% /export

[root@paper ~]> mount
/dev/da1s1a on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
tank on /tank (zfs, NFS exported, local)
/dev/label/EXPORT on /export (ufs, local)


This is now a persistent label. To be safe, I’ll have to boot off of a CD/USB drive and modify the root device.

I’ve always had a vested interest in the entire nvidia display driver for FreeBSD project, and I’ve pretty attached to the project. So much, that back in 2001 I started a little petition, got enough attention (and more importantly, a large list of people who signed my petition), and ever since 2002 FreeBSD users have been able to use high quality nvidia drivers. It wasn’t all me, whoever ran nvidia.netexplorer.org asked me to combine efforts, and I gave them my list, and they continued to market it and work with some folks at nvidia.

It is really nice to see that both the FreeBSD team and nvidia have worked together to do the necessary kernel development and get a 64bit driver. I used to use FreeBSD as my primary desktop at work, and it was great to use the hardware drivers for my displays. What is also nice is people in the nvidia forums are also asking for CUDA drivers on FreeBSD, that would be slick as well.

Digg the story if you want too:
http://digg.com/linux_unix/Official_64bit_NVIDIA_drivers_for_FreeBSD

So, it looks like FreeBSD 8.0 has been pre-released; the official date is going to be 11/25, as noted in src/UPDATING:

Updating Information for FreeBSD current users

This file is maintained and copyrighted by M. Warner Losh
.  See end of file for further details.  For commonly
done items, please see the COMMON ITEMS: section later in the file.

Items affecting the ports and packages system can be found in
/usr/ports/UPDATING.  Please read that file before running
portupgrade.

NOTE TO PEOPLE WHO THINK THAT FreeBSD 8.x IS SLOW ON IA64 OR SUN4V:
        For ia64 the INVARIANTS and INVARIANT_SUPPORT kernel options
        were left in the GENERIC kernel because the kernel does not
        work properly without them.  For sun4v all of the normal kernel
        debugging tools present in HEAD were left in place because
        sun4v support still needs work to become production ready.

20091125:
        8.0-RELEASE.
...

Thanks for the warning, and I don’t feel that 8.0 is slow in any way

You can now update to FreeBSD 8.0 with either syncing your source with csup:

*default host=cvsup.FreeBSD.org
*default base=/usr
*default prefix=/usr
*default delete use-rel-suffix
*default compress
src-all release=cvs tag=RELENG_8_0

Or with freebsd-update(8):

# freebsd-update -r 8.0-RELEASE upgrade

then

# freebsd-update install

and after the reboot, possibly another round of ‘freebsd-update install” to finish things up. You can actually upgrade from 7.2 to 8.0, which is pretty impressive since they are considered major releases (and minor release upgrades work just fine as well).

Why would you upgrade to 8.0 over 7.2? Well, Ivan Voras already has a very nice page on the notable features in 8:
http://ivoras.sharanet.org/freebsd/freebsd8.html
In case you want my short list version of that, here are the big highlights for me:

• Kernel Stuff
  • Kernel limit on amd64 increased (this greatly benefits ZFS)
  • Superpages
  • Network stack virtualization, equal cost multipath routing and other really cool network improvements
  • NGROUPS has been increased from 16 to 1024
  • Other kernel improvements like light weight threads, the new ULE 3.0 Scheduler
  • NFS Locking
  • Qlogic 8GB HBA support
  • New AHCI driver
• Userland Stuff
  • Parallel port builds
  • Jails v2
  • Dtrace
  • CLANG/LLVM Compiler

One of the cool things about FreeBSD is its focus on improving what is there. There have been some really big additions to FreeBSD from time to time, but overall, the goal has been to constantly refine and improve the performance. That is what I’m mostly excited about, the continual refinement of an already robust OS.

There are other features, like CLANG and LLVM or Dtrace, where I’m excited about them, but only because I can’t wait to see how others use them. I myself cannot obtain a lot of useful information from Dtrace, however, a kernel developer who knows what they are doing probably can, and that helps them out (which sometimes helps me out).

I’ve used the BETA and RC versions of 8.0, so not only was I pleased with the experience, I’m also excited to see its adoption with the new improvements. I’ve seen some PostgreSQL and MySQL benchmarks and there was a clear performance gain between 7 and 8.

Now is also a good time to mention that the FreeBSD Foundation is rounding up this years donations.

It’s pretty amazing that FreeBSD is a non-profit group; they do not have a CEO, a marketing department, or a horde of full-time developers… and yet they put out a extremely well engineered OS ( that is the boon of not having a marketing department all decisions are driven by the community demand and the developers, and not buzz-words like “the cloud”) with a killer network stack, and over 22,000 available ports.

Step 1: The Question

Background:
I was watching a new episode of Good Eats (Season 13, Episode 7) and Alton Brown was going over a good coffee recipe he called “Man Coffee”. It had a decent ratio of grounds to water, a coarse grind, and he employed the use of a French Press. Then, to my surprise, he added a bit of Kosher Salt.

So, being the unapologetic AB fan that I am, I decided to see how the addition of salt would taste. I had it on my coffee over the last two weeks, off and on, and I decided that I liked the additional flavor. Michele was skeptical, and didn’t think I could actually tell the difference and it was all in my fan-boy head. I admitted this was a likely scenario, and as a good scientist I have to be honest with the data; and that is the Placebo Effect does have a measurable outcome. So, I told her that I would do a blind taste test and see if there was any noticeable flavor in salted coffee.

So, the question is: Can I taste the difference between regular coffee, and coffee brewed with kosher salt.

Step 2: Research

I did a little google-ing about why people put salt in their coffee. The consensus seemed to be that it can help take away the bitterness, however, if you use enough coffee grounds you should not taste any of the acidic bitterness. Coffee becomes bitter for two reasons:

• the essential oils are washed away during the extraction process, leaving the acidic compounds (those coffee compounds are volatile, so you also have to grind the beans and brew them quickly!)
• the coffee is left on a burner longer than it should

Okay, there are more than two, but I’m not a barista or a chemist so that is all I can cover

I normally grind and brew immediately, and I use about 6 tablespoons for 4 cups of coffee which is “a lot”. I think I avoid the common bitterness pitfalls, and adding salt (to me) just seemed to add more body to the coffee.

The test is only to answer the question about detection, and not if it enhances the coffee’s flavor. That would have to be another trial, and I would need more voulenteers and cups…

Step 3: Propose a Hypothesis

I hypothesize that a pinch of salt to the coffee grounds does add a noticeable effect.

That is pretty straight-forward

Step 4: Test with an Experiment

I decided on a regular blind taste test. I wanted to do a double blind test, however, since I was the subject and Caralyne can’t make coffee, Michele had to brew and pour the samples. I also had to settle for 4 cups (those were the only clean and similar cups we had), I wanted a dozen at least so I would have a large enough sample size.

The problem with this test is that both Michele and I are biased; she knew which cup had what, and I knew what the test was about and what I was looking for in the coffee.

The test is simple, I ground up about 12 tablespoon’s worth of Kirkland Coffee.

I put 6 in a cup with a pinch of salt, and 6 all alone. Each one was brewed with the same coffee maker, with 4 “cups” of water and poured into identical plastic pitchers.

Michele labeled the tasting cups, and I sat down to have a taste of each one, marking which ones I liked more than the others.

Step 5: Analyze Results, Draw a Conclusion

After my 4 samples, Michele and I both decided that there was a difference in taste, and I did prefer the one with salt. She didn’t like either, but that is normal since she doesn’t like the strong coffee I make anyway (she calls it “Cat Pee coffee”, which makes me think of Tiger Brand Coffee:

Tiger Brand Coffee is a real treat, even tigers prefer a cup of it to real meat

).

Conclusion

The salt does add flavor, so this is almost like me drinking a Holiday Spice or Hazelnut coffee. It’s not real untampered coffee, but it is a nice treat. I also tend to drink more water in the morning when I have it, so thats not a bad thing since coffee is a diuretic.

I’d like to acknowledge

• Alton Brown and his excellent show Good Eats
• Science Buddies dot org for their excellent page on The Scientific Method
• Wikipedia for their articles on Blind and Double Blind Testing
• My good friend and co-worker, Jenny, for giving me some great ideas on how to help narrow the test down to get useful results.
• Mr Swarm, my 7th grade science teacher at Park Jr. High, who introduced an entire class to the Scientific Method and let us perform real experiments with fire and food.
• The Skeptics Guide to the Universe, a great science podcast that teaches me the difference between “theory” and “hypothesis”

Have a Scien-tastic day!