My Wushu Blog

I need to share this. When I google for “Samba performance”, I never see real numbers, real configuration files, or real hardware environments. All I read are anecdotal recollections, and that is not good enough. I like numbers, and I’ll let the numbers speak for themselves:

    > netstat -I em0 -w 1
                input          (em0)           output
       packets  errs      bytes    packets  errs      bytes colls
         90166     0   98762637      95363     0    5332847     0
         18131     0   24713156      20042     0    1123684     0
             4     0        310          1     0        178     0
             8     0        518          1     0        178     0
         10153     0   10952920      10696     0     598129     0
         92990     0  102837002      98476     0    5514994     0
         92025     0  102680574      97277     0    5439496     0
         92080     0  101799874      97403     0    5448637     0
         75348     0   90861608      80972     0    4537737     0
         90895     0  100323946      95781     0    5360948     0
         89313     0   97371154      94364     0    5278618     0
         81363     0   89229738      85861     0    4803589     0
             2     0        126          3     0        286     0

I was so shocked that I had to use gstat and zpool iostat to verify the information:

    dT: 1.002s  w: 1.000s  filter: da0
     L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
       35   1476      0      0    0.0   1476 188421   23.7  100.0| da0

    > zpool iostat  1
                   capacity     operations    bandwidth
    pool         used  avail   read  write   read  write
    ----------  -----  -----  -----  -----  -----  -----
    tank        5.68T  4.32T      1     81   250K  10.1M
    tank        5.68T  4.32T      0  1.37K      0   175M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M

This is all through Samba (3.3.9), There was no local work being done. I unfortunately didn’t configure MRTG correctly, so it had built a malformed graph while all this happened. Having a picture from all of this would have been nice.

The underlying storage is a SATABoy2 RAID6 array, with a simple “flat” ZFS filesystem (version 13). As cheap as the SATABoy’s are (and come on, they have a terrible IIS web interface), they can at least keep up with the current load.

I have felt that if you are going to use ZFS, you should let it manage the RAID, and not bother with a hardware RAID controller. While the hardware RAID may be faster, ZFS’s ability to self-correct bad blocks is a great feature despite the performance set back. However, RAID6 is pretty good in itself, and having dual parity would ideally reduce the risk of a bad block being detrimental.

One thing I noticed with Samba is it doesn’t seem to be a threaded daemon. When I do a top(1) -H, there are only 2-3 smbd processes, and one of them is running around 30%. Though I don’t really know how well Samba can scale out, this environment only has about 10 users. I would like to see how samba reacts if there are a couple hundred active users. Furthermore, how does a native Windows server handle a couple hundred users? It may handle it a little better, however, I don’t think I would enjoy watching NTFS handling a multi-terabyte volume… it would be like watching a stroke victim eat a bowl of soup. I do admit I am biased and I have no working experience with Windows as a large file server, most of them that I have worked on are horribly limited and underpowered, and no one seems to care if they perform well or not.

Hardware

CPU information

    Machine class:    amd64
    CPU Model:    Dual Core AMD Opteron(tm) Processor 285
    No. of Cores:    4
    Cores per CPU:

RAM information

    Memory information from dmidecode(8)
    Maximum Capacity: 8 GB
    Number Of Devices: 4
    Maximum Capacity: 8 GB
    Number Of Devices: 4

    INFO: Run `dmidecode -t memory` to see further information.

    System memory summary
    Total real memory available:    8048 MB
    Logically used memory:        2876 MB
    Logically available memory:    5172 MB

    Swap information
    Device          1K-blocks     Used    Avail Capacity
    /dev/da1s1b       8373844      28K     8.0G     0%

Storage information

    Available hard drives:
    cd0:  Removable CD-ROM SCSI-0 device
    cd0: 1.000MB/s transfers
    da2:  Fixed Direct Access SCSI-5 device
    da2: 300.000MB/s transfers
    da2: Command Queueing enabled
    da2: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C)
    da1:  Fixed Direct Access SCSI-2 device
    da1: 300.000MB/s transfers
    da1: Command Queueing enabled
    da1: 69618MB (142577664 512 byte sectors: 255H 63S/T 8875C)
    da0:  Fixed Direct Access SCSI-5 device
    da0: 200.000MB/s transfers
    da0: Command Queueing enabled
    da0: 10491861MB (21487333120 512 byte sectors: 255H 63S/T 1337524C)

    Raid controllers:
    umass-sim0:
    mpt0:
    vendor='LSI Logic (Was: Symbios Logic, NCR)'
    device='SAS 3000 series, 4-port with 1064 -StorPort'
    isp0:
    vendor='QLogic Corporation'
    device='QLA6322 Fibre Channel Adapter'

    Currently mounted filesystems:
    /dev/da1s1a on /
    devfs on /dev
    tank on /tank
    /dev/ufs/EXPORT on /export

    I/O statistics:
           tty             da0              da1              da2             cpu
     tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
       0    40 63.61 167 10.36  16.53   2  0.03  61.65   0  0.00   1  0  4  0 94
    INFO: Run iostat(8) or gstat(8) to see live statistics.

    Disk usage:
    Filesystem         Size    Used   Avail Capacity  Mounted on
    /dev/da1s1a         58G    3.4G     50G     6%    /
    devfs              1.0K    1.0K      0B   100%    /dev
    tank               9.8T    5.7T    4.1T    58%    /tank
    /dev/ufs/EXPORT    126G    148K    116G     0%    /export

Software

• FreeBSD 8.0-RELEASE-p1 FreeBSD 8.0-RELEASE-p1 amd64
• samba-3.3.9 A free SMB and CIFS client and server for UNIX

Samba 3.3.9 Compile-Time Config

> make showconfig
===> The following configuration options are available for samba-3.3.9:
     LDAP=on "With LDAP support"
     ADS=on "With Active Directory support"
     CUPS=off "With CUPS printing support"
     WINBIND=on "With WinBIND support"
     SWAT=off "With SWAT WebGUI"
     ACL_SUPPORT=on "With ACL support"
     AIO_SUPPORT=on "With Asyncronous IO support"
     FAM_SUPPORT=on "With File Alteration Monitor"
     SYSLOG=on "With Syslog support"
     QUOTAS=on "With Disk quota support"
     UTMP=off "With UTMP accounting support"
     PAM_SMBPASS=on "With PAM authentication vs passdb backends"
     DNSUPDATE=off "With dynamic DNS update(require ADS)"
     DNSSD=off "With DNS service discovery support"
     EXP_MODULES=on "With experimental modules"
     POPT=on "With system-wide POPT library"
     MAX_DEBUG=off "With maximum debugging"
     SMBTORTURE=off "With smbtorture"
===> Use 'make config' to modify these settings

System Tuning

The Kernel

I enabled device polling, and took out debugging in the kernel (Sanders, get it! Mmm, I’m hungry…)

diff /usr/src/sys/amd64/conf/GENERIC /usr/src/sys/amd64/conf/SANDERS
    33d32
    < makeoptions    DEBUG=-g        # Build kernel with gdb(1) debug symbols
    78c77
    <
    ---
    > options        DEVICE_POLLING

/boot/loader.conf

    ispfw_load="YES"
    kern.hz="2000"
    aio_load="YES"

/etc/sysctl.conf

    kern.coredump=0
    security.bsd.see_other_uids=0
    security.bsd.see_other_gids=0
    kern.ipc.maxsockbuf=16777216
    kern.ipc.nmbclusters=32768
    kern.ipc.somaxconn=32768
    kern.maxfiles=65536
    kern.maxfilesperproc=32768
    kern.maxvnodes=800000
    net.inet.tcp.delayed_ack=0
    net.inet.tcp.inflight.enable=0
    net.inet.tcp.path_mtu_discovery=0
    net.inet.tcp.recvbuf_auto=1
    net.inet.tcp.recvbuf_inc=524288
    net.inet.tcp.recvbuf_max=16777216
    net.inet.tcp.recvspace=65536
    net.inet.tcp.sendbuf_auto=1
    net.inet.tcp.sendbuf_inc=524288
    net.inet.tcp.sendspace=65536
    net.inet.udp.maxdgram=57344
    net.inet.udp.recvspace=65536
    net.local.stream.recvspace=65536
    net.inet.tcp.sendbuf_max=16777216
    net.inet.tcp.mssdflt=9142

rc.conf (em0 flags)

I want to thank Zilla (see post comments) for the sysctl.conf help.

    ifconfig_em0="inet xxx.xxx.xxx.xxx  netmask 255.255.255.0 polling tso mtu 9194"

smb.conf

        min receivefile size = 131072
        aio read size = 1
        aio write size = 1
        use sendfile = yes
        lock directory = /var/run/samba/
        keepalive = 300

I’m also using LDAP users and group. I wasn’t sure if there would be a noticible performance hit for local users or LDAP users. There doesn’t seem to be one.

We use Active Directory, and since Quest/Vintela still won’t make a FreeBSD client for the Quest Authentication Servers ( a sales rep once told me “There are just too many versions of BSD…”) , I have to use all the open source utilities like OpenSSL, OpenLDAP Client and Kerberos. I don’t mind having to do it, but it is always nice if you can maintain one standard process across ALL systems, and we have a lot more Linux and Solaris systems than FreeBSD. I’m the odd one.

That aside, I use the latest OpenSSL in FreeBSD 8.0, OpenLDAP 2.4.20, and the built-in version of Heimdal Kerberos.

I get similar performance form NFS, however, most desktop users have are either on a Windows or OS X, and CIFS seems to be the unifying network storage protocol.

One thing I have yet to really figure out is configuring Samba to use proper NT ACL’s. However, if you can live with UNIX style permissions, a setup like this is pretty good at serving out lots and lots of data. Maybe that will be next.

Patti called me on Friday, and invited me to come Sunday for a workout and food. This has always been a nice little tradition at Wushu West, Patti likes to do this around the holidays.

I was pretty excited, I haven’t been to wushu since June/July, when they all went to China for the summer. I have sort of enjoyed my free Tuesday and Thursday nights, and so has Caralyne. A month or so ago she told me she was really glad that I didn’t go to wushu anymore. That bummed me out a little. Wushu is something I have enjoyed for the last 8+ years, and all of Caralyne’s life I’ve been there two nights out of the week. So, she has the right to want me around, and I should at the same time be allowed to continue one of my big passions in life.

So, I got her ready Sunday morning and took her with me, we both went as students

She was pretty excited, and I was very proud that she not only jogged with all of us in the beginning, but she fully participated in the whole class with the other kids. Sunday classes are great for this, since its one class for all levels. The also let me get a great workout, and after class we ate some delicious Chineses food (and Kringle, but it wasn’t that popular…), then she ran around and played tag for an hour while I talked with Pierre, James, Miles and all the other people there.

I’ve missed that place, and I’ve missed the killer workout only Wushu West can give you. I feel I’m in pretty good shape, yet its no where near the kind of “Wushu Shape” that you need to be in to survive a whole class. Today was pretty light, but I did manage to push myself hard enough to destroy my right leg. All the kicking and jumping really seized it up.

We had a good time, Patti said we could come together on Sundays. I’m going to think about it, paying for the two of us to go might be hard, but at least we get to do something together in the process.

James was very excited to see me, so he told me a bunch of different ways I could work wushu back in my schedule. Pierre offered his own special “Pierre brand” of advice on the subject of payments. We’ll see, today was a great class, I had a great attitude while training and I was very happy that I could do frog leaps, and look over to see Caralyne kicking some target pads. James had said that he’s been leading the classes with a heavy focus on basics and leg strengthening, so he tried to woo me even more with promises of brutal leg conditioning. I can believe it, we did a awesome ab workout at the end (synchronized situps, then synchronized double trunk twisters).

We’ll see.

Tools

• 2 cookie sheets
• 1 large mixing bowl
• 1 medium mixing bowl
• parchment paper

The Ingredients

The Dry Stuff

• 4 cups of AP flour
• 1 cup of Vegetable Shortening (original recipe called for lard, Michele winced at the mention of that though )
• 3 tablespoons of sugar
• 1 teaspoon of kosher salt

The Wet Stuff

• 1 package of dry yeast
• 1/2 cup of lukewarm water
• 3 egg yolks
• 1 cup of lukewarm milk

The Icing

• 1 tablespoon of water
• 1 cup of powdered sugar
• 1/2 teaspoon of vanilla extract

Fillings

Fruits can be used, like { raisins, dates, prunes, apricots, cherries}, I use a brown sugar and chopped walnuts filling, with some small butter chunks. This was how my Grandma Mary made it, and that is how I like it.

Directions:

Dissolve the dry yeast in 1 cup of lukewarm water. It will take a few minutes for the yeast to “wake up”, so set it off to the side.

In a large bowl, mix the Dry Stuff (flour, shortening, sugar and salt). Mix it well enough so it will clump together, and break apart. Like a pie crust.

Now for the Wet Stuff. In another bowl, separate the 3 egg yolks. Beat them together, and add in 1 cup of lukewarm milk and the yeast.

Combine the Wet Stuff into the Dry Stuff, mix well.

Cover the bowl with saran wrap, and place in the refrigerator over night.

Sleep…

Day Two

Prepare a surface area to roll the dough out on. Sprinkle with flour.

Prepare the cookie sheets by covering them with the parchment paper.

Split the dough into 4 equal parts.

Roll each one out, make them as long as your cookie sheet and 9 inches wide.

When you roll one out, transfer it to a cookie sheet. Sprinkle your filling, in this case, brown sugar+chopped walnuts+a little bit of butter down the middle.

Fold both sides, one over the other. Repeat for the next three. You should be able to fit two on one cookie sheet.

Let them rise for 2-3 hours. This may vary, I let mine rise for 4 hours in a colder environment. My Dad told me that they were originally left out for 2 – 3 hours next to their stove.

Two Hours later…

Bake in the oven for 20 – 30 minutes at 350F

Mix your icing ingredients together, and while the Kringle is still warm, coat generously

You may also sprinkle some crushed walnuts on top as well.

EAT!

I totally ripped off Alton Brown for this format While I was making the dough, Caralyne said I looked like “the Good Eats guy…”. She said it in a tone that meant she wasn’t complimenting me, but I’m going to take it as one anyway.

This recipe was from my Grandma Mary, and she made a bunch of them every year around Christmas. She taught my sister and Michele how to make it, and last year I helped Michele, but this year I want to do as much of it as I could myself. I feel very proud that I can continue this tradition.

My Grandparents were originally from Wisconsin, where there is more of a Dutch and Swedish community. It is also, as was explained to me by my Grandma, extremely cold. When my Grandpa Hap was stationed at Camp Stoneman, Pittsburg CA, my Grandma moved out here with him. My Grandma liked the warm California weather, and when it came time to go back home she demanded they stay. My Grandpa was a little resistant, so my Grandma asked,

What does Wisconsin have that California doesn’t?!

to which my Grandpa simply said,

Danish Kringle.

So, the deal was made, my Grandma would make Kringle, and they would stay in Sunny California.

That is the Carlson legend anyway.

My Dad also said that he only knows of one other place in California that makes Kringle, and that is in Solvang, CA.

!!!
I came home with Owen last night, it was dark in the house. I was going to Owen’s room to get him ready for bed, and I went to the corner of the room to turn the light on. What I forgot, and didn’t see, was my guitar that I left of the floor. I normally lean it up against my computer, or the wall, but sometimes I’ll leave it on the floor to encourage Owen to play with it. I guess that wasn’t such a good habit to get into.

Repairing a neck like that would cost about as much as I bought that guitar for, so I’ll pull down Michele’s old acoustic guitar.

I recently had a little problem with a new FreeBSD install, and it is one of those times were I sort of appreciate how FreeBSD assigns device handles, yet at the same time hate it

The setup is this:
The OS was installed on a mirrored hardware raid device (using the mpt(4) driver), and then I had a large RAID6 array attached via a FC controller (using the isp(4) driver). When I installed the OS, the mpt device was showing up as da0. So I went ahead with the install and rebooted the system, so far so good.

What I didn’t realize was the FC device was not seen yet, so after some fiddling, Jenny and I got the large RAID6 array to show up… unfortunately, the isp card was before the mpt card on the PCI bus:

isp0@pci0:2:1:0: class=0x0c0400 card=0x01321077 chip=0x63221077 rev=0x03 hdr=0x00
vendor = 'QLogic Corporation'
device = 'QLA6322 Fibre Channel Adapter'
class = serial bus
subclass = Fibre Channel
mpt0@pci0:2:3:0: class=0x010000 card=0x30601000 chip=0x00501000 rev=0x02 hdr=0x00
vendor = 'LSI Logic (Was: Symbios Logic, NCR)'
device = 'SAS 3000 series, 4-port with 1064 -StorPort'
class = mass storage
subclass = SCSI

and the RAID6 now became da0, and the OS device now became da1.

Doh!

The system prompted for the / drive, so I had to call out the correct device at the mount> prompt:

mount> ufs:/dev/da1s1a

After that, the system continue to boot into mult-user mode, which cause some very strange console behavior (it acted like the return key was being held down), and my only option was to SSH in as local user, su to root, and then fix /etc/fstab.

This was not devastating, however, it show the importance of using disk labels instead of device handles in certain use cases. I haven’t fixed the / mount, but to get a comfort level with using GEOM labels I added another drive to the system and called it EXPORT.

You can assign a permanent label in two ways (that I know of). When you newfs the device, you can specify the L flag (BTW, -O2 means to use UFS2, and -U will use Soft-Updates):
[root@paper ~]> newfs -O2 -U -L EXPORT /dev/da2s1a
OR using glabel (which is what you would have to do for a non UFS filesystem.
[root@paper ~]> glabel create EXPORT da2s1a
Now we can see our newly labeled device in action:
[root@paper ~]> ls /dev/label
. .. EXPORT
[root@paper ~]> glabel status
Name Status Components
label/EXPORT N/A da2s1a

To add it to /etc/fstab, you can either edit the file, or append the correctly tab-delimited line like so:

[root@paper ~]> echo "/dev/label/EXPORT\t/export\tufs\trw\t2\t2" >> /etc/fstab
[root@paper ~]> mkdir /export
[root@paper ~]> mount export

Hurray!

[root@paper ~]> df
Filesystem 1K-blocks Used Avail Capacity Mounted on
/dev/da1s1a 60931274 4754540 51302234 8% /
devfs 1 1 0 100% /dev
tank 10569645824 107237376 10462408448 1% /tank
/dev/label/EXPORT 132022788 4 121460962 0% /export

[root@paper ~]> mount
/dev/da1s1a on / (ufs, local, soft-updates)
devfs on /dev (devfs, local, multilabel)
tank on /tank (zfs, NFS exported, local)
/dev/label/EXPORT on /export (ufs, local)


This is now a persistent label. To be safe, I’ll have to boot off of a CD/USB drive and modify the root device.

A few weeks ago, I was snapping some pictures of Owen in the back yard

And I brought the camera focus on my long time yard nemesis:

No one was home (beside Owen), which means I didn’t have any adult supervision, and I wanted to see how difficult it would be to cut the tree down. I grabbed my small hand saw and cut down one of the trees. It was pretty easy, and felling that tree felt pretty good. With that, I started cutting away with my hand saw.

Shortly after cutting down a few other branches, things got complicated:

the branch that fell in the neighbors yard caused a little dilemma. I didn’t see anyone home, and the branch was crushing the fence, and, the heaviest part was hovering over their pool pump. I’ve never met them, but I felt it was best if I hopped the fence with my primitive tools to cut it up, and then clean it up, hopefully before they got back ( I didn’t know how they would feel about a stranger in their yard). After deducing that they didn’t own a dog, I hopped over and go busy. It took maybe 30 minutes to cut it all up and heave it over to my yard. When I was almost done, I heard a voice behind me…

You can put the rest in my yard waste bin on the side of the house… and thank you for cutting that down.

I was pretty happy that he wasn’t upset with the situation (and its a good thing he only saw the small stuff, and not the entire branch ), and it was even nice to feel like he appreciated getting rid of that eye-sore of a tree.

The next day I was able to tackle the main trunk. It had a few off-shoots, and I got as many as I could, however, I ended up making the tree back-heavy by taking off a large cluster on the front. That meant that all the weight was hovering other the other neighbors yard, and if this tree fell improperly, it would cause a lot of damage.

So the next day, Michele called her Dad, and then my Step Dad (Big Mike), and they had it down in 5 minutes. This was while I was at work, and I felt a little robbed, but it was a little hazardous to leave that tree up in its partially hacked away state. And hey, at least they left the clean-up duty for me

It took me 3 weekends, each weekend I took a full truckload to the wood recycle center on Loveridge in Pittsburg, to get the majority of it. I spread it out over 3 weekends because I cringed every time I had to pay $22 for a truckload.

This weekend I was able to get almost the rest of the debris in our yard waste bin:

I had a lot of fun doing it… that seems odd for me to say. As much as I hated doing this same sort of thing with my Step Dad for YEARS, I’m actually starting to appreciate this sort of activity. I was on such a roll that I decided to put up our lights the same weekend I did the tree:

I’ve always had a vested interest in the entire nvidia display driver for FreeBSD project, and I’ve pretty attached to the project. So much, that back in 2001 I started a little petition, got enough attention (and more importantly, a large list of people who signed my petition), and ever since 2002 FreeBSD users have been able to use high quality nvidia drivers. It wasn’t all me, whoever ran nvidia.netexplorer.org asked me to combine efforts, and I gave them my list, and they continued to market it and work with some folks at nvidia.

It is really nice to see that both the FreeBSD team and nvidia have worked together to do the necessary kernel development and get a 64bit driver. I used to use FreeBSD as my primary desktop at work, and it was great to use the hardware drivers for my displays. What is also nice is people in the nvidia forums are also asking for CUDA drivers on FreeBSD, that would be slick as well.

Digg the story if you want too:
http://digg.com/linux_unix/Official_64bit_NVIDIA_drivers_for_FreeBSD