My Wushu Blog » ZFS

Bacula in the Enterprise – Part 2

Mike Carlson — Sat, 23 Jul 2011 19:10:26 +0000

Software

As mentioned many times, this is a FreeBSD based environment. Some good sysinfo output below:

Operating system release: FreeBSD 8.2-RELEASE
OS architecture: amd64
Kernel build dir location: /usr/obj/usr/src/sys/GENERIC
Currently booted kernel: /boot/kernel/kernel

Currently loaded kernel modules (kldstat(8)):
zfs.ko
opensolaris.ko

Bootloader settings for the Director/Database node:

The /boot/loader.conf has the following contents:
kern.ipc.semmni=1024
kern.ipc.semmns=2048
kern.ipc.semmnu=1024

All of the storage nodes and the director are running a GENERIC kernel with very few system tweaking. One of the storage nodes has a Chelsio 10Gb controller, but that hasn’t had a high enough load to crack the 1Gb/sec barrier.

I’m using Bacula from the ports tree, and the directory has a special Make flag to build with gcc’s debugging symbols. Jenny worked on getting that setup when we were having some stability issues.

The Bacula configuration one the director node is backed by a git repository. It adds a little bit of complexity for a systems administrator, when they want to add a client, but the benefit is clear. This backup project actually enforces change control and tracks all of the commits by who.

I’ve also setup Redmine as a project front-end, and I’ve begun to file tickets and reference what commit fixed what. This not only tracks my progress, but it is the first time I’ve had a backup server that was clearly documented and had some type of accountability.

A snippet of the Redmine site

The Structure

I’ve compared projects like bacula to a large box of LegosTM. It doesn’t enforce a structure by any means, and I’ve taken it upon myself to add meaning to the otherwise flat and incomprehensible bacula-dir.conf

The Bacula Port on FreeBSD installs all configuration files in /usr/local/etc.

Write, the Director, only contains the following in /usr/local/etc/bacula-dir.conf:

@/usr/local/etc/bacula/bacula-dir.conf
@/usr/local/etc/bacula/storage.conf
@/usr/local/etc/bacula/clients.conf
@/usr/local/etc/bacula/messages.conf
@/usr/local/etc/bacula/schedules.conf
@/usr/local/etc/bacula/pools.conf

As you can see, I place everything in etc/bacula/.

Here is a beautiful output of tree(1):

bacula
|-- bacula-dir.conf
|-- bin
|   |-- create_client.sh
|   `-- package_list.sh
|-- clients.conf
|-- clients.d
|   |-- 10am
|   |-- 10pm
|   |-- 11pm
|   |-- 12am
|   |-- 1am
|   |-- 2am
|   |-- 3am
|   |-- 4am
|   |-- 4pm
|   |-- 5am
|   |-- 5pm
|   |-- 6am
|   |-- 6pm
|   |-- 7am
|   |-- 7pm
|   |-- 8am
|   |-- 8pm
|   |-- 9am
|   |-- 9pm
|   |-- TEMPLATE-mac
|   |-- TEMPLATE-unix
|   `-- TEMPLATE-win32
|-- excludes.d
|   |-- common.conf
|   |-- mac.conf
|   |-- unix.conf
|   `-- win32.conf
|-- messages.conf
|-- pools.conf
|-- schedules.conf
|-- storage.conf
`-- storage.d
    |-- write-01.conf
    |-- write-02.conf
    |-- write-03.conf
    |-- write-04.conf
    |-- write-05.conf
    `-- write-06.conf

Storage Nodes

All of the storage nodes are using ZFS as the filesystem/Volume manager.

write-06# zpool list
NAME         SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
filevol001  90.6T  33.3T  57.3T    36%  ONLINE  -

They all have one volume, /filevol001, and I created 512 “drives” within that volume. Effectivly, each storage node has 512 drives, and clients are randomly assigned a drive.

Since I have 6 storage nodes, I wrote a little shell script to handle the directory creation:

#!/bin/bash
i=1
 
while [ $i -le 512 ]
do
        install -d -o bacula -g bacula -m 770 /filevol001/drive$i
        ((i++))
done

Simple, right? I also wrote a script to generate the bacula-sd.conf file on a storage node as well:

#!/bin/bash
 
usage()
{
cat << EOF
    Usage $0 NUMBER > /usr/local/etc/bacula-sd.conf
 
    Where "NUMBER" is just a single digit indicating which storage node this is.
 
    Example, for write-07:
    $ make_sd.sh 7 > /usr/local/etc/bacula-sd.conf
EOF
}
 
i=1
 
if [[ -z $1 ]]
then
    usage
    exit
fi
 
printf "Storage {\n"
printf "\tName = write-0$1.llnl.gov-sd\n"
printf "\tSDAddress = write-0$1.llnl.gov\n"
printf "\tSDPort = 9103\n"
printf "\tWorkingDirectory = \"/var/db/bacula\"\n"
printf "\tPid Directory = \"/var/run\"\n"
printf "\tMaximum Concurrent Jobs = 516\n"
printf "}\n"
 
printf "#\n"
printf "# List Directors who are permitted to contact Storage daemon\n"
printf "#\n"
printf "Director {\n"
printf "\tName = write.llnl.gov-dir\n"
printf "\tPassword = \"ItsASecret\"\n"
printf "}\n"
 
printf "#\n"
printf "# Restricted Director, used by tray-monitor to get the\n"
printf "#   status of the storage daemon\n"
printf "#\n"
printf "Director {\n"
printf "\tName = write.llnl.gov-mon\n"
printf "\tPassword = \"ItsANotherSecret\"\n"
printf "\tMonitor = yes\n"
printf "}\n"
 
printf "Messages {\n"
printf "\tName = Standard\n"
printf "\tdirector = write.llnl.gov-dir = all\n"
printf "}\n"
 
 
printf "Device {\n"
printf "\tName = W0$1FileStorage\n"
printf "\tMedia Type = File\n"
printf "\tArchive Device = /filevol001\n"
printf "\tLabelMedia = yes;\n"
printf "\tRandom Access = Yes;\n"
printf "\tAutomaticMount = yes;\n"
printf "\tRemovableMedia = no;\n"
printf "\tAlwaysOpen = no;\n"
printf "\tMaximum Concurrent Jobs = 2\n"
printf "}\n"
 
while [ $i -le 512 ]
do
        printf "\n"
        printf "Device {\n"
        printf "\tName = W0$1FileStorageD$i\n"
        printf "\tMedia Type = File\n"
        printf "\tArchive Device = /filevol001/drive$i\n"
        printf "\tLabelMedia = yes;\n"
        printf "\tRandom Access = Yes;\n"
        printf "\tAutomaticMount = yes;\n"
        printf "\tRemovableMedia = no;\n"
        printf "\tAlwaysOpen = no;\n"
        printf "\tMaximum Concurrent Jobs = 2\n"
        printf "}\n"
        ((i++))
done

On the Directory, a storage node definition is saved in /usr/local/etc/bacula/storage.d/write-0{N}.conf, which is included in /usr/local/etc/bacula/storage.conf:

@/usr/local/etc/bacula/storage.d/write-01.conf
@/usr/local/etc/bacula/storage.d/write-02.conf
@/usr/local/etc/bacula/storage.d/write-03.conf
@/usr/local/etc/bacula/storage.d/write-04.conf
@/usr/local/etc/bacula/storage.d/write-05.conf
@/usr/local/etc/bacula/storage.d/write-06.conf

Client Generation

There are two components, the TEMPLATE file (there are three, TEMPLATE-unix, TEMPLATE-win32 and TEMPATE-mac) and the shell script.

The Client TEMPLATE File

Here is what one of the TEMPLATE files looks like:

#
# Client Definition, the Password here must match
#  the clients bacula-fd.conf Client definition.
#
# Using Vi/m, you can easily replaced HOSTNAME with
#  the short hostname of the client with:
#  %s/HOSTNAME/yourhostname/
#
#

Client {
    Name = HOSTNAME.llnl.gov
    Address = HOSTNAME.llnl.gov
    FDPort = 9102
    Catalog = Catalog001
    Password = "ItsASecret"
    File Retention = 40 days
    Job Retention = 1 months
    AutoPrune = yes
    Maximum Concurrent Jobs = 10
    Heartbeat Interval = 300
}

Console {
    Name = HOSTNAME.llnl.gov-acl
    Password = ItsASecret
    JobACL = "HOSTNAME.llnl.gov RestoreFiles", "HOSTNAME.llnl.gov"
    ScheduleACL = *all*
    ClientACL = HOSTNAME.llnl.gov
    FileSetACL = "HOSTNAME.llnl.gov FileSet"
    CatalogACL = Catalog001
    CommandACL = *all*
    StorageACL = *all*
    PoolACL = HOSTNAME.llnl.gov-File
}

Job {
    Name = "HOSTNAME.llnl.gov"
    Type = Backup
    Level = Incremental
    FileSet = "HOSTNAME.llnl.gov FileSet"
    Client = "HOSTNAME.llnl.gov"
    Storage = FileStorageD##
    Pool = HOSTNAME.llnl.gov-File
    Schedule = "@@"
    Messages = Standard
    Priority = 10
    Write Bootstrap = "/var/db/bacula/%c.bsr"
    Maximum Concurrent Jobs = 10
    Reschedule On Error = yes
    Reschedule Interval = 1 hour
    Reschedule Times = 1
    Max Wait Time = 30 minutes
    Cancel Lower Level Duplicates = yes
    Allow Duplicate Jobs = no
    RunScript {
        RunsWhen = Before
        FailJobOnError = no
        Command = "/etc/scripts/package_list.sh"
        RunsOnClient = yes
    }
}

Pool {
    Name = HOSTNAME.llnl.gov-File
    Pool Type = Backup
    Recycle = yes
    AutoPrune = yes
    Volume Retention = 1 months
    Maximum Volume Bytes = 10G
    Maximum Volumes = 100
    LabelFormat = "HOSTNAME.llnl.govFileVol"
    Maximum Volume Jobs = 5
}

Job {
    Name = "HOSTNAME.llnl.gov RestoreFiles"
    Type = Restore
    Client= HOSTNAME.llnl.gov
    FileSet="HOSTNAME.llnl.gov FileSet"
    Storage = FileStorageD##
    Pool = HOSTNAME.llnl.gov-File
    Messages = Standard
    #Where = /tmp/bacula-restores
}

FileSet {
    Name = "HOSTNAME.llnl.gov FileSet"
    Include {
        Options {
            signature = MD5
            compression = GZIP6
                        fstype = ext2
                        fstype = xfs
                        fstype = jfs
                        fstype = ufs
                        fstype = zfs
                        onefs = no
                        Exclude = yes
                        @/usr/local/etc/bacula/excludes.d/common.conf
        }
                File = /
                File = /usr/local
                Exclude Dir Containing = .excludeme
    }
    Exclude {
        @/usr/local/etc/bacula/excludes.d/unix.conf
    }
}

The Create Client Script

So here is what really makes creating clients easy for us, the create_client script.

I didn’t want to do it this way, really, so part of me is very ashamed of this tool. I would have preferred to re-write this in Python, or make a web page out of it, and let admins create clients from their desktop. Or, I would have loved to create a puppet module to handle this automagically (but that would exlcude everything that *isn’t* running Puppet, which is huge).

With that disclaimer, here is my create_client shell script:

#!/usr/bin/env bash
# usage: cclient -t unix -s 12am -h hostname
#
 
umask 022
 
# Variables
## Randomize Schedule
SCHEDULES="4pm 5pm 6pm 7pm 8pm 9pm 10pm 11pm 12am 1am 2am 3am 4am 5am 6am 7am 8am 9am 10am"
s=($SCHEDULES)
num_s=${#s[*]}
RAND_SCHED=${s[$((RANDOM%num_s))]}
# Randomize which storage node we use
NODES="write-06 write-01 write-06 write-01 write-02 write-03 write-04 write-05"
n=($NODES)
num_n=${#n[*]}
RAND_NODE=${n[$((RANDOM%num_n))]}
 
export DRIVE=`jot -r 1 1 512`
export BDIR="/usr/local/etc/bacula"
export TYPE="unix"
export SCHEDULE=$RAND_SCHED
export HOSTNAME=""
export STORAGE_NODE=$RAND_NODE
export GIT_DIR="/usr/local/etc/bacula/.git"
export CLASS="desktop"
 
if [ $(whoami) == "root" ]
then
cat << EOF
                Please do not run this as root. This script runs a
                git add/commit, which is how changes are managed and
                tracked. If you run this as root, then it shows up
                as carlson39 or root.
 
                If you encounter a problem with your normal OUN account,
                please contact Mike Carlson, or submit a bug here:
                https://st-scm.llnl.gov/redmine/snt/projects/bacula/issues/new
EOF
 
exit 1
fi
 
usage()
{
cat << EOF
 
        Usage: $0 [OPTION]... -h HOSTNAME
 
        This script will generate a bacula client definition.
 
        OPTIONS:
        -s      schedule, (4pm|5pm|6pm|7pm|8pm|9pm|10pm|11pm|12am|1am|2am|3am|4am|5am|6am|7am|8am|9am). The default schedule is random.
        -t      type, (unix|win32|mac), unix is the default
        -n      storage node (write-01|write-02|...), the default is random.
        -h      hostname (use the short hostname)
EOF
}
 
cd $BDIR
 
while getopts 'c:t:s:n:h:' OPTION
do
        case $OPTION in
                c)
                        CLASS=$OPTARG
                        ;;
                t)
                        TYPE=$OPTARG
                        ;;
                s)
                        SCHEDULE=$OPTARG
                        ;;
                h)
                        HOSTNAME=$OPTARG
                        echo $HOSTNAME | egrep -q "(llnl.gov|ucllnl.org)"
                        if [ $? -eq 0 ]
                        then
                        HOSTNAME=`echo $HOSTNAME|sed -e 's/.llnl.gov//' -e 's/.ucllnl.org//'`
                        fi
 
                        ;;
                n)
                        STORAGE_NODE=$OPTARG
                        ;;
                ?)
                        usage
                        exit
                        ;;
        esac
done
 
if [[ -z $CLASS ]] || [[ -z $TYPE ]] || [[ -z $SCHEDULE ]] || [[ -z $HOSTNAME ]] || [[ -z $STORAGE_NODE ]]
then
        usage
        exit 1
fi
 
grep -w $HOSTNAME $BDIR/clients.conf
if [ $? -eq 0 ]
then
        echo 'client '$HOSTNAME 'already exists...'
else
        export RETRY_COUNT="2"
 
        if [ $STORAGE_NODE == "write-01" ]
        then
                DRIVE=`jot -r 1 33 512`
                sed -e 's/HOSTNAME/'$HOSTNAME'/g' -e 's/FileStorageD##/FileStorageD'$DRIVE'/' -e 's/\@\@/'$SCHEDULE'/' -e 's/RETRY_COUNT/'$RETRY_COUNT'/g' $BDIR/clients.d/TEMPLATE-$TYPE > $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
                echo \@$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf >> $BDIR/clients.conf
        else
                export SN=`echo $STORAGE_NODE | cut -c 7-8`
                sed -e 's/HOSTNAME/'$HOSTNAME'/g' -e 's/FileStorageD##/W'$SN'FileStorageD'$DRIVE'/' -e 's/\@\@/'$SCHEDULE'/' -e 's/RETRY_COUNT/'$RETRY_COUNT'/g' $BDIR/clients.d/TEMPLATE-$TYPE > $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
                echo \@$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf >> $BDIR/clients.conf
        fi
 
        chgrp st-bacula-admins $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
        git add $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf $BDIR/clients.conf
        git commit
        echo 'created client definition: '$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
        echo 'for '$HOSTNAME'.llnl.gov'
fi

This is always a work in progress, but at the core, it is a simple sed wrapper with a lot of randomization and a git commit.

Why all the randomization?

Because I had to add around 1000 clients in a VERY short amount of time. We didn’t have a problem pushing the Bacula client to all of the platforms, nor the bacula-fd.conf file either. What I could not do was spend the time to create and manage all of the resources for each client. That is why I have so many devices/drives, so I can attempt to have a 1:1 without having to actually think about it.

So, I wrote ANOTHER script to wrap around this one when I need to do bulk client creations. I’m not going to post that, it just loops through the above command.

Pre-Job command – Package List

I only do this on the Unix/Linux clients, and I thought it was a cool idea.

Yeah, I will pat myself on the back a little bit for that :)

I exclude the Operating System from backups for two reasons, 1) to reduce backing up duplicate and reproducible data and 2) Our build/Imaging process is so quick and clean it is just faster to rebuild than restore everything.

Still, I needed a way to keep the state of installed packages/software.

This is where the pre-job command comes in handy. This part right here:

    RunScript {
        RunsWhen = Before
        FailJobOnError = no
        Command = "/etc/scripts/package_list.sh"
        RunsOnClient = yes
    }

That package_list.sh file looks like this:

 
#!/usr/bin/env bash
 
export PLIST="/root/plist.txt"
 
case "`uname -s`" in
Linux)
                if [ -x /usr/bin/lsb_release ]; then
                        DIST=`lsb_release -d`
                fi
 
                # RHEL
                if [ -x /usr/bin/up2date ]; then
                        rpm -qa > $PLIST
                fi
 
                # RHEL 5
                if [ -x /usr/bin/yum ]; then
                        if [ -f /var/run/yum.pid ]; then
                                echo "Yum currently in use, exiting gracefully..."
                                exit 0
                        else
                        /usr/bin/yum list installed | awk '{print $1}' > $PLIST
                        fi
                fi
 
                # Ubuntu
                if [ -x /usr/bin/dpkg ]; then
                        /usr/bin/dpkg --get-selections | awk '{print $1}' > $PLIST
                fi
                ;;
 
FreeBSD)
                pkg_info|awk '{print $1}' > $PLIST
                ;;
SunOS)
                pkginfo |awk '{print $1}' > $PLIST
                ;;
esac

That file, /root/plist.txt, gets backed up.

Now we have a record of what was installed on our Unix platforms :)

That is it for now, see you at Part 3

Bacula in the Enterprise – Part 1

Mike Carlson — Sat, 23 Jul 2011 00:45:21 +0000

I’ve been using Bacula, the open source backup software, for over a year now. Things have been going well, and I would like to dedicate a post or two to the environment I built.

Background

Over a year ago, I took it upon myself to replace a single Legato Networker server with Bacula. One of our collaborators had decided to ship us (for no reason at all really, I think they were cleaning out their data center) a Sun X4200 AMD server, and two StorageTek/Sun NAS servers.

I had no reason for the NAS heads, but the JBOD was full of drives and the Sun X4200 was useful enough. So, I gutted them (Since the StoragTek NAS heads were identical in almost every way to a standard X4200), put as much memory and CPU’s as I could in one system. This was my first Bacula server. It had around 2TB of FC storage and it made a nice replacement backup server for the 50 or so clients that were on the Networker server. The OS was not Solaris, as you might guess since I was using Sun hardware, but FreeBSD.

Since was focusing only on disk based backups, and FreeBSD has two fantastic large file systems (UFS2 and ZFS), this was my underlying storage platform. Combine that with the current choice of software (in the case, Bacula 5.0.x and PostgreSQL 8/9) from the Ports tree, it really makes the perfect open source software and hardware stack.

After spending a good amount of time wrapping my head around Bacula, and really, just carelessly diving into it, I was very happy with how fast and stable it was shaping up to be.

Around the same time, I was asked to pick up the project that literally went no where for years: The dreaded Backup Project for all of the S&T directorate. A mix of all OS’s, desktops, servers, laptops, etc… and around 3000 active machines online with lots of important data.

No small feat, and there are many reasons why this had been a difficult project to wrangle. One thing for sure, is we knew we had a lot of unique programmatic data.

I knew what software I wanted to use, and I was pretty set on using commodity hardware and reasonably priced storage. The next part was to define some constants in the environment.

The Initial Concepts

First up, I knew of the largest painful aspects of our computing environment:

Budget constraints – We are not rich, and IT always seems to be underfunded. This project had to be as frugal as possible, yet still deliver.
Diverse platforms – We have Windows, OS X and a mix of RHEL and Ubuntu for desktops. Server platforms range all across the board: Windows, OSX, RHEL, Solaris, FreeBSD, AIX, etc…
Mission critical data – Lets face it, the Lab doesn’t make a car, or a VCR. We have a LOT of critical scientific knowledge that is only stored in bits and bytes. That is our product, scientific data that is truly unique.
A campus like geography – the Lab is 1 sq. mile with a mixture of trailers, new buildings, and some buildings that are over 50 years old. The network backbone ranges from 10Gbps to 10Mbps. The poses a problem when it comes to backups.

With this in mind, theses are some of the initial concepts I latched onto:

A distributed disk based storage backend for Bacula
Smaller retention window – 1 month
Reduce the amount of data that has to even go over the network

Skip “reproducible data” such has installed programs like Office, and exclude the OS itself
Enable client side compression. The effectively distributes the compression for all clients. Saves a lot of disk space :)

Skip virtual machine images, like vmdk and vmem files, and treat important virtual machines as separate clients.

For Bacula itself, I had decided that each client would have its own resources. Very little is shared between each client, so for example client-001 would get all of these resources:

Pool = “Client-001-File”

Storage = a randomly assigned drive on one of the storage nodes
Maximum Volume bytes = 10Gb
Maximum Volumes = 100
AutoPrune and Recycling enabled

Job = “client-001″ (one for backup and one for restore)
FileSet = “client-001 FileSet”

What this prevents is cross-client data contamination. What is also prevented, and I found out later, was concurrent backups. More on that later though.

After feeling a bit more comfortable with a simple all-in-one server, I was ready to spec out new hardware. This was good, because at the same time it was our end of year budget crunch, and hardware had to be procured.

The Hardware

For storage, I had a few concepts I was floating around.

One, was to use MooseFS, a distributed filesystem , across a bunch of cheap node with a modest amount of storage.

The other, idea was to buy a handful of servers with a lot of internal storage, around 18TB or so. Them distribute them across the “campus” as a kind of Bacula storage node cloud.

The last idea was to take a more traditional backup server approach, and buy a server with as much expandable storage as possible and back everything up to that.

In the end, when a bunch of hardware showed up (I had *some* control over the hardware, but not all aspects since some of the managers took it upon themselves to purchase everything), I scrapped the MooseFS idea after talking it over with Jenny, and we took the last two: 4 HP servers with 17TB or RAID6 internal storage, and a large 140TB SAN array (Winchester Systems, great stuff!) as the primary backup node:

Our Environment

This model served us well for a while. We had a primary storage node that backed up a users primary desktop, and the smaller storage nodes were used to back up servers and infrastructure data.

There was room for improvement right away though. As soon as it was in production, I quickly mapped out what I felt were the next steps in making a robust backup environment.

FreeBSD and Multipath

Mike Carlson — Fri, 25 Jun 2010 06:06:58 +0000

I didn’t find any blog posts of discussions on FreeBSD and multipath (for storage) that wasn’t a man page.

That means it is up to me to write about it :)

Hardware

CPU

Machine class:	amd64
CPU Model:	Intel(R) Xeon(R) CPU           E5530  @ 2.40GHz
No. of Cores:	16

Memory

Total real memory available:	65511 MB
Logically used memory:		3945 MB
Logically available memory:	61565 MB

Storage

The storage is a large ~90TB Enterprise class Fibre Channel array, a Data Direct Networks S2A9900. Connected to that are two, dual port QLogic 2532 8Gb HBA’s. We also have two SSD drives (configured as a RAID1 device) for the ZFS Intent Log.

The storage array was configured from 120 1TB, 7200RPM Hitachi drives. It has 12 volumes in total, composed of 10 of the SATA drives (1 parity, 1 Spare), or ~7TB.

The S2N9900 has two controllers, one controller is responsible for LUN’s 1-6, the other controller is responsible for LUN’s 7-12. every LUN is presented to all four Fibre Channel ports. This got a little messy, trying to sort out 48 raw disk devices takes some patience and a decent attention span…

yeah, I did make a few typo’s here and there, thankfully creating and clearing disk labels is easy.

# camcontrol devlist|grep lun\ 0
                at scbus0 target 0 lun 0 (pass0,da0)
                at scbus1 target 0 lun 0 (pass6,da6)
                at scbus4 target 0 lun 0 (pass24,da24)
                at scbus5 target 0 lun 0 (pass30,da30)

# camcontrol inquiry da0 -S
108EA1B10001
# camcontrol inquiry da6 -S
108EA1B10001
# camcontrol inquiry da24 -S
108EA1B10001
# camcontrol inquiry da30 -S
108EA1B10001
# gmultipath label -v DDN-v00 /dev/da0 /dev/da6 /dev/da24 /dev/da30
Done.

# gmultipath status
             Name  Status  Components
multipath/DDN-v00     N/A  da0
                           da6
                           da24
                           da30

Now, to do that 12 more times…

Whew, hard work!

Now, to create a simple ZFS volume across all 12 luns:

# zpool create zfs multipath/DDN-v00 multipath/DDN-v01 multipath/DDN-v02 multipath/DDN-v03 multipath/DDN-v04 multipath/DDN-v05 multipath/DDN-v06 multipath/DDN-v07 multipath/DDN-v08 multipath/DDN-v09 multipath/DDN-v10 multipath/DDN-v11 log mfid1

# zpool status
  pool: zfs
 state: ONLINE
 scrub: none requested
config:

	NAME                 STATE     READ WRITE CKSUM
	zfs                   ONLINE       0     0     0
	  multipath/DDN-v00  ONLINE       0     0     0
	  multipath/DDN-v01  ONLINE       0     0     0
	  multipath/DDN-v02  ONLINE       0     0     0
	  multipath/DDN-v03  ONLINE       0     0     0
	  multipath/DDN-v04  ONLINE       0     0     0
	  multipath/DDN-v05  ONLINE       0     0     0
	  multipath/DDN-v06  ONLINE       0     0     0
	  multipath/DDN-v07  ONLINE       0     0     0
	  multipath/DDN-v08  ONLINE       0     0     0
	  multipath/DDN-v09  ONLINE       0     0     0
	  multipath/DDN-v10  ONLINE       0     0     0
	  multipath/DDN-v11  ONLINE       0     0     0
	logs                 ONLINE       0     0     0
	  mfid1              ONLINE       0     0     0

errors: No known data errors

Results

These results wre obtained from two similar servers. The other server is using a Winchester Systems Storage array, and has 24GB of system memory. The Winchester Storage is ~40TB of 2TB SATA disks:

Another RAID array, just for a comparison

I used IOZone for the test (iozone -a). The default iozone test is using 64k files to 512MB files, and since I’m trying to see how the server might actually react to the real worl, I’m okay with this (ie, I fully understand that a LOT of caching is taking place, and I want that for right now).

Forward Re-Write

Forward Re-Read

Forward Read

Backwards Read

Random Read

Re-Read

Rec? Re-Write

Write

Strided Read

The S2N9900 is a pretty nice device. Although you have to use TELNET (yeesh, couldn’t they spend a few more bucks on a small ARM processor and use ssh?), the controllers have a decent command line environment with HELP pages. What is also nice is the company provides the documentation for their products for free, and no registration is required. Good Job!

As far as raw read and write speeds, that is hard to nail down. I’ve been using IOZone, and when I run that, and take a look at ‘zpool iostat 1′, the ZFS Pool stays at a constant 200MB/sec for writes. I’ve seen in pop up higher, like 250MB to 500MB, but 200 seems to be the ceiling. I’ve done with and without a dedicated log device, with and without gmultipath, and finally, using the SSD RAID1 as a L2ARC cache device. All results are nearly identical. Reads are pretty crazy though, with 64GB of system memory, reading a file is nearly 1GB/sec.

FreeBSD 8.0 = A Great NAS Server

Mike Carlson — Wed, 23 Dec 2009 04:30:23 +0000

I need to share this. When I google for “Samba performance”, I never see real numbers, real configuration files, or real hardware environments. All I read are anecdotal recollections, and that is not good enough. I like numbers, and I’ll let the numbers speak for themselves:

    > netstat -I em0 -w 1
                input          (em0)           output
       packets  errs      bytes    packets  errs      bytes colls
         90166     0   98762637      95363     0    5332847     0
         18131     0   24713156      20042     0    1123684     0
             4     0        310          1     0        178     0
             8     0        518          1     0        178     0
         10153     0   10952920      10696     0     598129     0
         92990     0  102837002      98476     0    5514994     0
         92025     0  102680574      97277     0    5439496     0
         92080     0  101799874      97403     0    5448637     0
         75348     0   90861608      80972     0    4537737     0
         90895     0  100323946      95781     0    5360948     0
         89313     0   97371154      94364     0    5278618     0
         81363     0   89229738      85861     0    4803589     0
             2     0        126          3     0        286     0

I was so shocked that I had to use gstat and zpool iostat to verify the information:

    dT: 1.002s  w: 1.000s  filter: da0
     L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
       35   1476      0      0    0.0   1476 188421   23.7  100.0| da0

    > zpool iostat  1
                   capacity     operations    bandwidth
    pool         used  avail   read  write   read  write
    ----------  -----  -----  -----  -----  -----  -----
    tank        5.68T  4.32T      1     81   250K  10.1M
    tank        5.68T  4.32T      0  1.37K      0   175M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M
    tank        5.68T  4.32T      0  1.44K      0   184M

This is all through Samba (3.3.9), There was no local work being done. I unfortunately didn’t configure MRTG correctly, so it had built a malformed graph while all this happened. Having a picture from all of this would have been nice.

The underlying storage is a SATABoy2 RAID6 array, with a simple “flat” ZFS filesystem (version 13). As cheap as the SATABoy’s are (and come on, they have a terrible IIS web interface), they can at least keep up with the current load.

I have felt that if you are going to use ZFS, you should let it manage the RAID, and not bother with a hardware RAID controller. While the hardware RAID may be faster, ZFS’s ability to self-correct bad blocks is a great feature despite the performance set back. However, RAID6 is pretty good in itself, and having dual parity would ideally reduce the risk of a bad block being detrimental.

One thing I noticed with Samba is it doesn’t seem to be a threaded daemon. When I do a top(1) -H, there are only 2-3 smbd processes, and one of them is running around 30%. Though I don’t really know how well Samba can scale out, this environment only has about 10 users. I would like to see how samba reacts if there are a couple hundred active users. Furthermore, how does a native Windows server handle a couple hundred users? It may handle it a little better, however, I don’t think I would enjoy watching NTFS handling a multi-terabyte volume… it would be like watching a stroke victim eat a bowl of soup. I do admit I am biased and I have no working experience with Windows as a large file server, most of them that I have worked on are horribly limited and underpowered, and no one seems to care if they perform well or not.

Hardware

CPU information

    Machine class:    amd64
    CPU Model:    Dual Core AMD Opteron(tm) Processor 285
    No. of Cores:    4
    Cores per CPU:

RAM information

    Memory information from dmidecode(8)
    Maximum Capacity: 8 GB
    Number Of Devices: 4
    Maximum Capacity: 8 GB
    Number Of Devices: 4

    INFO: Run `dmidecode -t memory` to see further information.

    System memory summary
    Total real memory available:    8048 MB
    Logically used memory:        2876 MB
    Logically available memory:    5172 MB

    Swap information
    Device          1K-blocks     Used    Avail Capacity
    /dev/da1s1b       8373844      28K     8.0G     0%

Storage information

    Available hard drives:
    cd0:  Removable CD-ROM SCSI-0 device
    cd0: 1.000MB/s transfers
    da2:  Fixed Direct Access SCSI-5 device
    da2: 300.000MB/s transfers
    da2: Command Queueing enabled
    da2: 140009MB (286739329 512 byte sectors: 255H 63S/T 17848C)
    da1:  Fixed Direct Access SCSI-2 device
    da1: 300.000MB/s transfers
    da1: Command Queueing enabled
    da1: 69618MB (142577664 512 byte sectors: 255H 63S/T 8875C)
    da0:  Fixed Direct Access SCSI-5 device
    da0: 200.000MB/s transfers
    da0: Command Queueing enabled
    da0: 10491861MB (21487333120 512 byte sectors: 255H 63S/T 1337524C)

    Raid controllers:
    umass-sim0:
    mpt0:
    vendor='LSI Logic (Was: Symbios Logic, NCR)'
    device='SAS 3000 series, 4-port with 1064 -StorPort'
    isp0:
    vendor='QLogic Corporation'
    device='QLA6322 Fibre Channel Adapter'

    Currently mounted filesystems:
    /dev/da1s1a on /
    devfs on /dev
    tank on /tank
    /dev/ufs/EXPORT on /export

    I/O statistics:
           tty             da0              da1              da2             cpu
     tin  tout  KB/t tps  MB/s   KB/t tps  MB/s   KB/t tps  MB/s  us ni sy in id
       0    40 63.61 167 10.36  16.53   2  0.03  61.65   0  0.00   1  0  4  0 94
    INFO: Run iostat(8) or gstat(8) to see live statistics.

    Disk usage:
    Filesystem         Size    Used   Avail Capacity  Mounted on
    /dev/da1s1a         58G    3.4G     50G     6%    /
    devfs              1.0K    1.0K      0B   100%    /dev
    tank               9.8T    5.7T    4.1T    58%    /tank
    /dev/ufs/EXPORT    126G    148K    116G     0%    /export

Software

FreeBSD 8.0-RELEASE-p1 FreeBSD 8.0-RELEASE-p1 amd64
samba-3.3.9 A free SMB and CIFS client and server for UNIX

Samba 3.3.9 Compile-Time Config

> make showconfig
===> The following configuration options are available for samba-3.3.9:
     LDAP=on "With LDAP support"
     ADS=on "With Active Directory support"
     CUPS=off "With CUPS printing support"
     WINBIND=on "With WinBIND support"
     SWAT=off "With SWAT WebGUI"
     ACL_SUPPORT=on "With ACL support"
     AIO_SUPPORT=on "With Asyncronous IO support"
     FAM_SUPPORT=on "With File Alteration Monitor"
     SYSLOG=on "With Syslog support"
     QUOTAS=on "With Disk quota support"
     UTMP=off "With UTMP accounting support"
     PAM_SMBPASS=on "With PAM authentication vs passdb backends"
     DNSUPDATE=off "With dynamic DNS update(require ADS)"
     DNSSD=off "With DNS service discovery support"
     EXP_MODULES=on "With experimental modules"
     POPT=on "With system-wide POPT library"
     MAX_DEBUG=off "With maximum debugging"
     SMBTORTURE=off "With smbtorture"
===> Use 'make config' to modify these settings

System Tuning

The Kernel

I enabled device polling, and took out debugging in the kernel (Sanders, get it! Mmm, I’m hungry…)

diff /usr/src/sys/amd64/conf/GENERIC /usr/src/sys/amd64/conf/SANDERS
    33d32
    < makeoptions    DEBUG=-g        # Build kernel with gdb(1) debug symbols
    78c77
    <
    ---
    > options        DEVICE_POLLING

/boot/loader.conf

    ispfw_load="YES"
    kern.hz="2000"
    aio_load="YES"

/etc/sysctl.conf

    kern.coredump=0
    security.bsd.see_other_uids=0
    security.bsd.see_other_gids=0
    kern.ipc.maxsockbuf=16777216
    kern.ipc.nmbclusters=32768
    kern.ipc.somaxconn=32768
    kern.maxfiles=65536
    kern.maxfilesperproc=32768
    kern.maxvnodes=800000
    net.inet.tcp.delayed_ack=0
    net.inet.tcp.inflight.enable=0
    net.inet.tcp.path_mtu_discovery=0
    net.inet.tcp.recvbuf_auto=1
    net.inet.tcp.recvbuf_inc=524288
    net.inet.tcp.recvbuf_max=16777216
    net.inet.tcp.recvspace=65536
    net.inet.tcp.sendbuf_auto=1
    net.inet.tcp.sendbuf_inc=524288
    net.inet.tcp.sendspace=65536
    net.inet.udp.maxdgram=57344
    net.inet.udp.recvspace=65536
    net.local.stream.recvspace=65536
    net.inet.tcp.sendbuf_max=16777216
    net.inet.tcp.mssdflt=9142

rc.conf (em0 flags)

I want to thank Zilla (see post comments) for the sysctl.conf help.

    ifconfig_em0="inet xxx.xxx.xxx.xxx  netmask 255.255.255.0 polling tso mtu 9194"

smb.conf

        min receivefile size = 131072
        aio read size = 1
        aio write size = 1
        use sendfile = yes
        lock directory = /var/run/samba/
        keepalive = 300

I’m also using LDAP users and group. I wasn’t sure if there would be a noticible performance hit for local users or LDAP users. There doesn’t seem to be one.

We use Active Directory, and since Quest/Vintela still won’t make a FreeBSD client for the Quest Authentication Servers ( a sales rep once told me “There are just too many versions of BSD…”) , I have to use all the open source utilities like OpenSSL, OpenLDAP Client and Kerberos. I don’t mind having to do it, but it is always nice if you can maintain one standard process across ALL systems, and we have a lot more Linux and Solaris systems than FreeBSD. I’m the odd one.

That aside, I use the latest OpenSSL in FreeBSD 8.0, OpenLDAP 2.4.20, and the built-in version of Heimdal Kerberos.

I get similar performance form NFS, however, most desktop users have are either on a Windows or OS X, and CIFS seems to be the unifying network storage protocol.

One thing I have yet to really figure out is configuring Samba to use proper NT ACL’s. However, if you can live with UNIX style permissions, a setup like this is pretty good at serving out lots and lots of data. Maybe that will be next.

ZFS updated in FreeBSD 7.2!

Mike Carlson — Sat, 18 Jul 2009 21:11:47 +0000

FreeBSD 7.x has been using version 6 of ZFS, and originally only 8.0 was going to have the newly updated ZFS version: 13.

Last week the core team MFC’d (Merge From Current) the ZFS updates to 7.2, so I cvsup’d and re-build my server’s kernel and world ( with a simple “make buildworld && make buildkernel && make installworld && make installkernel ), rebooted, and now I have the latest ZFS version running:

[root@server ~]> zpool upgrade -v
This system is currently running ZFS pool version 13.

The following versions are supported:

VER  DESCRIPTION
---  --------------------------------------------------------
 1   Initial ZFS version
 2   Ditto blocks (replicated metadata)
 3   Hot spares and double parity RAID-Z
 4   zpool history
 5   Compression using the gzip algorithm
 6   bootfs pool property
 7   Separate intent log devices
 8   Delegated administration
 9   refquota and refreservation properties
 10  Cache devices
 11  Improved scrub performance
 12  Snapshot properties
 13  snapused property
For more information on a particular version, including supported releases, see:

http://www.opensolaris.org/os/community/zfs/version/N

Where 'N' is the version number.

FreeBSD 7.0, ZFS and iSCSI

Mike Carlson — Thu, 26 Jun 2008 01:19:50 +0000

A friend of mine got my the coolest birthday present I think I’ve ever recieved, and that was a bunch of new and super kick ass hardware. This will soon replace my current server, which is in such bad shape it cannot compile java code, or perl from source. Before I replace it, I wanted to play around with ZFS that comes with FreeBSD 7.0.

Here is a quick rundown of it all:

Intel Core 2 Duo E6750 (2.66Mhz 4MB cache)
Intel S975XBX2 workstation motherboard
AMCC 3Ware 9650SE 4 port SATA RAID controller (4x PCI-e)
- Batter backup for the 3Ware so I can enable cached writes
2GB ECC Crucial Memory Kit
750 Watt PC Power & Cooling power supply
ASUS EN6200 LE 16x PCI-e nVidia GFX card
Plextor DVD+RW PX-810SA SATA
4 Western Digital 1TB Drives

All of this was of the highest quality, and Chris said since he got my into FreeBSD, he felt I should have a stable and rock solid system since my current “server” has died 6 times. So after getting it all put together and powering it up… it wouldn’t post. I swapped the cpu with an older Pentium 4D that I’ve had lying around until my htpc comes back up and that worked. It turns out, the motherboard doesn’t support the 1333Mhz bus speed of the E6750 Core 2 Duo. So I’ve done some testing with the p4 as the cpu, and most of the tests were I/O bound and not CPU.

First off was to test out ZFS. Since it is new to FreeBSD (new in general really) I followed a ZFS tuning guide for 7.0 and followed some pretty stock directions:

$ zpool create tank raidz da0 da1 da2 da3

Which automatically mounted a 2.7TB filesystem. This was a lot nicer than fooling around with partitioning and filesystem tools. I also like the feature set of ZFS compared to standard RAID’s like self healing and data checksums. Performance was a little slower. Doing a simple ‘dd’ with 1mb block size showed about 101MB/sec

 dd if=/dev/zero of=/tank/1gb.dat bs=1m count=1000

1000+0 records in

1000+0 records out

1048576000 bytes transferred in 10.420047 secs (100630640 bytes/sec)

Not bad, and since my GigE network cannot saturate that type of I/O I’m pretty satisfied with those results. I use this system as a network file server, among the other network related services like NAT, www, and mail, so my biggest concern was getting more than 80MB/sec (around gigabit ethernet’s limit).

BUUUT I am underutilizing my fancy 3Ware raid controller, so I can’t just leave that alone. I blew away the ZFS volume and created a full RAID5 with the controller. Everything was fine with the exception of fdisk, which isn’t usable for large volumes. I emailed 3Ware’s support wondering why my 2.78TB volume was only being partitioned at 722GB. They quickly responded with ‘use gpt’, which I did:

$ gpt create /dev/da0

$ gpt add -t ufs /dev/da0

/dev/da0p1 added

$ gpt show /dev/da0

	       start        size  index  contents

	           0           1         PMBR

	           1           1         Pri GPT header

	           2          32         Pri GPT table

	          34  5859311549      1  GPT part - FreeBSD UFS/UFS2

	  5859311583          32         Sec GPT table

	  5859311615           1         Sec GPT header

I did a newfs -U -O2 and mounted the new /SafeKeg. Another dd test showed higher numbers, about 150MB/sec, and with the tw_cli tool installed I can manage the 3Ware card from FreeBSD itself. Very cool. I did play with iSCSI, exporting a 50GB file to my Windows desktop, but the performance was incredibly slow, like 1-2MB/sec. Terrible! I’m not sure if Free/NetBSD’s iscsi-target is at fault, or Window’s iscs initiator, or if iscsi is just not all that up to snuff. I thought it would be nice to utilize, but I’d like to expierement with it more to see exactly how much performance I can squeeze out of it.

UPDATE (10/09/2008):

After getting a few comments on my poor iscsi performance, I have patched to the latest version and I have been MUCH happier with the results. On my little home network I’ve gotten around 40MB/sec for reads and writes.

My Wushu Blog » ZFS

Bacula in the Enterprise – Part 2

Software

The Structure

Storage Nodes

Client Generation

The Client TEMPLATE File

The Create Client Script

Pre-Job command – Package List

Bacula in the Enterprise – Part 1

Background

The Initial Concepts

The Hardware

FreeBSD and Multipath

Hardware

CPU

Machine class: amd64 CPU Model: Intel(R) Xeon(R) CPU E5530 @ 2.40GHz No. of Cores: 16

Memory Total real memory available: 65511 MB Logically used memory: 3945 MB Logically available memory: 61565 MB

Storage

Results

FreeBSD 8.0 = A Great NAS Server

Hardware

CPU information

RAM information

Storage information

Software

Samba 3.3.9 Compile-Time Config

System Tuning

The Kernel

/boot/loader.conf

/etc/sysctl.conf

rc.conf (em0 flags)

smb.conf

ZFS updated in FreeBSD 7.2!

FreeBSD 7.0, ZFS and iSCSI

Memory

Total real memory available: 65511 MB Logically used memory: 3945 MB Logically available memory: 61565 MB