My Wushu Blog » Backups

Bacula in the Enterprise – Part 2

Mike Carlson — Sat, 23 Jul 2011 19:10:26 +0000

Software

As mentioned many times, this is a FreeBSD based environment. Some good sysinfo output below:

Operating system release: FreeBSD 8.2-RELEASE
OS architecture: amd64
Kernel build dir location: /usr/obj/usr/src/sys/GENERIC
Currently booted kernel: /boot/kernel/kernel

Currently loaded kernel modules (kldstat(8)):
zfs.ko
opensolaris.ko

Bootloader settings for the Director/Database node:

The /boot/loader.conf has the following contents:
kern.ipc.semmni=1024
kern.ipc.semmns=2048
kern.ipc.semmnu=1024

All of the storage nodes and the director are running a GENERIC kernel with very few system tweaking. One of the storage nodes has a Chelsio 10Gb controller, but that hasn’t had a high enough load to crack the 1Gb/sec barrier.

I’m using Bacula from the ports tree, and the directory has a special Make flag to build with gcc’s debugging symbols. Jenny worked on getting that setup when we were having some stability issues.

The Bacula configuration one the director node is backed by a git repository. It adds a little bit of complexity for a systems administrator, when they want to add a client, but the benefit is clear. This backup project actually enforces change control and tracks all of the commits by who.

I’ve also setup Redmine as a project front-end, and I’ve begun to file tickets and reference what commit fixed what. This not only tracks my progress, but it is the first time I’ve had a backup server that was clearly documented and had some type of accountability.

A snippet of the Redmine site

The Structure

I’ve compared projects like bacula to a large box of LegosTM. It doesn’t enforce a structure by any means, and I’ve taken it upon myself to add meaning to the otherwise flat and incomprehensible bacula-dir.conf

The Bacula Port on FreeBSD installs all configuration files in /usr/local/etc.

Write, the Director, only contains the following in /usr/local/etc/bacula-dir.conf:

@/usr/local/etc/bacula/bacula-dir.conf
@/usr/local/etc/bacula/storage.conf
@/usr/local/etc/bacula/clients.conf
@/usr/local/etc/bacula/messages.conf
@/usr/local/etc/bacula/schedules.conf
@/usr/local/etc/bacula/pools.conf

As you can see, I place everything in etc/bacula/.

Here is a beautiful output of tree(1):

bacula
|-- bacula-dir.conf
|-- bin
|   |-- create_client.sh
|   `-- package_list.sh
|-- clients.conf
|-- clients.d
|   |-- 10am
|   |-- 10pm
|   |-- 11pm
|   |-- 12am
|   |-- 1am
|   |-- 2am
|   |-- 3am
|   |-- 4am
|   |-- 4pm
|   |-- 5am
|   |-- 5pm
|   |-- 6am
|   |-- 6pm
|   |-- 7am
|   |-- 7pm
|   |-- 8am
|   |-- 8pm
|   |-- 9am
|   |-- 9pm
|   |-- TEMPLATE-mac
|   |-- TEMPLATE-unix
|   `-- TEMPLATE-win32
|-- excludes.d
|   |-- common.conf
|   |-- mac.conf
|   |-- unix.conf
|   `-- win32.conf
|-- messages.conf
|-- pools.conf
|-- schedules.conf
|-- storage.conf
`-- storage.d
    |-- write-01.conf
    |-- write-02.conf
    |-- write-03.conf
    |-- write-04.conf
    |-- write-05.conf
    `-- write-06.conf

Storage Nodes

All of the storage nodes are using ZFS as the filesystem/Volume manager.

write-06# zpool list
NAME         SIZE   USED  AVAIL    CAP  HEALTH  ALTROOT
filevol001  90.6T  33.3T  57.3T    36%  ONLINE  -

They all have one volume, /filevol001, and I created 512 “drives” within that volume. Effectivly, each storage node has 512 drives, and clients are randomly assigned a drive.

Since I have 6 storage nodes, I wrote a little shell script to handle the directory creation:

#!/bin/bash
i=1
 
while [ $i -le 512 ]
do
        install -d -o bacula -g bacula -m 770 /filevol001/drive$i
        ((i++))
done

Simple, right? I also wrote a script to generate the bacula-sd.conf file on a storage node as well:

#!/bin/bash
 
usage()
{
cat << EOF
    Usage $0 NUMBER > /usr/local/etc/bacula-sd.conf
 
    Where "NUMBER" is just a single digit indicating which storage node this is.
 
    Example, for write-07:
    $ make_sd.sh 7 > /usr/local/etc/bacula-sd.conf
EOF
}
 
i=1
 
if [[ -z $1 ]]
then
    usage
    exit
fi
 
printf "Storage {\n"
printf "\tName = write-0$1.llnl.gov-sd\n"
printf "\tSDAddress = write-0$1.llnl.gov\n"
printf "\tSDPort = 9103\n"
printf "\tWorkingDirectory = \"/var/db/bacula\"\n"
printf "\tPid Directory = \"/var/run\"\n"
printf "\tMaximum Concurrent Jobs = 516\n"
printf "}\n"
 
printf "#\n"
printf "# List Directors who are permitted to contact Storage daemon\n"
printf "#\n"
printf "Director {\n"
printf "\tName = write.llnl.gov-dir\n"
printf "\tPassword = \"ItsASecret\"\n"
printf "}\n"
 
printf "#\n"
printf "# Restricted Director, used by tray-monitor to get the\n"
printf "#   status of the storage daemon\n"
printf "#\n"
printf "Director {\n"
printf "\tName = write.llnl.gov-mon\n"
printf "\tPassword = \"ItsANotherSecret\"\n"
printf "\tMonitor = yes\n"
printf "}\n"
 
printf "Messages {\n"
printf "\tName = Standard\n"
printf "\tdirector = write.llnl.gov-dir = all\n"
printf "}\n"
 
 
printf "Device {\n"
printf "\tName = W0$1FileStorage\n"
printf "\tMedia Type = File\n"
printf "\tArchive Device = /filevol001\n"
printf "\tLabelMedia = yes;\n"
printf "\tRandom Access = Yes;\n"
printf "\tAutomaticMount = yes;\n"
printf "\tRemovableMedia = no;\n"
printf "\tAlwaysOpen = no;\n"
printf "\tMaximum Concurrent Jobs = 2\n"
printf "}\n"
 
while [ $i -le 512 ]
do
        printf "\n"
        printf "Device {\n"
        printf "\tName = W0$1FileStorageD$i\n"
        printf "\tMedia Type = File\n"
        printf "\tArchive Device = /filevol001/drive$i\n"
        printf "\tLabelMedia = yes;\n"
        printf "\tRandom Access = Yes;\n"
        printf "\tAutomaticMount = yes;\n"
        printf "\tRemovableMedia = no;\n"
        printf "\tAlwaysOpen = no;\n"
        printf "\tMaximum Concurrent Jobs = 2\n"
        printf "}\n"
        ((i++))
done

On the Directory, a storage node definition is saved in /usr/local/etc/bacula/storage.d/write-0{N}.conf, which is included in /usr/local/etc/bacula/storage.conf:

@/usr/local/etc/bacula/storage.d/write-01.conf
@/usr/local/etc/bacula/storage.d/write-02.conf
@/usr/local/etc/bacula/storage.d/write-03.conf
@/usr/local/etc/bacula/storage.d/write-04.conf
@/usr/local/etc/bacula/storage.d/write-05.conf
@/usr/local/etc/bacula/storage.d/write-06.conf

Client Generation

There are two components, the TEMPLATE file (there are three, TEMPLATE-unix, TEMPLATE-win32 and TEMPATE-mac) and the shell script.

The Client TEMPLATE File

Here is what one of the TEMPLATE files looks like:

#
# Client Definition, the Password here must match
#  the clients bacula-fd.conf Client definition.
#
# Using Vi/m, you can easily replaced HOSTNAME with
#  the short hostname of the client with:
#  %s/HOSTNAME/yourhostname/
#
#

Client {
    Name = HOSTNAME.llnl.gov
    Address = HOSTNAME.llnl.gov
    FDPort = 9102
    Catalog = Catalog001
    Password = "ItsASecret"
    File Retention = 40 days
    Job Retention = 1 months
    AutoPrune = yes
    Maximum Concurrent Jobs = 10
    Heartbeat Interval = 300
}

Console {
    Name = HOSTNAME.llnl.gov-acl
    Password = ItsASecret
    JobACL = "HOSTNAME.llnl.gov RestoreFiles", "HOSTNAME.llnl.gov"
    ScheduleACL = *all*
    ClientACL = HOSTNAME.llnl.gov
    FileSetACL = "HOSTNAME.llnl.gov FileSet"
    CatalogACL = Catalog001
    CommandACL = *all*
    StorageACL = *all*
    PoolACL = HOSTNAME.llnl.gov-File
}

Job {
    Name = "HOSTNAME.llnl.gov"
    Type = Backup
    Level = Incremental
    FileSet = "HOSTNAME.llnl.gov FileSet"
    Client = "HOSTNAME.llnl.gov"
    Storage = FileStorageD##
    Pool = HOSTNAME.llnl.gov-File
    Schedule = "@@"
    Messages = Standard
    Priority = 10
    Write Bootstrap = "/var/db/bacula/%c.bsr"
    Maximum Concurrent Jobs = 10
    Reschedule On Error = yes
    Reschedule Interval = 1 hour
    Reschedule Times = 1
    Max Wait Time = 30 minutes
    Cancel Lower Level Duplicates = yes
    Allow Duplicate Jobs = no
    RunScript {
        RunsWhen = Before
        FailJobOnError = no
        Command = "/etc/scripts/package_list.sh"
        RunsOnClient = yes
    }
}

Pool {
    Name = HOSTNAME.llnl.gov-File
    Pool Type = Backup
    Recycle = yes
    AutoPrune = yes
    Volume Retention = 1 months
    Maximum Volume Bytes = 10G
    Maximum Volumes = 100
    LabelFormat = "HOSTNAME.llnl.govFileVol"
    Maximum Volume Jobs = 5
}

Job {
    Name = "HOSTNAME.llnl.gov RestoreFiles"
    Type = Restore
    Client= HOSTNAME.llnl.gov
    FileSet="HOSTNAME.llnl.gov FileSet"
    Storage = FileStorageD##
    Pool = HOSTNAME.llnl.gov-File
    Messages = Standard
    #Where = /tmp/bacula-restores
}

FileSet {
    Name = "HOSTNAME.llnl.gov FileSet"
    Include {
        Options {
            signature = MD5
            compression = GZIP6
                        fstype = ext2
                        fstype = xfs
                        fstype = jfs
                        fstype = ufs
                        fstype = zfs
                        onefs = no
                        Exclude = yes
                        @/usr/local/etc/bacula/excludes.d/common.conf
        }
                File = /
                File = /usr/local
                Exclude Dir Containing = .excludeme
    }
    Exclude {
        @/usr/local/etc/bacula/excludes.d/unix.conf
    }
}

The Create Client Script

So here is what really makes creating clients easy for us, the create_client script.

I didn’t want to do it this way, really, so part of me is very ashamed of this tool. I would have preferred to re-write this in Python, or make a web page out of it, and let admins create clients from their desktop. Or, I would have loved to create a puppet module to handle this automagically (but that would exlcude everything that *isn’t* running Puppet, which is huge).

With that disclaimer, here is my create_client shell script:

#!/usr/bin/env bash
# usage: cclient -t unix -s 12am -h hostname
#
 
umask 022
 
# Variables
## Randomize Schedule
SCHEDULES="4pm 5pm 6pm 7pm 8pm 9pm 10pm 11pm 12am 1am 2am 3am 4am 5am 6am 7am 8am 9am 10am"
s=($SCHEDULES)
num_s=${#s[*]}
RAND_SCHED=${s[$((RANDOM%num_s))]}
# Randomize which storage node we use
NODES="write-06 write-01 write-06 write-01 write-02 write-03 write-04 write-05"
n=($NODES)
num_n=${#n[*]}
RAND_NODE=${n[$((RANDOM%num_n))]}
 
export DRIVE=`jot -r 1 1 512`
export BDIR="/usr/local/etc/bacula"
export TYPE="unix"
export SCHEDULE=$RAND_SCHED
export HOSTNAME=""
export STORAGE_NODE=$RAND_NODE
export GIT_DIR="/usr/local/etc/bacula/.git"
export CLASS="desktop"
 
if [ $(whoami) == "root" ]
then
cat << EOF
                Please do not run this as root. This script runs a
                git add/commit, which is how changes are managed and
                tracked. If you run this as root, then it shows up
                as carlson39 or root.
 
                If you encounter a problem with your normal OUN account,
                please contact Mike Carlson, or submit a bug here:
                https://st-scm.llnl.gov/redmine/snt/projects/bacula/issues/new
EOF
 
exit 1
fi
 
usage()
{
cat << EOF
 
        Usage: $0 [OPTION]... -h HOSTNAME
 
        This script will generate a bacula client definition.
 
        OPTIONS:
        -s      schedule, (4pm|5pm|6pm|7pm|8pm|9pm|10pm|11pm|12am|1am|2am|3am|4am|5am|6am|7am|8am|9am). The default schedule is random.
        -t      type, (unix|win32|mac), unix is the default
        -n      storage node (write-01|write-02|...), the default is random.
        -h      hostname (use the short hostname)
EOF
}
 
cd $BDIR
 
while getopts 'c:t:s:n:h:' OPTION
do
        case $OPTION in
                c)
                        CLASS=$OPTARG
                        ;;
                t)
                        TYPE=$OPTARG
                        ;;
                s)
                        SCHEDULE=$OPTARG
                        ;;
                h)
                        HOSTNAME=$OPTARG
                        echo $HOSTNAME | egrep -q "(llnl.gov|ucllnl.org)"
                        if [ $? -eq 0 ]
                        then
                        HOSTNAME=`echo $HOSTNAME|sed -e 's/.llnl.gov//' -e 's/.ucllnl.org//'`
                        fi
 
                        ;;
                n)
                        STORAGE_NODE=$OPTARG
                        ;;
                ?)
                        usage
                        exit
                        ;;
        esac
done
 
if [[ -z $CLASS ]] || [[ -z $TYPE ]] || [[ -z $SCHEDULE ]] || [[ -z $HOSTNAME ]] || [[ -z $STORAGE_NODE ]]
then
        usage
        exit 1
fi
 
grep -w $HOSTNAME $BDIR/clients.conf
if [ $? -eq 0 ]
then
        echo 'client '$HOSTNAME 'already exists...'
else
        export RETRY_COUNT="2"
 
        if [ $STORAGE_NODE == "write-01" ]
        then
                DRIVE=`jot -r 1 33 512`
                sed -e 's/HOSTNAME/'$HOSTNAME'/g' -e 's/FileStorageD##/FileStorageD'$DRIVE'/' -e 's/\@\@/'$SCHEDULE'/' -e 's/RETRY_COUNT/'$RETRY_COUNT'/g' $BDIR/clients.d/TEMPLATE-$TYPE > $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
                echo \@$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf >> $BDIR/clients.conf
        else
                export SN=`echo $STORAGE_NODE | cut -c 7-8`
                sed -e 's/HOSTNAME/'$HOSTNAME'/g' -e 's/FileStorageD##/W'$SN'FileStorageD'$DRIVE'/' -e 's/\@\@/'$SCHEDULE'/' -e 's/RETRY_COUNT/'$RETRY_COUNT'/g' $BDIR/clients.d/TEMPLATE-$TYPE > $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
                echo \@$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf >> $BDIR/clients.conf
        fi
 
        chgrp st-bacula-admins $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
        git add $BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf $BDIR/clients.conf
        git commit
        echo 'created client definition: '$BDIR/clients.d/$SCHEDULE/$HOSTNAME.conf
        echo 'for '$HOSTNAME'.llnl.gov'
fi

This is always a work in progress, but at the core, it is a simple sed wrapper with a lot of randomization and a git commit.

Why all the randomization?

Because I had to add around 1000 clients in a VERY short amount of time. We didn’t have a problem pushing the Bacula client to all of the platforms, nor the bacula-fd.conf file either. What I could not do was spend the time to create and manage all of the resources for each client. That is why I have so many devices/drives, so I can attempt to have a 1:1 without having to actually think about it.

So, I wrote ANOTHER script to wrap around this one when I need to do bulk client creations. I’m not going to post that, it just loops through the above command.

Pre-Job command – Package List

I only do this on the Unix/Linux clients, and I thought it was a cool idea.

Yeah, I will pat myself on the back a little bit for that :)

I exclude the Operating System from backups for two reasons, 1) to reduce backing up duplicate and reproducible data and 2) Our build/Imaging process is so quick and clean it is just faster to rebuild than restore everything.

Still, I needed a way to keep the state of installed packages/software.

This is where the pre-job command comes in handy. This part right here:

    RunScript {
        RunsWhen = Before
        FailJobOnError = no
        Command = "/etc/scripts/package_list.sh"
        RunsOnClient = yes
    }

That package_list.sh file looks like this:

 
#!/usr/bin/env bash
 
export PLIST="/root/plist.txt"
 
case "`uname -s`" in
Linux)
                if [ -x /usr/bin/lsb_release ]; then
                        DIST=`lsb_release -d`
                fi
 
                # RHEL
                if [ -x /usr/bin/up2date ]; then
                        rpm -qa > $PLIST
                fi
 
                # RHEL 5
                if [ -x /usr/bin/yum ]; then
                        if [ -f /var/run/yum.pid ]; then
                                echo "Yum currently in use, exiting gracefully..."
                                exit 0
                        else
                        /usr/bin/yum list installed | awk '{print $1}' > $PLIST
                        fi
                fi
 
                # Ubuntu
                if [ -x /usr/bin/dpkg ]; then
                        /usr/bin/dpkg --get-selections | awk '{print $1}' > $PLIST
                fi
                ;;
 
FreeBSD)
                pkg_info|awk '{print $1}' > $PLIST
                ;;
SunOS)
                pkginfo |awk '{print $1}' > $PLIST
                ;;
esac

That file, /root/plist.txt, gets backed up.

Now we have a record of what was installed on our Unix platforms :)

That is it for now, see you at Part 3

Bacula in the Enterprise – Part 1

Mike Carlson — Sat, 23 Jul 2011 00:45:21 +0000

I’ve been using Bacula, the open source backup software, for over a year now. Things have been going well, and I would like to dedicate a post or two to the environment I built.

Background

Over a year ago, I took it upon myself to replace a single Legato Networker server with Bacula. One of our collaborators had decided to ship us (for no reason at all really, I think they were cleaning out their data center) a Sun X4200 AMD server, and two StorageTek/Sun NAS servers.

I had no reason for the NAS heads, but the JBOD was full of drives and the Sun X4200 was useful enough. So, I gutted them (Since the StoragTek NAS heads were identical in almost every way to a standard X4200), put as much memory and CPU’s as I could in one system. This was my first Bacula server. It had around 2TB of FC storage and it made a nice replacement backup server for the 50 or so clients that were on the Networker server. The OS was not Solaris, as you might guess since I was using Sun hardware, but FreeBSD.

Since was focusing only on disk based backups, and FreeBSD has two fantastic large file systems (UFS2 and ZFS), this was my underlying storage platform. Combine that with the current choice of software (in the case, Bacula 5.0.x and PostgreSQL 8/9) from the Ports tree, it really makes the perfect open source software and hardware stack.

After spending a good amount of time wrapping my head around Bacula, and really, just carelessly diving into it, I was very happy with how fast and stable it was shaping up to be.

Around the same time, I was asked to pick up the project that literally went no where for years: The dreaded Backup Project for all of the S&T directorate. A mix of all OS’s, desktops, servers, laptops, etc… and around 3000 active machines online with lots of important data.

No small feat, and there are many reasons why this had been a difficult project to wrangle. One thing for sure, is we knew we had a lot of unique programmatic data.

I knew what software I wanted to use, and I was pretty set on using commodity hardware and reasonably priced storage. The next part was to define some constants in the environment.

The Initial Concepts

First up, I knew of the largest painful aspects of our computing environment:

Budget constraints – We are not rich, and IT always seems to be underfunded. This project had to be as frugal as possible, yet still deliver.
Diverse platforms – We have Windows, OS X and a mix of RHEL and Ubuntu for desktops. Server platforms range all across the board: Windows, OSX, RHEL, Solaris, FreeBSD, AIX, etc…
Mission critical data – Lets face it, the Lab doesn’t make a car, or a VCR. We have a LOT of critical scientific knowledge that is only stored in bits and bytes. That is our product, scientific data that is truly unique.
A campus like geography – the Lab is 1 sq. mile with a mixture of trailers, new buildings, and some buildings that are over 50 years old. The network backbone ranges from 10Gbps to 10Mbps. The poses a problem when it comes to backups.

With this in mind, theses are some of the initial concepts I latched onto:

A distributed disk based storage backend for Bacula
Smaller retention window – 1 month
Reduce the amount of data that has to even go over the network

Skip “reproducible data” such has installed programs like Office, and exclude the OS itself
Enable client side compression. The effectively distributes the compression for all clients. Saves a lot of disk space :)

Skip virtual machine images, like vmdk and vmem files, and treat important virtual machines as separate clients.

For Bacula itself, I had decided that each client would have its own resources. Very little is shared between each client, so for example client-001 would get all of these resources:

Pool = “Client-001-File”

Storage = a randomly assigned drive on one of the storage nodes
Maximum Volume bytes = 10Gb
Maximum Volumes = 100
AutoPrune and Recycling enabled

Job = “client-001″ (one for backup and one for restore)
FileSet = “client-001 FileSet”

What this prevents is cross-client data contamination. What is also prevented, and I found out later, was concurrent backups. More on that later though.

After feeling a bit more comfortable with a simple all-in-one server, I was ready to spec out new hardware. This was good, because at the same time it was our end of year budget crunch, and hardware had to be procured.

The Hardware

For storage, I had a few concepts I was floating around.

One, was to use MooseFS, a distributed filesystem , across a bunch of cheap node with a modest amount of storage.

The other, idea was to buy a handful of servers with a lot of internal storage, around 18TB or so. Them distribute them across the “campus” as a kind of Bacula storage node cloud.

The last idea was to take a more traditional backup server approach, and buy a server with as much expandable storage as possible and back everything up to that.

In the end, when a bunch of hardware showed up (I had *some* control over the hardware, but not all aspects since some of the managers took it upon themselves to purchase everything), I scrapped the MooseFS idea after talking it over with Jenny, and we took the last two: 4 HP servers with 17TB or RAID6 internal storage, and a large 140TB SAN array (Winchester Systems, great stuff!) as the primary backup node:

Our Environment

This model served us well for a while. We had a primary storage node that backed up a users primary desktop, and the smaller storage nodes were used to back up servers and infrastructure data.

There was room for improvement right away though. As soon as it was in production, I quickly mapped out what I felt were the next steps in making a robust backup environment.

Using Duplicity

Mike Carlson — Thu, 21 Jul 2011 05:04:40 +0000

A while ago, I posted about how I backup my server with Duplicity to Amazon’s S3 storage.

To follow up, here is a little guide I wrote on using Duplicity in the everyday work environment

Overview

Duplicity is a backup tool that will create compressed and encrypted (uses gnupg) backup archives. It can use a variety of protocols as the target (file, ftp, webdav, imap, ssh/scp, rsync, hsi, s3 and hsi).

Since it is aware of previous backup jobs, incremental backups from that last full, and some basic collection management, it is preferred over simpler tools like rsync.

Since Duplicity uses GnuPG to compress and encrypt the save sets, you’ll need to enter a passphrase. This make duplicity a little difficult to automate, you can do one of two things:

embed the the gnupg passphrase in the backup job, or use a gnupg-agent
disable encryption and just compress the save set

To only compress and not encrypt, use the ”’–no-encryption”’ option.

Installation

Duplicity is available in the EPEL channel, so on RHEL 5 systems just type

yum install duplicity

On Ubuntu, you can install it using aptitude:

aptitude install duplicity

On FreeBSD, you can install it using:

pkg_add -r duplicity

Preparation

You’ll need a location to store your archives. Lets assume it is a locally mounted volume at /backups.

Usage

The syntax is pretty simple:

duplicity [full|incr|collection-status|list-current-files] source target-url

The ‘target-url’ can be one of many protocols:

file:///backups
scp://user@host/backups
webdav://user@backup-server/backups/

Since we are assuming a locally mounted device, we’ll use the ”’file:///”’ url.

Full Backups

duplicity full ~/ file:///backups/

GnuPG passphrase:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: none
Retype passphrase to confirm: 

--------------[ Backup Statistics ]--------------
StartTime 1274121911.36 (Mon May 17 11:45:11 2010)
EndTime 1274126391.50 (Mon May 17 12:59:51 2010)
ElapsedTime 4480.14 (1 hour 14 minutes 40.14 seconds)
SourceFiles 53369
SourceFileSize 43240385480 (40.3 GB)
NewFiles 53369
NewFileSize 43240385480 (40.3 GB)
DeletedFiles 0
ChangedFiles 0
ChangedFileSize 0 (0 bytes)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 53369
RawDeltaSize 43219504708 (40.3 GB)
TotalDestinationSizeChange 37539659285 (35.0 GB)
Errors 0
-------------------------------------------------

Incremental Backup

Incremental backups are easy, just replace ‘full’ with ‘incr’, and here, we also upped the Gzip compression level to ’9′

duplicity incr --gpg-options "-z 9" /home/mcarlson file:///media/0654-E203/backups
GnuPG passphrase:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon May 17 11:45:04 2010
--------------[ Backup Statistics ]--------------
StartTime 1274130893.40 (Mon May 17 14:14:53 2010)
EndTime 1274131032.77 (Mon May 17 14:17:12 2010)
ElapsedTime 139.36 (2 minutes 19.36 seconds)
SourceFiles 53437
SourceFileSize 43628122482 (40.6 GB)
NewFiles 172
NewFileSize 434867245 (415 MB)
DeletedFiles 96
ChangedFiles 58
ChangedFileSize 1175654577 (1.09 GB)
ChangedDeltaSize 0 (0 bytes)
DeltaEntries 326
RawDeltaSize 444524537 (424 MB)
TotalDestinationSizeChange 442395156 (422 MB)
Errors 0
-------------------------------------------------

Restore

You can either restore the entire archive with the ”’restore”’ option, or a set of files with ”’files-to-restore”’:

Lets assume we did something like this:

rm -rf FreeBSD

This was obviously a mistake and we quickly need those files back:

duplicity --file-to-restore src/FreeBSD file:///backups/ FreeBSD
GnuPG passphrase:
Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon May 17 11:45:04 2010

Lets verify that our files are in fact there again:

ls -al FreeBSD
total 24
drwxr-xr-x  6 mcarlson mcarlson 4096 2009-08-31 15:51 .
drwxrwxr-x 65 mcarlson mcarlson 4096 2010-05-17 16:23 ..
drwxr-xr-x  2 mcarlson mcarlson 4096 2009-08-31 15:51 6.4
drwxr-xr-x  2 mcarlson mcarlson 4096 2009-08-31 15:25 7.0
drwxr-xr-x  2 mcarlson mcarlson 4096 2009-08-31 15:50 8.0
drwxr-xr-x  6 mcarlson mcarlson 4096 2009-08-31 15:52 i386

Managing your Archive(s)

Duplicity does not store any policy or retention details, so it is up the the individual to remove older save sets.

Collection Status

First thing you can do is run get a status of your save sets. Duplicity will scan your target-url, in this case ‘file:///backups’ and print a simple report on what backups types have ran, the date it was ran at, and the number of files:

duplicity collection-status file:///media/0654-E203/backups

Local and Remote metadata are synchronized, no sync needed.
Last full backup date: Mon May 17 11:45:04 2010
Collection Status
-----------------
Connecting with backend: LocalBackend
Archive dir: /home/mcarlson/.cache/duplicity/0e6da04424dcfd02a2dae71879be23fa

Found 0 secondary backup chains.

Found primary backup chain with matching signature chain:
-------------------------
Chain start time: Mon May 17 11:45:04 2010
Chain end time: Mon May 17 14:14:48 2010
Number of contained backup sets: 2
Total number of contained volumes: 1447
 Type of backup set:                            Time:      Num volumes:
                Full         Mon May 17 11:45:04 2010              1430
         Incremental         Mon May 17 14:14:48 2010                17
-------------------------
No orphaned or incomplete backup sets found.

Collection Pruning

You can remove older backup archives with ”’remove-older-than”’ or ”’remove-all-but-n-full”’, and you can clean out failed save sets with the ”’cleanup”’ options.

All of these require the target-url as the 2nd argument.

Here, we will remove any archive that is older than 1 month (1M).

duplicity remove-older-than 1M file:///backups/

Last full backup date: Sat May  1 00:20:02 2010
There are backup set(s) at time(s):
Thu Apr  1 00:15:17 2010
Sun Apr  4 00:01:23 2010
Sun Apr 11 00:01:29 2010
Which can't be deleted because newer sets depend on them.
Deleting backup sets at times:
Sun Apr 26 12:15:33 2009
Sun Apr 26 14:45:25 2009
Sun Apr 26 14:58:38 2009
Sun May  3 00:01:37 2009
Sun May 10 00:00:39 2009
Sun May 17 00:00:42 2009
Sun May 24 00:00:53 2009
Sun May 31 00:00:50 2009
Sun Jun  7 00:01:04 2009
Sun Jun 14 00:00:55 2009
Sun Jun 21 00:01:17 2009
Sun Jun 28 00:01:10 2009
Sun Jul  5 00:03:09 2009
Sun Jul 12 00:01:10 2009
Sun Jul 19 00:01:18 2009
Sun Jul 26 00:01:52 2009
Sun Aug  2 00:01:27 2009
Sun Aug  9 00:02:39 2009
Sun Aug 16 00:01:52 2009
Sun Aug 23 00:01:33 2009
Sun Aug 30 00:01:42 2009
Sat Sep  5 17:23:02 2009
Sun Sep  6 00:00:41 2009
Sun Sep 13 00:00:47 2009
Sun Mar  7 00:04:47 2010
Sun Mar 14 00:07:12 2010
Sun Mar 21 00:01:36 2010
Sun Mar 28 00:02:14 2010

Using Amazon’s S3 for Backups

Mike Carlson — Sun, 26 Apr 2009 01:47:31 +0000

I don’t have a backup system for home (which is where this site, and others are located), and I have generally relied on duplicating enough of my important stuff between friends and other computers. That, and I have a RAID5 setup for my large storage, and then home directories and website stuff is on a RAID1 ZFS volume. This doesn’t prevent accidental “oh-no”s, but it does protect me from some hardware failures.

Last year when I upgraded to the new server, I lost a lot of data because I forgot to backup all of my MySQL databases. I like to think I can learn from my mistakes, so a full year later I finally did something about it and signed up for Amazon’s S3 service.

The pricing is pretty nice, and I don’t have all that much data to backup. I figure, I’ll use up a few GB in total, and keep the monthly price around $1 – $2. That seems worth the price for off-site backup’s.

Now, I have 3 main websites that I need to backup, and one test site that I like to play around with:

http://www.m87-blackhole.org/ <- This is the first domain that I owned, and its the site where my family checks out new photos
http://www.willowoakboarding.com/ <- My parent’s site for their boarding ranch. I’m glad they have no concept of a SLA, or that things need to be backed up :)
http://www.mywushublog.com/ <- This site of course, where I claim my own identity on the internet
http://www.evil-genius-network.com/ <- a test domain, but now I run a little OpenID service, For one…

After a quick “FreeBSD s3 backup” Google search, I found Gary Dalton’s blog post: http://dvector.com/oracle/2008/10/18/backing-up-to-amazon-s3/. After reading this post, I formulated my plan of attack:

Sign up for S3, create a “bucket” for each site
Use something to interface with S3 ( duplicity )
Automate MySQL and PostgreSQL backups
Create a service account to run both s3 and db backup scripts as
Set up a cron job for backups

So, after I signed up for S3, I had to create the buckets. I couldn’t find a way to do this though my Amazon account settings, so I created a little ruby script.
$ sudo gem install aws-s3 $ vim make-bucket.rb

#!/usr/local/bin/ruby
 
require 'aws/s3'
 
AWS::S3::Base.establish_connection!(
:access_key_id     => 'my-s3-key-id',
:secret_access_key => 'my-s3-secret-access-key'
)
AWS::S3::Bucket.create('mywushublog')
AWS::S3::Bucket.create('willowoak')
AWS::S3::Bucket.create('m87-blackhole')
AWS::S3::Bucket.create('evil-genius-network')

$ ./make-bucket.rb
Next, I had to install duplicity and py-boto
[root@server ~] cd /usr/ports/sysutils/duplicity [root@server duplicity] make install ... [root@server duplicity] cd ../../devel/py-boto [root@server py-boto] make install clean ... [root@server py-boto]
Next step, create a user (with access to shared data, and website data) to run the backups with the adduser command…
[root@server py-boto] adduser -g shared-data -G www -s /bin/tcsh -w random s3backupuser ... [roott@server py-boto] su - s3backupuser In tcsh, you can `set autolist' to have the shell automatically show all the possible matches when doing filename/directory expansion. %
I’ll have to set my Access ID and Access Key in the s3backupuser’s environment, as well as a GnuPG passphrase so the backups are encrypted (and compressed). I mean, I trust Amazon, but not THAT much :)
% vim .cshrc setenv AWS_ACCESS_KEY_ID my-s3-key-id setenv AWS_SECRET_ACCESS_KEY my-s3-secrect-access-key setenv PASSPRASE AVeryRandonPassphraseForGnuPG
Next, I copied the very useful automysqlbackup.sh script into a separate script for each website. I could have just dumped every database that was running, but I wanted to segregate each site’s databases into a different directory. So, I’m complicating my cron job by running multiple backup scripts, but I really want to make the end result easily readable and identifiable by me. So for each site, I create a directoy under /u01/backups:
%ll /u01/backups/ total 8 drwxr-x--- 5 s3-backupuser mysql 5 Apr 25 15:46 evil-genius-network drwxr-x--- 5 s3-backupuser mysql 5 Apr 25 15:47 m87-blackhole drwxr-x--- 5 s3-backupuser mysql 5 Apr 25 15:46 mywushublog drwxr-x--- 5 s3-backupuser mysql 5 Apr 25 15:47 willowoak
Next was the s3-backups.sh script, which is very crude and simple. If I’m really motivated, I’ll make it nicer but I’m lazy and if I don’t need anymore functionality then I’ll just leave it. One thing I initially forgot was that I set my Amazon S3 variables in the users .cshrc profile. This is not a good place to have those things, it was just handy as I was running the duplicity commands manually. So I had to add those in, otherwise the cron job would fail.

~/bin/s3-backups.sh:

#!/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/s3/bin
 
# Amazon S3 keys, and GnuPG keys
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
PASSPHRASE=
export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY
export PASSPHRASE
 
echo "*************************************************"
echo "*   Backing up Website content....              *"
echo "*                                               *"
echo "*     www.willowoakboarding.com...              *"
duplicity /www/www.willowoakboarding.com s3+http://s3.amazon.com/willowoak/www
echo "*     www.mywushublog.com...                    *"
duplicity /www/www.mywushublog.com s3+http://s3.amazon.com/mywushublog/www
echo "*     www.m87-blackhole.org...                  *"
duplicity /www/www.m87-blackhole.org s3+http://s3.amazon.com/m87-blackhole/www
echo "*************************************************"
echo "*   Backing up databases....                    *"
echo "*                                               *"
echo "*     www.willowoakboard.com...                 *"
duplicity /u01/backups/willowoak s3+http://s3.amazon.com/willowoak/db
echo "*     www.mywushublog.com...                    *"
duplicity /u01/backups/mywushublog s3+http://s3.amazon.com/mywushublog/db
echo "*     www.m87-blackhole.org...                  *"
duplicity /u01/backups/m87-blackhole s3+http://s3.amazon.com/m87-blackhole/db
echo "*************************************************"

And last but not least, a cronjob to tie it all together:
% crontab -e @weekly ~/bin/s3-backups.sh @weekly ~/bin/mywushublog-mysql-backup.sh @weekly ~/bin/willowoak-mysql-backup.sh @weekly ~/bin/m87-blackhole-mysql-backup.sh @weekly ~/bin/evil-genius-network-mysql-backup.sh
I can check the status of a backup by running duplicity with the ‘collection-status‘ flag:
%duplicity collection-status s3+http://s3.amazon.com/mywushublog/db Last full backup date: Sat Apr 25 15:08:02 2009 Collection Status ----------------- Connecting with backend: BotoBackend Archive dir: None Found 0 backup chains without signatures. Found a complete backup chain with matching signature chain: ------------------------- Chain start time: Sat Apr 25 15:08:02 2009 Chain end time: Sat Apr 25 15:08:02 2009 Number of contained backup sets: 1 Total number of contained volumes: 1 Type of backup set: Time: Num volumes: Full Sat Apr 25 15:08:02 2009 1 ------------------------- No orphaned or incomplete backup sets found.
I can also list the files:
%duplicity list-current-files s3+http://s3.amazon.com/mywushublog/db Last full backup date: Sat Apr 25 15:08:02 2009 Sat Apr 25 15:05:11 2009 . Sat Apr 25 15:05:10 2009 daily Sat Apr 25 15:05:10 2009 daily/mywushublog Sat Apr 25 15:05:10 2009 monthly Sat Apr 25 15:05:10 2009 weekly Sat Apr 25 15:05:11 2009 weekly/mywushublog Sat Apr 25 15:05:11 2009 weekly/mywushublog/mywushublog_week.17.2009-04-25_15h05m.sql.gz
Pretty sweet automated backup process. It is a lot cheaper than tapes or additional disk storage. With S3, I also don’t have to worry about buying additional hardware, the maintenance of a library or tape drive (which is what I had a few years ago, what a headache).