My Wushu Blog

Freddy, what I am saying is: know your limitations. You are a moron.

This movie was complete shit… Ha, just kidding Jenny

No, really, it was hilarious. There is a lot of quick come backs and one liners that you really have to pay attention to catch. At the risk of sounding (even more) like an old man, they don’t make comedies like they used to. Really, you don’t see this kind of a movie now, this is the kind of comedy I like, way more than what Judd Apatow and the like pump out. It’s classy, yet its about coning people which isn’t classy at all.

Now, back to chasing kids off my lawn….

mike Movies Jenny

I don’t have a backup system for home (which is where this site, and others are located), and I have generally relied on duplicating enough of my important stuff between friends and other computers. That, and I have a RAID5 setup for my large storage, and then home directories and website stuff is on a RAID1 ZFS volume. This doesn’t prevent accidental “oh-no”s, but it does protect me from some hardware failures.

Last year when I upgraded to the new server, I lost a lot of data because I forgot to backup all of my MySQL databases. I like to think I can learn from my mistakes, so a full year later I finally did something about it and signed up for Amazon’s S3 service.

The pricing is pretty nice, and I don’t have all that much data to backup. I figure, I’ll use up a few GB in total, and keep the monthly price around $1 – $2. That seems worth the price for off-site backup’s.

Now, I have 3 main websites that I need to backup, and one test site that I like to play around with:

• http://www.m87-blackhole.org/ <- This is the first domain that I owned, and its the site where my family checks out new photos
• http://www.willowoakboarding.com/ <- My parent’s site for their boarding ranch. I’m glad they have no concept of a SLA, or that things need to be backed up
• http://www.mywushublog.com/ <- This site of course, where I claim my own identity on the internet
• http://www.evil-genius-network.com/ <- a test domain, but now I run a little OpenID service, For one…

After a quick “FreeBSD s3 backup” Google search, I found Gary Dalton’s blog post: http://dvector.com/oracle/2008/10/18/backing-up-to-amazon-s3/. After reading this post, I formulated my plan of attack:

• Sign up for S3, create a “bucket” for each site
• Use something to interface with S3 ( duplicity )
• Automate MySQL and PostgreSQL backups
• Create a service account to run both s3 and db backup scripts as
• Set up a cron job for backups

So, after I signed up for S3, I had to create the buckets. I couldn’t find a way to do this though my Amazon account settings, so I created a little ruby script.

$ sudo gem install aws-s3
$ vim make-bucket.rb


#!/usr/local/bin/ruby
 
require 'aws/s3'
 
AWS::S3::Base.establish_connection!(
:access_key_id     => 'my-s3-key-id',
:secret_access_key => 'my-s3-secret-access-key'
)
AWS::S3::Bucket.create('mywushublog')
AWS::S3::Bucket.create('willowoak')
AWS::S3::Bucket.create('m87-blackhole')
AWS::S3::Bucket.create('evil-genius-network')

$ ./make-bucket.rb
Next, I had to install duplicity and py-boto
[root@server ~] cd /usr/ports/sysutils/duplicity
[root@server duplicity] make install
...
[root@server duplicity] cd ../../devel/py-boto
[root@server py-boto] make install clean
...
[root@server py-boto]
Next step, create a user (with access to shared data, and website data) to run the backups with the adduser command…
[root@server py-boto] adduser -g shared-data -G www -s /bin/tcsh -w random s3backupuser
...
[roott@server py-boto] su - s3backupuser
In tcsh, you can `set autolist' to have the shell automatically show
all the possible matches when doing filename/directory expansion.
%
I’ll have to set my Access ID and Access Key in the s3backupuser’s environment, as well as a GnuPG passphrase so the backups are encrypted (and compressed). I mean, I trust Amazon, but not THAT much
% vim .cshrc
setenv AWS_ACCESS_KEY_ID my-s3-key-id
setenv AWS_SECRET_ACCESS_KEY my-s3-secrect-access-key
setenv PASSPRASE AVeryRandonPassphraseForGnuPG
Next, I copied the very useful automysqlbackup.sh script into a separate script for each website. I could have just dumped every database that was running, but I wanted to segregate each site’s databases into a different directory. So, I’m complicating my cron job by running multiple backup scripts, but I really want to make the end result easily readable and identifiable by me. So for each site, I create a directoy under /u01/backups:
%ll /u01/backups/
total 8
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:46 evil-genius-network
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:47 m87-blackhole
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:46 mywushublog
drwxr-x---  5 s3-backupuser  mysql  5 Apr 25 15:47 willowoak
Next was the s3-backups.sh script, which is very crude and simple. If I’m really motivated, I’ll make it nicer but I’m lazy and if I don’t need anymore functionality then I’ll just leave it. One thing I initially forgot was that I set my Amazon S3 variables in the users .cshrc profile. This is not a good place to have those things, it was just handy as I was running the duplicity commands manually. So I had to add those in, otherwise the cron job would fail.

~/bin/s3-backups.sh:

#!/bin/sh
PATH=/sbin:/bin:/usr/sbin:/usr/bin:/usr/games:/usr/local/sbin:/usr/local/bin:/home/s3/bin
 
# Amazon S3 keys, and GnuPG keys
AWS_ACCESS_KEY_ID=
AWS_SECRET_ACCESS_KEY=
PASSPHRASE=
export AWS_ACCESS_KEY_ID
export AWS_SECRET_ACCESS_KEY
export PASSPHRASE
 
echo "*************************************************"
echo "*   Backing up Website content....              *"
echo "*                                               *"
echo "*     www.willowoakboarding.com...              *"
duplicity /www/www.willowoakboarding.com s3+http://s3.amazon.com/willowoak/www
echo "*     www.mywushublog.com...                    *"
duplicity /www/www.mywushublog.com s3+http://s3.amazon.com/mywushublog/www
echo "*     www.m87-blackhole.org...                  *"
duplicity /www/www.m87-blackhole.org s3+http://s3.amazon.com/m87-blackhole/www
echo "*************************************************"
echo "*   Backing up databases....                    *"
echo "*                                               *"
echo "*     www.willowoakboard.com...                 *"
duplicity /u01/backups/willowoak s3+http://s3.amazon.com/willowoak/db
echo "*     www.mywushublog.com...                    *"
duplicity /u01/backups/mywushublog s3+http://s3.amazon.com/mywushublog/db
echo "*     www.m87-blackhole.org...                  *"
duplicity /u01/backups/m87-blackhole s3+http://s3.amazon.com/m87-blackhole/db
echo "*************************************************"

And last but not least, a cronjob to tie it all together:
% crontab -e
@weekly ~/bin/s3-backups.sh
@weekly ~/bin/mywushublog-mysql-backup.sh
@weekly ~/bin/willowoak-mysql-backup.sh
@weekly ~/bin/m87-blackhole-mysql-backup.sh
@weekly ~/bin/evil-genius-network-mysql-backup.sh
I can check the status of a backup by running duplicity with the ‘collection-status‘ flag:
%duplicity collection-status s3+http://s3.amazon.com/mywushublog/db
Last full backup date: Sat Apr 25 15:08:02 2009
Collection Status
-----------------
Connecting with backend: BotoBackend
Archive dir: None
Found 0 backup chains without signatures.
Found a complete backup chain with matching signature chain:
-------------------------
Chain start time: Sat Apr 25 15:08:02 2009
Chain end time: Sat Apr 25 15:08:02 2009
Number of contained backup sets: 1
Total number of contained volumes: 1
Type of backup set:                            Time:      Num volumes:
Full         Sat Apr 25 15:08:02 2009                 1
-------------------------
No orphaned or incomplete backup sets found.
I can also list the files:
%duplicity list-current-files s3+http://s3.amazon.com/mywushublog/db
Last full backup date: Sat Apr 25 15:08:02 2009
Sat Apr 25 15:05:11 2009 .
Sat Apr 25 15:05:10 2009 daily
Sat Apr 25 15:05:10 2009 daily/mywushublog
Sat Apr 25 15:05:10 2009 monthly
Sat Apr 25 15:05:10 2009 weekly
Sat Apr 25 15:05:11 2009 weekly/mywushublog
Sat Apr 25 15:05:11 2009 weekly/mywushublog/mywushublog_week.17.2009-04-25_15h05m.sql.gz
Pretty sweet automated backup process. It is a lot cheaper than tapes or additional disk storage. With S3, I also don’t have to worry about buying additional hardware, the maintenance of a library or tape drive (which is what I had a few years ago, what a headache).

mike Geekyness Backups, FreeBSD, MySQL, PostgreSQL, S3

!!! UPDATE on 12/29/2009!!!
Since this blog post seems to get a good amount or hits from google, if you are reading this, please see my updated post: http://www.mywushublog.com/2009/12/freebsd-8-0-a-great-nas-server/ which has some additional information about FreeBSD 8.0
EOF

Lately at work, I’ve been involved with a very large file system that is being export from Solaris 10/ZFS to windows and OS X users via Samba. Even with a very large Sun server (T5220) a lot of users are complaining about the slow performance of the system. I’m not going to go into details, but what it has prompted me to do is to look into what I use at home (FreeBSD + Samba) for my network storage needs, and see if I can improve the performance of it.

Well, I was checking out the very useful Whats cooking for FreeBSD 8, unrelated to my Samba needs, when I noticed this post on Ivan Voras’ blog. Ivan has some really cool information there, and with this new knowledge I began my update from running Samba 3.0.28a,2 to 3.3.3,0 (the comma represents the Ports version).

Hvala Ivan!

I’ve discussed my new FreeBSD environment before here, with that, here are some quick details:

• OS: FreeBSD 7.1
• Intel Core 2 Duo E6750 (2.66Mhz 4MB cache)
• Intel S975XBX2 workstation motherboard
• AMCC 3Ware 9650SE 4 port SATA RAID controller (4x PCI-e)
  • BatterY backup for the 3Ware so I can enable cached writes
• 2GB ECC Crucial Memory Kit
• ASUS EN6200 LE 16x PCI-e nVidia GFX card
• 4 Western Digital 1TB Drives

The 4 1TB drives create a nice 2.6TB (RAID5) array which I used Samba to share out to my 4 other systems in the house (which is a mix of Windows XP and Vista, sorry, no OS X). I do a lot with this array, any work that Michele and I do like photo editing, word documents, media files, etc.. all gets saved to this volume. Needless to say, this volume is accessed A LOT, and if the house ever caught on fire, I’d save the server before the family (Hey, I could still look at their pictures and videos…).

The Old Configuration

samba 3.0.28a:

[root@server samba3]> make showconfig
===> The following configuration options are available for samba-3.0.34,1:
     LDAP=on "With LDAP support"
     ADS=on "With Active Directory support"
     CUPS=on "With CUPS printing support"
     WINBIND=on "With WinBIND support"
     ACL_SUPPORT=on "With ACL support
     AIO_SUPPORT=on "With Asyncronous IO support"
     FAM_SUPPORT=on "With File Alteration Monitor"
     SYSLOG=on "With Syslog support"
     QUOTAS=off "With Disk quota support"
     UTMP=on "With UTMP accounting support"
     PAM_SMBPASS=off "With PAM authentication vs passdb backends"
     CLUSTER=off "With experimental cluster support"
     DNSUPDATE=off "With dynamic DNS update(require ADS)"
     EXP_MODULES=off "With experimental modules"
     POPT=on "With system-wide POPT library"
     PCH=on "With precompiled headers optimization"
     MAX_DEBUG=off "With maximum debugging"
     SMBTORTURE=off "With smbtorture"
===> Use 'make config' to modify these settings

The New Configuration

samba 3.3 config:

[root@server samba33]> make showconfig
===> The following configuration options are available for samba-3.3.3:
     LDAP=on "With LDAP support"
     ADS=off "With Active Directory support"
     CUPS=on "With CUPS printing support"
     WINBIND=on "With WinBIND support"
     SWAT=off "With SWAT WebGUI"
     ACL_SUPPORT=on "With ACL support"
     AIO_SUPPORT=on "With Asyncronous IO support"
     FAM_SUPPORT=off "With File Alteration Monitor"
     SYSLOG=off "With Syslog support"
     QUOTAS=off "With Disk quota support"
     UTMP=off "With UTMP accounting support"
     PAM_SMBPASS=on "With PAM authentication vs passdb backends"
     DNSUPDATE=off "With dynamic DNS update(require ADS)"
     DNSSD=off "With DNS service discovery support"
     EXP_MODULES=off "With experimental modules"
     SHARED_LIBS=off "With shared libraries"
     POPT=on "With system-wide POPT library"
     MAX_DEBUG=off "With maximum debugging"
     SMBTORTURE=off "With smbtorture"
===> Use 'make config' to modify these settings

If you have never dealt with FreeBSD’s Ports system, setting these compile time options is a breeze with ‘make config’. These options are presented in a nurses interface (which is great for ssh or other terminal based sessions) like this:

FreeBSD's make config screen

The noticable options that I chose are AIO=yes and ADS=off. I’ve always compiled ADS support thinking that I would also get around to configuring Samba as an Active Directory-like server. But you know, there are really only two active users here, myself and Michele, and I don’t see the benefit right now. It could also slow down samba with the extra system calls so again, I’m leaving it out. AIO, Asyncronous IO, is new, and it is reported to increase size (oh wait, wrong advertisement) I mean, network IO.

smb.conf options

The performance related options in smb.conf are here:

        socket options=SO_RCVBUF=131072 SO_SNDBUF=131072 TCP_NODELAY IPTOS_LOWDELAY
        min receivefile size=16384
        aio read size = 16384
        aio write size = 16384
        aio write behind = true

Again, I got these from Ivan’s post, plus what I’ve used for the past 8 years.

Benchmarking with IOzone

I ran a simple

iozone -Ra -b samba3.0.28.xls

for both versions of samba. I’d create some nice 3d charts but I really don’t know excel well enough. So, I’ll just link these here and you can see for yourself what the substantial differences are. Some quick glances while the tests ran, I saw around 50MB/sec for Samba 3.0.28 here and there, topping out at ~60MB/sec. This was only with 256MB and above files. Smaller files always stayed around 5-10MB/sec.

Samba 3.3.3 – with AIO enabled started up fine, but iozone crashed after wrting a few small 64K bytes. This was a little disappointing, however, I did continue the benchmark with the new send and recieve sizes.

UPDATE:
After doing a little more reading, I found out I was supposed to load the aio kernel module. After running:

$ kldload aio

I restarted Samba with the AIO options enabled, re-ran iozone, and it all worked.

ufs2-3ware-raid5-freebsd71

samba3028

samba333

mike Geekyness Benchmarking, FreeBSD, Samba

Tonight was Mark’s last class before he leaves for Idaho for a month. Last week, he joked that he wanted everyone to still be recovering when he gets back in June… He tried his best to make that happen, and I think Patti bailed us out in the end

We got some time to warm up for a few minutes, I think he wanted us ready by 7:20, so we jogged around for a bit and did some light stretching. Mark went around to assist us, which always helps out, but its a tad painful. Ahh good ol’ pain, its such a gift. Then I think Mark rubbed his hands together, muttered “Let us begin…” and cackled…

Tonight was all about basics, so our first drill was simple: keep your arms out straight, palms flat, and do a series or 20 front stretch kicks. The trick was, is we would do a round of kicks, using a bit of our momentum from the previous kick, to really get it up there – head to toe if you can. The alternating set was to NOT use the momentum from the previous kick. So you would kick, and stop immediately on the way down, wait, and do it again. Doing it this was is a lot more difficult, and its great because it really strengthens your leg muscles more, and you end up working your core more as well. Between those, and I think we did around 100 kicks, we still had to keep our arms up. I’m happy to say I never wussed out and took a break, but as of Saturday, days later, I can still feel it in my forearms and shoulders.

The next drill was normal front stretch kicks, and we were broken up in the three typical groups. Simple, but the trick this time was Mark wanted all of us to synchronize our kicks. Since we have a few people that always like to march to their own drum, this can take a while to get right So its always hilarious when people are getting frustrated after the 5th or 6th try and still hearing Mark say “Almost, but not quite, again.” Felicia is yelling at people, Eli is ready to quit, everyone is throwing in their two cents on who should follow who and who needs to slow down Its funny I tell ya.

This lasted an hour, and I think Mark wanted to work on more basic stance work but Patti suggested we all work on weapons. This was good, because my quads felt like they were pulled off, they were so exhausted that I think they detached themselves and walked off, giving me the finger on the way out.

Mark had us alternate between a group of long weapons (which I was in, staff), and then a group of short weapons (straight sword/broadsword). We did basic stuff like figure 8’s and reverse flowers for a full minute, switching out groups. After doing that for a while, Mark had Johnny and myself work on some nangun basics. Now, most of what I’ve learned has been from one seminar 4 or so years ago when the Beijing Wushu Team came out. I’ve practiced this one form from video clips over and over again, and when Mark worked with me on some of the basic moves, it was like learning it all over again. It really breathes new life into things when you get to correct the simple moves. I’ll have to make sure I get more from him in June.

After class, we went to Diamo’s to eat, and I stayed until midnight we had a lot of fun, and some good conversations.

mike Wushu

Sunday we celebrated Easter, as well as Owens 1st birthday (its really April 22nd, but we took advantage of having most of the family in one place), at my parents place in Knightsen.

Here is a link to the gallery on my other site, m87-blackhole.org:
http://www.m87-blackhole.org/photos/2009-04-12-easter/

I also took some pictures of my parents house in the making. It is starting to shape up nicely, my step dad is a pretty talented carpenter.

mike Family Easter, Owen

Saturday was the 17th annual Cal Martial Arts Tournament, at Cal Berkeley. For the 4rd year in a row I went, and as the last three years I was solely a spectator. I took some footage, and snapped a few pictures, but I wasn’t into staying from 8am till 11pm running around to each ring trying to film everyone. My experience with that last year solidified the idea that no matter what, someone (usually more than one too) will be unhappy with what you’ve worked hard to do, and I wasn’t interested in repeating that thankless job. You know, things like “You got my score wrong” (I didn’t), “be sure to spell my name right” (that’s hard to do with a resource like Facebook), and “how come you didn’t get X form?”. So, I enjoyed just watching and hanging out with the usual suspects.

Collin "smiling", poor kid...

Normally, I take BART to Berkeley with an event like this. This year I decided to drive, for a few reasons:

• Taking BART turns a 35 minute drive to a 80-95 minute commute (I normally miss the first train during the transfer)
• BART doesn’t operate all night, and the last few years I’ve had to rush out of CMAT to catch one of the last trains
• BART is still expensive to me

So I ended up parking on Hearst, right next to Wushu West, and walked a good 30 minutes to Cal. I enjoyed it, it was a very peaceful walk, and I enjoyed the mild exercise. The extra bonus is I didn’t have to pay for parking.

I walked in just in time to talk with Connor and his parents (it was about 3:40pm, and he was done for the day) and I unfortunately already missed the Nandu (“difficulty”, all the flashy stuff) division. Chase got a Bronze though, good job. For the first few hours, things were pretty uneventful and it wasn’t until 8pm or so when the advanced divisions went on when things started to pick up again. It was sort of odd not to have Collin and Shahaub compete this year, and it seemed that the entire overall quality of competitors was lower than previous years. Also, there were not as many people from China, you know, the ringers, to completely destroy everyone else. Still, there were some good people, and Anton was very good (he is also new to wushu, which makes it even crazier).

Lynda, Patti and Mark. Mark is covering up his mysterious hair cut

The entire competition lasted till around 11pm. There was a young girl who must have competed in almost 6 or 7 divisions. She was the last female competitor and she did a very good job:

crazy girl in mid aerial

After it was over, I contemplated going over to Marks to watch videos and hang out with everyone, but man it was late, and I had to walk back to my truck. I was also getting a little worried that I may have been towed I’ve been towed away in the neighborhood before, and even though I asked the person who lived there if it was okay I still felt very uneasy. With that, I made it back to my truck, munching on a free veggie sandwich (it was from the judges lunchroom, they had a bunch left over). The evening was very nice, it was cool and quiet. I made it home at exactly 12am.

Collin judging, why so judge-y Collin?

Stephanie Lim and Anton Alexsiev(sp?)

Wong Wei and Shahaub judging two separate events

Now they are waving at us (Mark and I moved to the other side of the staduim)

Those are all the photos (that are worth showing, most of them didn’t come out). I’ll probably upload some of the footage I took to the Wushu West You Tube account later. Or I’ll give it to Mark and let him decide what he wants up there or not.

mike Wushu CMAT, Collin, mark, Pierre, Shahuab

Ip Man

It’s long overdue, but Mark, Shahaub and I finally got together for a movie. All three of us are movie geeks. Mark was part of the industry with Jet Li and Rotten Tomatoes, Shahaub is in film school right now (and posted a few demo reels, like this one, on vimeo.com), and I like to watch them and then write about what I watch. Our conversations immediately started off with George Lucas movies,  Arrested Development, and then oddly to my fear of Bees and Black Widows. How did we get there?

I figured since they were coming out (of the closet) from Berkeley, I’d prepare lunch for them. We had DELICIOUS cheese steak sandwiches, baked chicken (which Mark provided), and a big salad with Avocado. One thing with those two, is I don’t have to worry about pickyness or food allergies. They will eat anything, and everything on the table

The movie itself was pretty awesome, there are a lot of great fight scenes, and they all prompted Shahaub to imitate the Wing Chun style:

Despide his fierceness, Mark said it felt like a massage.

as well as trying to stand up from a kneeling position, using the strength of his feet (as in the movie):

ready?

and UP...

and back down, nice try Shabi

The movie is based on Grandmaster Ip Man (or Yip Man), who was most notable for teaching Bruce Lee Wing Chun. I’m sure there are some artistic liberties taken to make a more entertaining movie, the back story was still very interesting. After the movie, we went though a bunch of wushu clips that I had, then some on youtube, and then squeezed an episode of Arrested Development in (The “Nelly” episode, oh it feels so good to talk like this!). When Michele, Caralyne and Owen came back home, Caralyne had a lot of fun shooting all three of us. his fueled the inner child in Shahaub, and made him question why he doesn’t own Nerf weaponry. We know why, its because no one would be safe, and every crotch in a half mile radius would require medical attention. After a few rounds of destroying each other, Mark and Shahaub left to meet Chase and his family for dinner.

mike Movies, Wushu Home Theater, mark, Shahuab