Two node LVS-DR setup on CentOS

We've been working on getting our data center up and going, and part of that was setting up an honest to goodness load balanced system with fail over.

If you have gobs of money, the usual way to deal with high availability is to have your n+1 servers fronted by a few F5 load balancers. These take care of keeping track of what servers are up and send traffic only to those. Of course, you then need two of them, so that in case your site goes down things keep working. Only problem, is they are expensive. As in, "we don't list the price anywhere" expensive. (always a sure sign you can't afford it)

So since we aren't built out of money, and we always prefer things that are open and we can poke at, we decided to build our load balancer and high availability solution using the very same two nodes we will be serving traffic from.

The two pieces of software that make this possible (apart from the excellent Linux kernel itself) are heartbeat and ldirector.

Heartbeat takes care of the high availability problem. One box is picked as the primary owner of a service and the other watches and listens. If it no longer hears the primary (or the primary indicates that it is going down), the secondary box takes over the IP address, starts up the services and all is well.

ldirector on the other hand, serves the job of distributing load across a set of nodes. You define which nodes can answer queries, how to tell if they are up, and ldirector takes care of forwarding requests onto them.

By having Heartbeat own ldirector, then you have a high availability load balancer. All for free, all using the same hardware. This costs some CPU, but it is very little, and in our case, the bigger challenge is availability rather than scaling.

Settings this on up CentOS 4.4 is fairly straightforward except for a few gotchas that took some working through.

If you follow the ultra monkey example it will get you most of the way there. The one problem you'll run into though is that when heartbeat starts, it checks that it doesn't own the VIP and will kill your lo:0 loopback on the VIP which you use to serve pages as a working node.

The solution is to write a quick script which takes care of both the arp-problem and the lo:0 problem at once and which is controlled by heartbeat when bringing ldirector up and down.

You'll find my script below. You'll want to edit it to put in your own VIP, but otherwise it should work as is. I recommend copying it into /etc/ha.d/resource.d and naming it something like answer-arp. You can also link /etc/ha.d/resource.d/startstop to it and that will take care of removing and adding lo:0 at the right times so heartbeat doesn't complain or leave you in a state of not being a proper worker node.

To trigger the script to kick in when heartbeat starts or stops ldirector, add this script as a resource in haresources and make sure it's before the implicit IPAddr script. My haresources file looks something like:

# haresources
node1 arp-answer 192.168.0.120 ldirectord::ldirectord.cf

And finally, here's my arp-answer script:

#!/bin/bash
#
# Handles the various init procedures required on RHEL4 systems to
# use LVS-DR.
#
# Specifically:
#     - enables forwarding via sysctl
#     - shuts down local lo:0 when starting heartbeat to avoid complaints
#     - handles turning on/off arp responses via arptables
#
# You'll want to set this resource before IPaddr or IPaddr2 in your haresources
# as otherwise the lo:0 link will be killed by IPaddr.
#
# You can also use this script as the /etc/ha.d/resource.d/startstop script
# to make sure the system is in an ok state when heartbeat starts and a valid
# LVS-DR node when heartbeat stops. (IE, has a VIP on lo:0 and doesn't answer
# arps)
#
# You must set the VIP address to use here:
VIP=192.168.0.120

host=`/bin/hostname`
case "$1" in

# if linked as startstop this is triggered when heartbeat first loads
pre-start)
        # turn on forwarding (assumes sysctl.conf has been modified to
        # have net.ipv4.ip_forward = 1)
        /sbin/sysctl -p

        # give up lo:0
        /sbin/ifdown lo:0

        # don't answer arps yet
        /etc/ha.d/rc.d/arptables-noarp-addr_giveip $VIP
;;

# called when this resource becomes active
start)
        # give up lo:0
        /sbin/ifdown lo:0

        # we want to answer arps, do so
        /etc/ha.d/rc.d/arptables-noarp-addr_takeip $VIP
;;

# startstop hook, noop
post-start)
;;

# startstop hook, noop
pre-stop)
;;

# called either when heartbeat exits completely, or a resource
# is taken down by heartbeat.
stop|post-stop)
        # bring up lo:0 again, this will be our loopback with our VIP
        /sbin/ifup lo:0

        # but do not answer arps anymore
        /etc/ha.d/rc.d/arptables-noarp-addr_giveip $VIP
;;

# required by the resources constract
status)
        # make sure that eth0:0 is up AND lo:0 is down
        islothere=`/sbin/ifconfig lo:0 | grep $VIP`
        iseththere=`/sbin/ifconfig eth0:0 | grep $VIP`

        # are we ignoring arps on our VIP?
        isarpignore=`/sbin/arptables -L | grep $VIP`

        if [ "$islothere" -o "$isarpignore" -o ! "iseththere" ];then
            # eth0:0 isnt there or lo:0 is there or we are ignoring arps
            echo "LVS-DR director Stopped."
        else
            echo "LVS-DR director Running."
        fi
;;
*)
        # Invalid entry.
        echo "$0: Usage: $0 {pre-start|start|status|stop|post-stop}"
        exit 1
;;
esac

Auto last_modified_date and creation_date in MySQL 5.0

There's a few different solutions out there to have two auto updating timestamp columns in MySQL 5.0. One using two triggers works, but seems overly complicated as you can have the last_modified_date updated by MySQL without a trigger.

Here's my recipe after fooling around a bit:

1) Declare you table with an auto updating last_modified date:

CREATE TABLE foos (
id int primary key auto_increment,
bar varchar(64) not null,
creation_date timestamp default '0000-00-00 00:00:00',
last_modified_date timestamp on update current_timestamp default current_timestamp
)

The trick here is that you have to define a default for the creation_date timestamp, otherwise MySQL will complain that you can't have two updating timestamps. In this case I've chosen to use the update attribute on the last_modified_date column as I'm guessing that mechanism is faster than the trigger and I expect more updates in this table than inserts.

2) Create a trigger to insert the current time in the creation date column on insert:

create trigger foos_insert before insert on foos
for each row set NEW.creation_date = now();

This combination will cleanly keep both of your columns up to date as you perform inserts and updates. Note that if you'd like, you could default the last_modfied_date to null as well to easily be able to query for all rows that have never been updated.

Books I've been reading

Working for yourself brings the added pleasure of being able to dive into technical books that you otherwise wouldn't have time to explore. Of course the whole "I don't have time for that" argument is almost always a fallacy.

Anyways, I've been reading a few technical books lately and here are some quick reviews (presented in order of intrinsic goodness)

Enterprise Linux Clusters by Karl Kopper

Wow! To say this is a good book is an understatement. One of the best technical books I've ever read. It was so engaging and interesting I actually read through it in one sitting. Rarely is presentation so clearly thought out and rarer still is the book which covers such relevant topics. Highly highly recommended, even if you aren't building a cluster using LVS.

Jakarta Commons Cookbook by Tim O' Brien

Another excellent book. This does a good job of covering a variety of different Jakarta Commons packages and really driving home how vast the selection is. A few points docked for having a 'recipe' for each package and how to download it, but otherwise a very high content / fluff ratio. Fantastic book to buy that engineer you know who's always reinventing the wheel.

Postfix by Ralf Hilderbrandt and Patrick Koetter

A good overview of Postfix. I was hoping for a bit more from this book. Yes, the postfix config files are incredibly well documented, but I was a bit disappointed to see "RTFM" comments in a book written by the authors. (I thought this was the FM!) But it still does a good job of covering the basics of MTA's and why Postfix is the way it is. I've always been a qmail fan, but this book pushes me over the edge to Postfix.

Maven, A Developers Notebook

Ugh. This is a bad book. See my Maven post for more, but with a product so lacking in good documentation this only continues the trend. It covers only the common case, doesn't describe or explore the larger Maven architecture and doesn't help equip you to solve your own problems. I'll be putting this one up for sale on AMZN in short order.

Maven.. jury is still out

We just started a new project that will include a large server component so I thought I'd try out Maven for our build environment instead of the usual ant. I've never been a huge ant fan, but over the years its become familiar, and it has always been easy to extend or tweak to do what you want. The only bad thing is that you have to do it all from scratch (or likely the build.xml from your last project) every time.

Maven has some benefits in this respect. I kind of see Maven as the 'Ruby on Rails' of the build world. It does a lot for you, and makes bootstrapping a project incredibly quick. You can get a sane directory structure, a test runner and compilation going in 60 seconds flat. In Ant that would require a good deal of copying and hacking on your old build file, copying in your test runner etc..

So in that respect, it's pretty awesome. The layout of src and unit tests, resources etc is all sane and things I can live with.

One big things Maven does which is interesting is manage external dependencies. Instead of having a /lib dir checked in with your dependent jars, you now just specify what version of what library you require and Maven takes care of downloading it from the iBiblio repo, adding it to your classpath etc.. This is pretty neat I admit, though I'm a bit uneasy because it reminds me a bit of CPAN which in my experience works fantastic 95% of the time, but the remaining 5% drives you bonkers. Then again, this is Java WORA, so we shouldn't have those issues right? ;)

I'm not sure I buy into the dependency management being a huge win. Yes, it keeps you from having libs checked into SVN, but why is that such a bad thing anyways? Your SVN repository is probably on a server that is very fast for you to access, you are guaranteed it will work, and guaranteed that iBiblio won't one day blow up and leave you running around for your dependent jars.

Then comes all the automagic of Maven. When it works and you are doing something it expects, great. Otherwise.. be prepared for a time sync. I've worked with a lot of tools over the years, and Maven is by far the most obtuse and worst documented among them. Just figuring out how to include my own external jar that didn't exists in iBiblio took the better part of an hour. I can chalk a lot of this up to being new to it, but it is usually easier. I even bought a Maven book, "A Developer's Notebook" and this has to be one of the most useless texts I've ever used. All it focuses on is the common path. There are plenty of discussions of the common path on the web, when I buy books I want to know the details, and this book has none of them.

So so far, I think the jury is still out on Maven. I do appreciate how quick it was to set up a sane build environment for this project. We'll see how things progress as I want to customize the build in the future.

Just be nice

I spend a lot of time walking around downtown Seattle, to and from work, going out or whatnot. Invariably, and almost without exception I see a car honk at another or a pedestrian yelling at a driver. Not to mention all the bike/car interactions.

The thing is, I don't understand why everybody is so stressed. Sure in some cases, especially the bike/car conflicts, there very well might be some physical harm possible, but most of the time it's something simple.

Today on my way in, I witnessed a driver who was going slowly get a huge honk from someone behind them. It was quite obvious Mr Slow was lost and wasn't quite sure whether the upcoming intersection was the one he wanted to turn in. But did Mr Speedy really need to honk? I mean, how much time did he lose? 5 seconds? Maybe 10? What exactly did he gain by honking apart from raising his blood pressure and likely the other drivers?

The other day I saw a similar interaction with a pedestrian yelling at a car for being in the crosswalk. It was quite obvious she (the driver) was trying to take a right turn and didn't realize she would be in the crosswalk so long. But again, did the pedestrian really have to create such a negative experience for everybody involved? Was it such a big deal to walk 2 feet over to walk around the car?

Sometimes you don't even need to do anything wrong. I was waiting for a bus a few days ago and a fellow warned a guy standing in the street that a car was backing up to park. The potential victim responded "I don't fucking care, if they can't drive I'll sue them." in a really aggressive tone. What in the world? Here you have someone actually going out of their way to look out for you and you respond like that? This was one case where I wished I would have spoken up in defense of the good Samaritan.

I just don't get it. People get so stressed over the smallest inconsequential things. Why don't we all take a deep breath, relax and remember that we create the world we live in. All that negative energy you put out there shapes the world.

Building a cheap datacenter

Eric and I are building a cheap datacenter for Trileet that we can reasonably say has at least 4 9's (99.99%) uptime. Cheap is a relative term of course, plenty of companies would feel perfectly good about doing that with a $15k budget, but we wanted to aim for something closer to $5k.

Achieving 4 9's is in theory fairly simple:

1. Build software that doesn't break. Ok, that's not so simple, but doable, and that's something we are pretty good at. Without getting into the hows of accomplishing that, let's just say that's in the bag.
2. Make sure you don't have a single point of failure.
3. Configure everything to automatically reconfigure in case of failures.

#2 reminds me a lot of rock climbing techniques. Whenever climbing you pretty much always make sure that one thing going wrong won't kill you. IE, when you are climbing, you have someone belaying you, so that if you fall (one thing going wrong) you don't die. Or when setting up an anchor, you always make sure that no single piece failing will cause the whole thing to pull out and sending you to your death.

The same is true when building a data center. You need n+1 of everything, where n is the number you need to scale appropriately.

In our case, n is simply one. We aren't doing anything ridiculously expensive and servers are fast these days. So that means two servers.

After looking around a bit, we chose Dell 1950's. These seemed like a good balance between good performance, 1U of rack space and with enough headroom to upgrade if need be. Configured with an Intel 5130, 2 GB of ram and 160 GB of disk these should do fine for now.

Two of those brings us to about $3400, so that leaves $1600 for other things.

I'll cover what we spent the rest of our money on in upcoming posts, along with some pictures of our set up once complete.

Hello World

Am I late for the party?

Yes.. yes.. another blog, just what we needed.

I've always mocked blogs a bit, I mean, who really wants to hear about Joe's latest woes with his girlfriend? But then again, these days blogs of one form or another are the primary means of new content being created on the web.

So I decided to try to knock down my consumer/producer ratio a bit and add some content that might prove useful to others. By and large that'll be stuff related to software engineering, usually focused in the Java world, or the Linux world, or the Sidekick world.

So hopefully you'll find something useful here one day when you are doing that google search for why you are getting a ClassCastException.

And I promise no girlfriend woe stories.