Wednesday, September 2, 2009

Continuous Integration with TeamCity

I've worked with a few different continuous integration tools in the past and written a few perl scripts to handle the job at various times in my career. When we were looking for a new tool at Trileet, we started with Continuum. This worked for a long while but for reasons I no longer recall, we later switched to TeamCity by JetBrains (the Czech dudes that make IntelliJ IDEA) -- and holy smokes, I love it. It couldn't get any easier and this thing is packed with power.

If you haven't taken it for a spin yet, I implore you to give it a try. While its not open source (probably a big reason why its so easy to use), it does provide a lot of functionality for free. You can manage up to 20 different build configurations without spending a dime.

Thursday, March 19, 2009

Inconsistent Hierarchy Error in Eclipse

Occasionally I run into an error using Eclipse where it reports that the hierarchy for type XYZ is inconsistent. The problem isn't that there is a problem with the given class, but instead that the class it extends is not compiling properly. In my most recent case, the base class was missing a needed class in it's classpath.

Thursday, February 15, 2007

printf() Debugging

Over the years I've gotten various flack from coworkers on just how much I use printf() debugging when coding. They invariably point me to the awesome debugger in Eclipse, start trying to teach me how to attach to the process or somesuch..

It's not that I don't know HOW to use a debugger, I even will grant you that they are often superior. It's that over the years I've resisted using any tools that lock me into one particular platform. My primary value as a developer is in being able to code. To code in any language, regardless of the tools available to me.

That's why I've only recently moved from emacs as my primary editor (at some point Eclipse just become TOO good to ignore), and why I still use printf debugging as my first shot of figuring out what has gone wrong.

That has served me well. I think there's a certain amount of skill that is built over time as where to put those printf()'s. There's also something to be said for looking at the code first and seeing if it's obvious what is wrong before diving into a debugger. printf debugging is a skill that transcends languages, platforms and operating systems. It's probably safe to say that any platform I'll ever develop on will have some way of outputting something I can see.

I've coded in Perl, PHP, Java, C, C++ and others over the years, and in all cases there have been times where only the most rudimentary debugger was available. When I first started using Java back in the Alpha/Beta days we didn't even have jdb. I don't think PHP even offers a debugger.

A case in point. Today I was setting up some SysV init scripts for feedmonger. Astonishingly it seems everybody out there writes their own init script from scratch, and they all seem rather different. On my search for the 'right way' to do things, I came across start-stop-daemon, a (mostly) Debian utility that takes care of most of the dirty work of starting and stopping things appropriately. (checking if the process is running, sending it stop signals, making pid files etc..) For whatever reason, this excellent tool isn't really available on Redhat systems. Ok, no problem, I grabbed the source (one .h, one .c) and compiled it for our systems. Wrote my init script and doh, it doesn't seem to be working as expected: start-stop-daemon isn't finding the previously running process when duplicate start commands are sent and is trying to start it anyways. No good!

I pondered the docs a bit and really couldn't figure out what was going on. Figuring maybe it was something to do with our 64 bit setup, I decided to add in some.. wait for it.. printf debugging to see if I could figure out what was up.

Turns out my usage was wrong (good old meatsicle bug). If you want start-stop-daemon to check for the process before starting based on the command you cannot use the '--exec' switch, you must instead use the '--startas' switch. This really isn't made terribly clear in the documentation sadly.

Ok, so I'm an idiot sometimes too, but without a bit of printf debugging (on a plain old c executable I wasn't about to gdb) I wouldn't have figured out what was wrong.

Sunday, December 10, 2006

Two node LVS-DR setup on CentOS

We've been working on getting our data center up and going, and part of that was setting up an honest to goodness load balanced system with fail over.

If you have gobs of money, the usual way to deal with high availability is to have your n+1 servers fronted by a few F5 load balancers. These take care of keeping track of what servers are up and send traffic only to those. Of course, you then need two of them, so that in case your site goes down things keep working. Only problem, is they are expensive. As in, "we don't list the price anywhere" expensive. (always a sure sign you can't afford it)

So since we aren't built out of money, and we always prefer things that are open and we can poke at, we decided to build our load balancer and high availability solution using the very same two nodes we will be serving traffic from.

The two pieces of software that make this possible (apart from the excellent Linux kernel itself) are heartbeat and ldirector.

Heartbeat takes care of the high availability problem. One box is picked as the primary owner of a service and the other watches and listens. If it no longer hears the primary (or the primary indicates that it is going down), the secondary box takes over the IP address, starts up the services and all is well.

ldirector on the other hand, serves the job of distributing load across a set of nodes. You define which nodes can answer queries, how to tell if they are up, and ldirector takes care of forwarding requests onto them.

By having Heartbeat own ldirector, then you have a high availability load balancer. All for free, all using the same hardware. This costs some CPU, but it is very little, and in our case, the bigger challenge is availability rather than scaling.

Settings this on up CentOS 4.4 is fairly straightforward except for a few gotchas that took some working through.

If you follow the ultra monkey example it will get you most of the way there. The one problem you'll run into though is that when heartbeat starts, it checks that it doesn't own the VIP and will kill your lo:0 loopback on the VIP which you use to serve pages as a working node.

The solution is to write a quick script which takes care of both the arp-problem and the lo:0 problem at once and which is controlled by heartbeat when bringing ldirector up and down.

You'll find my script below. You'll want to edit it to put in your own VIP, but otherwise it should work as is. I recommend copying it into /etc/ha.d/resource.d and naming it something like answer-arp. You can also link /etc/ha.d/resource.d/startstop to it and that will take care of removing and adding lo:0 at the right times so heartbeat doesn't complain or leave you in a state of not being a proper worker node.

To trigger the script to kick in when heartbeat starts or stops ldirector, add this script as a resource in haresources and make sure it's before the implicit IPAddr script. My haresources file looks something like:

# haresources
node1 arp-answer

And finally, here's my arp-answer script:

# Handles the various init procedures required on RHEL4 systems to
# use LVS-DR.
# Specifically:
# - enables forwarding via sysctl
# - shuts down local lo:0 when starting heartbeat to avoid complaints
# - handles turning on/off arp responses via arptables
# You'll want to set this resource before IPaddr or IPaddr2 in your haresources
# as otherwise the lo:0 link will be killed by IPaddr.
# You can also use this script as the /etc/ha.d/resource.d/startstop script
# to make sure the system is in an ok state when heartbeat starts and a valid
# LVS-DR node when heartbeat stops. (IE, has a VIP on lo:0 and doesn't answer
# arps)
# You must set the VIP address to use here:

case "$1" in

# if linked as startstop this is triggered when heartbeat first loads
# turn on forwarding (assumes sysctl.conf has been modified to
# have net.ipv4.ip_forward = 1)
/sbin/sysctl -p

# give up lo:0
/sbin/ifdown lo:0

# don't answer arps yet
/etc/ha.d/rc.d/arptables-noarp-addr_giveip $VIP

# called when this resource becomes active
# give up lo:0
/sbin/ifdown lo:0

# we want to answer arps, do so
/etc/ha.d/rc.d/arptables-noarp-addr_takeip $VIP

# startstop hook, noop

# startstop hook, noop

# called either when heartbeat exits completely, or a resource
# is taken down by heartbeat.
# bring up lo:0 again, this will be our loopback with our VIP
/sbin/ifup lo:0

# but do not answer arps anymore
/etc/ha.d/rc.d/arptables-noarp-addr_giveip $VIP

# required by the resources constract
# make sure that eth0:0 is up AND lo:0 is down
islothere=`/sbin/ifconfig lo:0 | grep $VIP`
iseththere=`/sbin/ifconfig eth0:0 | grep $VIP`

# are we ignoring arps on our VIP?
isarpignore=`/sbin/arptables -L | grep $VIP`

if [ "$islothere" -o "$isarpignore" -o ! "iseththere" ];then
# eth0:0 isnt there or lo:0 is there or we are ignoring arps
echo "LVS-DR director Stopped."
echo "LVS-DR director Running."
# Invalid entry.
echo "$0: Usage: $0 {pre-start|start|status|stop|post-stop}"
exit 1

Thursday, November 30, 2006

Auto last_modified_date and creation_date in MySQL 5.0

There's a few different solutions out there to have two auto updating timestamp columns in MySQL 5.0. One using two triggers works, but seems overly complicated as you can have the last_modified_date updated by MySQL without a trigger.

Here's my recipe after fooling around a bit:

1) Declare you table with an auto updating last_modified date:

id int primary key auto_increment,
bar varchar(64) not null,
creation_date timestamp default '0000-00-00 00:00:00',
last_modified_date timestamp on update current_timestamp default current_timestamp
The trick here is that you have to define a default for the creation_date timestamp, otherwise MySQL will complain that you can't have two updating timestamps. In this case I've chosen to use the update attribute on the last_modified_date column as I'm guessing that mechanism is faster than the trigger and I expect more updates in this table than inserts.

2) Create a trigger to insert the current time in the creation date column on insert:
create trigger foos_insert before insert on foos
for each row set NEW.creation_date = now();
This combination will cleanly keep both of your columns up to date as you perform inserts and updates. Note that if you'd like, you could default the last_modfied_date to null as well to easily be able to query for all rows that have never been updated.

Tuesday, November 28, 2006

Books I've been reading

Working for yourself brings the added pleasure of being able to dive into technical books that you otherwise wouldn't have time to explore. Of course the whole "I don't have time for that" argument is almost always a fallacy.

Anyways, I've been reading a few technical books lately and here are some quick reviews (presented in order of intrinsic goodness)

Enterprise Linux Clusters by Karl Kopper

Wow! To say this is a good book is an understatement. One of the best technical books I've ever read. It was so engaging and interesting I actually read through it in one sitting. Rarely is presentation so clearly thought out and rarer still is the book which covers such relevant topics. Highly highly recommended, even if you aren't building a cluster using LVS.

Jakarta Commons Cookbook by Tim O' Brien

Another excellent book. This does a good job of covering a variety of different Jakarta Commons packages and really driving home how vast the selection is. A few points docked for having a 'recipe' for each package and how to download it, but otherwise a very high content / fluff ratio. Fantastic book to buy that engineer you know who's always reinventing the wheel.

Postfix by Ralf Hilderbrandt and Patrick Koetter

A good overview of Postfix. I was hoping for a bit more from this book. Yes, the postfix config files are incredibly well documented, but I was a bit disappointed to see "RTFM" comments in a book written by the authors. (I thought this was the FM!) But it still does a good job of covering the basics of MTA's and why Postfix is the way it is. I've always been a qmail fan, but this book pushes me over the edge to Postfix.

Maven, A Developers Notebook

Ugh. This is a bad book. See my Maven post for more, but with a product so lacking in good documentation this only continues the trend. It covers only the common case, doesn't describe or explore the larger Maven architecture and doesn't help equip you to solve your own problems. I'll be putting this one up for sale on AMZN in short order.

Saturday, November 25, 2006

Maven.. jury is still out

We just started a new project that will include a large server component so I thought I'd try out Maven for our build environment instead of the usual ant. I've never been a huge ant fan, but over the years its become familiar, and it has always been easy to extend or tweak to do what you want. The only bad thing is that you have to do it all from scratch (or likely the build.xml from your last project) every time.

Maven has some benefits in this respect. I kind of see Maven as the 'Ruby on Rails' of the build world. It does a lot for you, and makes bootstrapping a project incredibly quick. You can get a sane directory structure, a test runner and compilation going in 60 seconds flat. In Ant that would require a good deal of copying and hacking on your old build file, copying in your test runner etc..

So in that respect, it's pretty awesome. The layout of src and unit tests, resources etc is all sane and things I can live with.

One big things Maven does which is interesting is manage external dependencies. Instead of having a /lib dir checked in with your dependent jars, you now just specify what version of what library you require and Maven takes care of downloading it from the iBiblio repo, adding it to your classpath etc.. This is pretty neat I admit, though I'm a bit uneasy because it reminds me a bit of CPAN which in my experience works fantastic 95% of the time, but the remaining 5% drives you bonkers. Then again, this is Java WORA, so we shouldn't have those issues right? ;)

I'm not sure I buy into the dependency management being a huge win. Yes, it keeps you from having libs checked into SVN, but why is that such a bad thing anyways? Your SVN repository is probably on a server that is very fast for you to access, you are guaranteed it will work, and guaranteed that iBiblio won't one day blow up and leave you running around for your dependent jars.

Then comes all the automagic of Maven. When it works and you are doing something it expects, great. Otherwise.. be prepared for a time sync. I've worked with a lot of tools over the years, and Maven is by far the most obtuse and worst documented among them. Just figuring out how to include my own external jar that didn't exists in iBiblio took the better part of an hour. I can chalk a lot of this up to being new to it, but it is usually easier. I even bought a Maven book, "A Developer's Notebook" and this has to be one of the most useless texts I've ever used. All it focuses on is the common path. There are plenty of discussions of the common path on the web, when I buy books I want to know the details, and this book has none of them.

So so far, I think the jury is still out on Maven. I do appreciate how quick it was to set up a sane build environment for this project. We'll see how things progress as I want to customize the build in the future.

Tuesday, November 21, 2006

Just be nice

I spend a lot of time walking around downtown Seattle, to and from work, going out or whatnot. Invariably, and almost without exception I see a car honk at another or a pedestrian yelling at a driver. Not to mention all the bike/car interactions.

The thing is, I don't understand why everybody is so stressed. Sure in some cases, especially the bike/car conflicts, there very well might be some physical harm possible, but most of the time it's something simple.

Today on my way in, I witnessed a driver who was going slowly get a huge honk from someone behind them. It was quite obvious Mr Slow was lost and wasn't quite sure whether the upcoming intersection was the one he wanted to turn in. But did Mr Speedy really need to honk? I mean, how much time did he lose? 5 seconds? Maybe 10? What exactly did he gain by honking apart from raising his blood pressure and likely the other drivers?

The other day I saw a similar interaction with a pedestrian yelling at a car for being in the crosswalk. It was quite obvious she (the driver) was trying to take a right turn and didn't realize she would be in the crosswalk so long. But again, did the pedestrian really have to create such a negative experience for everybody involved? Was it such a big deal to walk 2 feet over to walk around the car?

Sometimes you don't even need to do anything wrong. I was waiting for a bus a few days ago and a fellow warned a guy standing in the street that a car was backing up to park. The potential victim responded "I don't fucking care, if they can't drive I'll sue them." in a really aggressive tone. What in the world? Here you have someone actually going out of their way to look out for you and you respond like that? This was one case where I wished I would have spoken up in defense of the good Samaritan.

I just don't get it. People get so stressed over the smallest inconsequential things. Why don't we all take a deep breath, relax and remember that we create the world we live in. All that negative energy you put out there shapes the world.

Monday, November 20, 2006

Building a cheap datacenter

Eric and I are building a cheap datacenter for Trileet that we can reasonably say has at least 4 9's (99.99%) uptime. Cheap is a relative term of course, plenty of companies would feel perfectly good about doing that with a $15k budget, but we wanted to aim for something closer to $5k.

Achieving 4 9's is in theory fairly simple:
  1. Build software that doesn't break. Ok, that's not so simple, but doable, and that's something we are pretty good at. Without getting into the hows of accomplishing that, let's just say that's in the bag.
  2. Make sure you don't have a single point of failure.
  3. Configure everything to automatically reconfigure in case of failures.
#2 reminds me a lot of rock climbing techniques. Whenever climbing you pretty much always make sure that one thing going wrong won't kill you. IE, when you are climbing, you have someone belaying you, so that if you fall (one thing going wrong) you don't die. Or when setting up an anchor, you always make sure that no single piece failing will cause the whole thing to pull out and sending you to your death.

The same is true when building a data center. You need n+1 of everything, where n is the number you need to scale appropriately.

In our case, n is simply one. We aren't doing anything ridiculously expensive and servers are fast these days. So that means two servers.

After looking around a bit, we chose Dell 1950's. These seemed like a good balance between good performance, 1U of rack space and with enough headroom to upgrade if need be. Configured with an Intel 5130, 2 GB of ram and 160 GB of disk these should do fine for now.

Two of those brings us to about $3400, so that leaves $1600 for other things.

I'll cover what we spent the rest of our money on in upcoming posts, along with some pictures of our set up once complete.

Hello World

Am I late for the party?

Yes.. yes.. another blog, just what we needed.

I've always mocked blogs a bit, I mean, who really wants to hear about Joe's latest woes with his girlfriend? But then again, these days blogs of one form or another are the primary means of new content being created on the web.

So I decided to try to knock down my consumer/producer ratio a bit and add some content that might prove useful to others. By and large that'll be stuff related to software engineering, usually focused in the Java world, or the Linux world, or the Sidekick world.

So hopefully you'll find something useful here one day when you are doing that google search for why you are getting a ClassCastException.

And I promise no girlfriend woe stories.