Avoiding Split Brains

Subsequent to my previous post, I received a few questions about how to handle split brain. Before answering, here's a short recap:

When you have a master-slave scenario, there are cases where you rather have no active master rather than having two (or more) masters running at the same time (split brain, in other words). The question is, how can you make sure no more than one server will be master when the master and slave get disconnected?

The obvious answer is to use one of the commercial products like Microsoft Cluster Server, Veritas, Linux-HA, and so on. Going with a commercial product is especially recommended if you need complex scenarios like geographically dispersed clusters, and the like.

Nevertheless, if you do find yourself in situations where you need to roll your own solution besides the obvious bit you need to establish between the server, you also need to solve the split brain problem of deciding who is master.

There are two approaches I know of (that work in commercial products):

• Have some resource which everyone can see (communicate with). The first to obtain a lock on the resource is the master if a server doesn't see the resource, it doesn't consider yourself part of the cluster.
• Another approach is based on voting. If you have three computers, the one that gets two votes wins.

Note that these strategies work for situations where there no master is better than two masters.if your requirements are different you need to consider other options.

P.S.
I got one comment from someone who thought the previous post was offensive for that applicant.This perspective didn't occur to me when I posted it. I am not sure he reads my column at all -- but if you do and you've found that post offensive, I heartedly apologize. That was not my intention. By the way, the actual applicant I interviewed is a brilliant person who has great design skill, and great depth of knowledge in the technology (.Net in that case). I would hire him as a lead developer without thinking twice. I just didn't find him ready for an architect role -- yet -- for the reasons I pointed out in the post

Posted by Arnon Rotem-Gal-Oz at 05:50 PM  Permalink