Split Brains and Half-baked Architects

About a week ago I interviewed a bright individual who is currently working as an Architect. We had a nice interview and I was impressed with the depth of his knowledge. Everything went well until he raved too much about the optimization he did for over the default serialization of the platform (.NET in this case). [strike 1].

The next thing I asked him then was "let's say you have a distributed system with a master and slave, when the master goes down for any reason, the slave should wake up and become the master. How can we prevent both the master and slave from being master when the communication between them breaks [a problem which is a called a split brain]".

The applicant thought about it for a while and told me that it isn't possible. Since when the communication is down the hearth beat between the computers will fail in both nodes and both will become master (thinking the other node failed). [strike 2]

Well, I said, I know several commercial products that do deal with this problem and don't suffer from split brains

"Okay", he said, tell me about them, "and I'll show you where they are broken.." [strike 3, you're out!]

I did tell him about the solution, but I also didn't hire him because:

• If you find yourself at a point you need optimize serialization -- it is, in my opinion, a sure sign that you are moving too much data back and forth -- which is most probably, the real problem in your architecture (see the fallacies of distributed computing)
• Depth is not enough, you also need breadth of knowledge
• The more important thing here is that it doesn't matter how bright you are, you need to know there are other bright people walking around the plant who might be just as bright as you are. So that:
  • You don't need to reinvent the wheel every time
  • You are not always right -- others may have a better solution

I call this person "half baked" because he does have potential, but he is in the stage where he thinks he knows best. You always need to be open to learn from other people. The understanding that you don't know everything and that you always need to learn is crucial for an architect.

Posted by Arnon Rotem-Gal-Oz at 12:03 AM  Permalink