IF YOU BUILD IT

... Will they Come?

October 2007

Microsoft has announced it newest SOA initiative, codenamed Oslo. Here are a few observations I have on this announcement:

Let's start with "what is it?" Well it isn't an "it" per se, since Oslo is a bunch of initiatives within the Microsoft offerings.

For one, it is some of the libraries within .NET 4.0 -- specifically the next versions of WCF and WF.

Secondly, it is a bunch of designers and tools that will be part of Visual Studio (beyond VS 2008).
The most interesting component of Oslo will be a new repository to allow version management of models and services. I guess it is safe to say it will be built upon Team Foundation Server (or a subset of which which will be used by both products).

The last part of the puzzle is of course V.Next of Biztalk and something currently branded as "Biztalk Services '1'". As far as I know Biztalk sells pretty, but I think it is both too bloated (e.g., think about the hardware needed to run this in high-performance solution) and builds on the wrong architecture (hub vs. bus). I hope Microsoft makes major updates this time (Biztalk 2004 to Biztalk 2006 mostly innovated around the business activity monitoring. While that's important I think more work on the engine was/is due).

Biztalk services would offer an implementation of some of the SOA patterns I talk about -- service host, workflodize, etc. -- to provide an infrastructure for building services. The relation between "Biztalk 6" and "Biztalk services 1" is not clear from the information provided by Microsoft; hopefully this is just a branding issue and not a tight relation between the products.

On the upside, one of the key persons working on this is Don Ferguson who, before joining Microsoft, was chief architect for IBM's software group. About a year ago I had the chance to hear him talk about SOA and all I can say is that Don is someone who really knows his stuff.

PS: It's amusing to see the press release talks about "model-driven" approach rather than software factories, but I guess that's just nitpicking.

Posted by Arnon Rotem-Gal-Oz at 07:46 PM  Permalink |

I am currently interviewing people for developer positions for my startup (PaperLnx), and as I described my plans to use multiple languages to one of the candidates, he immediately retorted "Why on earth would you want to do that?"

Why indeed?

As I mentioned in the previous post about F# and Erlang, it seems to me that the age of "one size fits all" is ending in a lot of areas. I think that it is also happening for languages.

When .NET was announced, one of Microsoft's messages was that .NET is the platform and that you can choose any language you want to develop in it. Microsoft also worked with partners to back up this
statement and released a lot of languages for the platform (e.g., Eiffel# , Fujitsu NetCobol etc.) vs. Java, which was a single language with a "write once, run everywhere" vision. Microsoft itself released two manages languages -- VB.NET and C# -- and managed extensions for C++. The C++ managed extensions were awkward to work with and only people who really had to used it. The non-MS languages seemed to be niche languages for legacy integration and not much more. VB.NET (we're talking .NET 1.0 here) seemed like something to try to distract VB heads while Microsoft was pulling the rug out from beneath them. Which basically left C# as the only viable language to develop in if you were serious about .NET (at least in my eyes). The "many languages" vision seemed like something to hide the (obvious) fact that Microsoft had to come up with an answer to Java (especially since around the same time Sun also sued Microsoft over the
J++ and JVM implementations)

Five or six years later the situation is a little different. For one thing, we see that the JVM is not just about Java anymore. Just like the original .NET vision, the JVM has become a platform supporting many different languages -- Jython, JRuby, Scala, and Groovy , to mention a few.

The .NET platform is also getting more interesting languages like Boo and Gardens Point Ruby.Net . .NET is also being extended by Microsoft itself (with IronPython, IronRuby, F# ).

Two interesting things you see are:

• Suddenly the languages are cross platform. e.g. I can program in Python and deploy it natively, .NET or on a JVM -- so in a way you get some platform independence.
• The other and more significant issue (to my eyes) is that the languages available on each platform are different enough in meaningful ways -- to a point where it is getting worthwhile to use more than one language in projects. Here are few examples from around the blogsphere that demonstrate this:
  • Neal Ford shows an example using the (excellent) Mocha mocking library (in JRuby for testing Java classes). So the dynamic properties of Ruby help make the testing simpler for the statically typed Java.
  • The BOO Build System (via Ayende) -- which is about building a DSL for builds (a la rake ). Again we see taking advantage of the properties of a dynamic language this time to help with .NET languages.
  • (not) Dennis Byrne shows how you can call Erlang from Java -- which demonstrate taking advantage of the concurrency features of Erlang from Java.

When it comes to the difference between general-purpose languages (i.e., Java, C#, VB.NET) we mainly see syntactic nuances. When it comes to dynamic languages vs. static languages vs. functional languages, the differences are more meaningful. What has changed now is that both platforms and open standards (e.g., REST over HTTP/WS-*) enable better integration and increase the motivation to use the "best tool for the job" approach over "the everything looks like a nail" approach .

I think the time for language pluralism is ripe. In my solution this would probably translate to a mix of C/C++, F#, C# and Ruby. What about your projects?

Posted by Arnon Rotem-Gal-Oz at 06:05 PM  Permalink |

A couple of thoughts on functional programming in general and on F# and Erlang in particular.

Erlang? Is it worth it?

I read Joe Armstrong's Erlang thesis and other Erlang related material with great interest. After all, Erlang claims some very compelling architectural benefits in the areas of scalability and availability. However one thing that prevents me from digging too deep into it is the utter ugliness of the language vs. (for example) Ruby which is very compelling.

I also tend to agree with Otaku that at least today you can achieve this resilience with other languages as well.

On the other hand, we start to see some really interesting things built on Erlang like:

• RabbitMQ, which is an AMQP implementation (. The Advanced Message Queue Protocol).
• MNesia persistence engine
• Yaws web server etc. (You can see here one guy who even likes Yaws + MNesia better than Ruby + Rails).

Well, I am still on the fence for this one, but I guess I will take a deeper look at Erlang eventually.

By the way, while other functional languages also have irritating syntax (e.g. Scheme). Some look more intereseting (like F# or OCaml).

Which brings me to the next tidbit...

Functional Languages Moving Mainstream

Functional programming is an old paradigm (see, for example, this paper from 1984 Why Functional Programming Matters explaining the advantages of functional programming over structured programming.)

For a decade or even more and even today the object-oriented paradigm has ruled the software development world. In a way, and as part of the end of the "one size fits all" paradigm (I also mentioned this trend here). We also see more pluralism for languages so we get more dynamic languages vs. static typed one, and we find a place for functional languages and not just object-oriented ones. (I'll expand more on that in another post).

Anyway, yesterday S. Somasegar (MS VP of DevDiv) announced on his blog that F# will be "productized":

We will be partnering with Don Syme and others in Microsoft Research to fully integrate the F# language into Visual Studio and continue innovating and evolving F#. In my mind, F# is another first-class programming language on the CLR.

Of course, Microsoft is not the center of the universe, but when a company like Microsoft chooses to bring something closer to mainstream it is a significant move which, in my opinion, shows that functional programming is getting more traction.

Posted by Arnon Rotem-Gal-Oz at 05:07 AM  Permalink |

Well it seems setting up a new company can keep you busy at times--which is my official excuse :) for being unusally quiet last week. Hopefully I'll have a little more free time this week.

Note: I am talking in this post about roles and capabilities., i.e. when I say "architect" I mean someone who has the capacity to be an architect (e.g., as demonstrated in the architect training suggestion I made) and takes that role. On a specific project you may have a person that performs multiple roles such as an architect and project manger or an architect and developer while other project warrant one or more full time architects.

Not all projects need architects. There, I've said it. Not all projects need architects and I am not talking here just about trivial projects. There are cases (maybe even many cases) where you can get by with what I call "off-the-shelf" architecture -- maybe with a few adjustments that any master developer (i.e., seasoned and experienced developer) can handle. For instance a lot of websites can do pretty well by using Model-View-Controller (or a derivative of that) along with a simple O/R mapper such as active-record. In fact a lot of them do when they use a framework like Rails that made these architectural choices for them*. Another example is the vanilla 3-tier architecture provided by vendors (such as this one by Microsoft). Yes, when you take something off-the-shelf the result might not be optimal but that doesn't mean it isn't sufficient. You
just have to be aware for the tradeoffs...

Another point is that "design" is not an exclusive architect thing. A developer is not a good developer unless she also known about  proper design. mastery of technical details of the language without understanding the wider context of design will just help you code a lot of crap faster.

Having said, that we need to consider a few issues -- When do you need architects? What do they Do? What's their relation and interaction with the developers ?

When do you need architects?

It is sometime a fine line between a project that can get by with an
"off-the-shelf" architecture and one that needs an architect. It would be nice if we could have something like a litmus test that would tell us if architects are needed or not. I don't have one. The closest thing I could get to this is something I call the SCLR test (pronounced "scaler". SCLR stands for Size, Complexity & Limited Resources.

• Size -- well if you are going to have something estimated at 1000 man-years or dozens of teams. I think it is pretty obvious that you can't just use something which isn't made to fit. If anything there's a need to divide the work between the teams in a way that would make sense so that you wouldn't get a big-ball of mud. There's also a lot of need to coordinate the efforts and keep the big-picture inline. Personally I think that it doesn't have to be a huge project to warrent some architects involvement. Since as Fred Brooks notes the number of interactions grows exponentially as we add more people. In my experience trouble starts even with more modest numbers -- more than 4 or even 3 different teams working concurrently is probably a good number to start thinking  about architects.
• Complexity -- There are many signs for complexity in a project. The vision statement can provide a hint. "Let's design the software to support the next mars mission", "best CRM platform ever" -- an ambitions project will not make-do with "average" architecture. Size (which  I already mentioned) is also a sign of complexity, while previously I talked about size of the project, the size of data, users etc. is also relevant when we're thinking about complexity . A lot of external interfaces is another sign. Integration doesn't seem very complicated, until you actually try to pull it off. When you have to do a lot of that in a project that's complex. And there are many other signs.
• Limited Resources -- Naturally every project has limited resources, but limited resources should be considered as a sign for architect involvement if the resources are extreme. When resources are extremely limited the tradeoffs that have to be done are more meaningful, which is why wed want people who can help with that (i.e., architects). For instance in a projects I worked on in the past we had a lot of availability and performance requirements on one hand but only so many "U"s in the rack and even limited electricity to make all this magic happen. This turned something which otherwise was a relatively standard IT project into something a lot more challenging.

Assuming I manage to convince you that some projects can't just choose one of the available blue-prints and need some more work -- the next step would be to convince you that architects are better suited to solve this than developers. I'll try to do that in the next post on this series where I'll explain what (I think) architects do and their place in the development team.

Posted by Arnon Rotem-Gal-Oz at 06:45 PM  Permalink |

Pete Lacey has a post called What Is SOA? where he defines SOA as follows:

• Network Oriented Computing (NOC): An approach to computing that makes business logic available over the network in a standardized and interoperable manner.
  • Service Oriented Architecture (SOA): A technical approach to NOC that has a non-uniform service interface as its principle abstraction. Today, SOAP/WS-* is the chief implementation approach.
  • Resource Oriented Architecture (ROA): A technical approach to NOC that has an addressable, stateful resource as its principle abstraction. Today, REST/HTTP is the chief implementation approach.
• Business Service Architecture (BSA): An unnecessary term (also not an architecture) that tries to make the obvious something special. Aka, business analysis. Aka, requirements gathering

I'm sorry, but I beg to differ. The first thing to note (again) is the architecture vs. architecture style differentiation I mentioned in a previous post. (You can see a similar definition by Stuart Charlton.) Here is a quick reminder :

Software architecture is the collection of the fundamental decisions about a software product/solution designed to meet the project's quality attribute requirements. The architecture includes the main components, their main attributes, and their collaboration (i.e. interactions and behavior) to meet the quality attributes. Architecture can and usually should be expressed in several levels of abstraction (depending on the project's size).

An Architectural style is a blue print that can be used when you desing an architecture. An architectural style defines some of the components and thier attributes as well as place constraints on how they can interact.

My claim is that SOA is an architectural style for distributed computing which puts extra emphasis on the interface (and hence gets the easier interoperability). Okay, if SOA is indeed an architectural style, we
should be able to define it as a set of components, interactions and attributes. Well, I already did that a while ago (in a paper called "What Is SOA Anyway?"). And while it may not be perfect, I think it is a reasonable definition all the same:

SOA is an architectural style for building systems based on interacting coarse grained autonomous components called services. Each service expose processes and behavior through contracts, which are composed of messages at discoverable addresses called endpoints. Services' behavior is governed by policies which can be set externally to the service itself.

You can see the above mentioned paper for a little more detail on each of the components.

ROA, in my opinion, is just a re-branding of REST so that it would be easier to discuss it as an architectural style and not connect it to the HTTP implementation -- which is what a lot of REST proponents are doing.

By the way, as I pointed out before, there are a few other important architectural styles that are related to distributed systems like Event driven architecture, Spaced based architecture, peer-to-peer etc.

As for "Business Service Architecture". I like to think about that as "SOA initiative" as in the strategic decision to try to implement an SOA in an organization while trying to achieve the more nebulous traits like
business and IT alignment etc. (which is why it is nether architecture nor architecture style).

Posted by Arnon Rotem-Gal-Oz at 07:21 AM  Permalink |

In a recent post Steve Vinoski said:

Frankly, if I were an enterprise architect today, and I were genuinely concerned about development costs, agility, and extensibility, I’d be looking to solve everything I possibly could with dynamic languages and REST, and specifically the HTTP variety of REST. I’d avoid ESBs and the typical enterprise middleware frameworks unless I had a problem that really required them (see below). I’d also try to totally avoid SOAP and WS-*.

It is easy to dismiss this as just another yahoo who goes against conventional wisdom until you remember that Steve spent more than a decade working in Iona in leading roles like Chief Engineer of product
innovations and helped develop some of the middleware standards for OMG and W3C.

Well, I guess that's becoming an epidemic now :). Just recently we had Michael Stonebreaker talking about the RDBMS demise, Pat Helland talking about life beyond distributed transactions. and now Steve on ESBs.

That trend aside, I think Steve is doing throwing the baby out with the bath water. The dream of a single infrastructure for an enterprise is ludicrous enough (remeber Peter Deutsch and the "The network is
homogeneous" fallacy). But if you drop the "E" from the ESB moniker you get a valuable middleware which is very usable in many situations and not just legacy system integration. For instance one thing that is missing form "HTTP variety of REST" implementation is reliable messaging. Location transparency is harder to solve with HTTP etc.

Another problem I have with the current approach of Steve is that he is replacing one dogma (EBSs are good) with another (ESBs are bad use Ruby, REST). This is not a healthy approach. The solution should match the problem, that's probably the primary reason why we need architects after all.

Posted by Arnon Rotem-Gal-Oz at 06:49 PM  Permalink |

In a previous post on the subject, I promised to expand on suggested content for a "Distributed Systems Architectures Workshop". So here is a short drill-down:

Even though most of the time should be spent on working, designing, and evaluating architectures, there's probably a little room for theory.

Module 1: The Basics (probably not more than half a day)

• What's software architecture
• The software architect role
• Activities
• Scenario based architectural design
• (documenting software architectures)
• Agile SDLC and architects

Module 2: Distributed Systems Background

Understanding the Fallacies of Distributed Computing Distributed architectures styles. It is important to understand the different architectural styles that can be used to implement distributed system. Within this, topics like clustering, computation and data grids, messaging , publish subscribe, etc should also be discussed.

• Client-server. The most basic distributed architectural style. It is based on the N=1 premise and isn't fit for most of today's challenges. However, it is still an option for some types of projects.
• Pipe and Filters . Not necessarily a distributed style, but it can be applied in distributed space.
• N-Tier. That's actually a moniker to anything where N>2 but usually it pertains to 3-tier architecture (front-end, server, database) or the Internet 4-tier version (client, webserver, application server, database).
• Event Driven Architecture
• Service Oriented Architecture
• REST
• Space-based architectures. Like JavaSpaces and its implementations such as Blitz (open source) and Gigaspaces (commercial).
• Peer-to-Peer. You know that's what all those file sharing tools use.

Distributed Consensus

• 2-phase commit, used by XA and COM+ distributed transactions.
  
• 3-phase commit, considered a non-blocking protocol (vs. 2-phase commit which is a blocking protocol).
  
• Paxos commits
  
• Sagas
  
• Eventually consistent (BASE) -- Basically Available Scalable/Soft state Eventually Consistent. An alternative to distributed transactions used by a lot of Internet-scale companies (see a post I made on eBay's architecture )
  

Module 3: Workshop. Most of the days should be focused on actually working to design architectures.

I would think that this would be handles best by working in groups. e.g. having each group focus on one architecture style.

The groups would be given a scenario which covers some architectural concern (integrity, performance, scalability, availability etc.) and would try to design strategies to handle the scenarios within the constraints of the architectural style. Present that to the other groups and then have a facilitated discussion on the
pros/cons of each strategy. The scenarios should be based on a large enough story to allow meaningful architectures to emerge (e.g. you can see the 10 Scenarios in my SOA Patterns Presentation)

Any comments or other ideas for what's needed for this kind of a workshop are welcomed.

Posted by Arnon Rotem-Gal-Oz at 09:54 AM  Permalink |

A side effect of my decision not to become an independent consultant at this time means that I have to shelve some of the projects I was considering. One of these projects was to create a training program for software architects which I was discussing with a couple of training centers here in Israel.

Since It seems I am not going to promote it, I thought I'd share what I think a training program for .NET/Java architects should look like in the hope that someone would find it useful and do promote it (or parts of it).

Soft Skills
The way I see it architects needs a bunch of soft skills to be able to perform their roles. Here is the list I identified in the past. (By the way, I began a series of posts on each of these skills and never got to finish it -- maybe it is time that I will :) )

• Leadership. Influencing others to accomplish tasks and following your guidance
• System thinking. Understand decisions and constrains in the wide scope pertaining to whole of the solution at hand. This includes
  the ability to abstract problems.
• Strategic thinking. Understanding decisions and constrains and their alighments to the overall business of the company.
• Organizational politics. Understand the environment you operate in and how it influences you.
• Communications. Making sure you get your point across.
• Human relations. Understand the "people" aspects or human
  factors and dynamics. This includes things like negotiation, pragmatism etc

I am not sure if you can teach all of them, but few courses that can help (in my opinion) include:

• Negotiation/Persuation. Anything based on Getting to yes, Getting past no, and The power of a positive no should be fine.
• Presentation Skills. While getting the architecture and technology right is what matters, if you can't explain it to the different stakeholders you're toast.
• Strategic Planning. This has to do with the vision thing I expect architects to manifest. Note that having a vision should not be confused with future-proofing a solution. future-proof means excess work not needed. Having a vision is knowing where you want to end -- it can still be perfectly valid to completely re-write your applications along the way

Project Management
While the architect is not the project manager (mostly anyway), I think understanding the constraints coming from the project management point of view is very important.
Since most environments call for a mix of agile and formal disciplines (hey, you've got to be pragmatic). I would train architect both in SCRUM and RUP (or some other formal methodology). Also while not all environments needs this I would give an 2 days overview of important standards. The first would be IEEE 1471, which defines a standard for documenting software architectures. I would also teach ISO 90003 and CMMI.

It should be noted that the ISO 90003 is much better than the previous incarnation (ISO 9003) as it basically lets you define what you want to do to cover the different areas. The standard just helps you make sure you think about the various parts of project management (requirements, environment etc.). For instance, I
demonstrated how key areas of 90003 can be mapped to SCRUM to get it approved on my last project.

Languages, Design, and Patterns
I would want the .Net/Java architect training program to include at least two of the following languages:

Ruby
Scala
Erlang
F#
Python
Groovy
OCaml

The reason for this is that these languages have different design goals than .NET and Java so learning them gives you additional perspectives and broaden your horizons for other ways of thinking (even if you don't
use them in your project directly). You might have noticed that there's no .NET or Java training here. The reason for that is that's a prerequisite as far as I am concerned. You should master at least one object-oriented principles -- hopefully aspiring architects already know this. However, I often see people who discovered some of the principles by themselves but haven't heard about all of them. I am talking about
principles such as Liskov Substitution, Principle , Open Closed Principle, Single Responsibility Principle , IoC containers, Don't Repeat Yourself, and YAGNI (I summarized my opinion on most of them in this paper)

The next step is to cover some design issues like Domain Driven Design, UI Design, Database modeling, Database alternatives l(after all the database is dead, right? )

Advanced Design Patterns
When most people hear the term design patterns they think about the GoF patterns. There are however literally hundreds of design patterns. Some of them are even worth learning :) . For instance, there are patterns for concurrent and parallel systems like Proactor, Reactor Half-sync/Half-Async etc; Workflow patterns like Cancelling partial Join, Recovery Action etc; SOA patterns (okay, so I am still working on that :) )

Architecture Workshops
Another important part of the training, in my opinion, is to do some workshops and actually try to apply some of the material covered.

• Architecture Evaluation. Workshop 2-3 days. It is probably worthwhile to delve a little on scenario based evaluation techniques such as LAAAM and ATAM. While I prefer evaluating architecture in code, the scenario based thinking is very valuable for eliciting architectural requirements
• Distributed Systems Architectures workshop. I'll expand on this in the next post

Lastly, there are also a few miscellaneous subjects like architect 101, the SPAMMED architecture framework, Agile architecture, Behavior Driven Design , common frameworks (though hopefully this would not be needed ) like Spring/Spring.Net, Hibernate/NHibernate, iBatis etc.

PS: Note that there are a few architect training programs available out there . One is offered by the Software Engineering Institute (SEI) and includes a six courses. SEI program seems to be focuses on formal
sides of architecture as it includes courses on documenting software architecture and ATAM (You can see an old presentation I have on ATAM here).

Dana Bredemeyer also offer architect training. Dana offers several workshops that cover the software architecture profession.

TOGAF (which is more of an enterprise architecture framework) offers both a certification and courses
Lastly, IASA is considering creating a software architect program and has a few courses in development

If you know any others I'd be happy to hear about them.

Posted by Arnon Rotem-Gal-Oz at 12:44 PM  Permalink |