IF YOU BUILD IT

... Will they Come?

December 2006

Blogjects are an interesting concept somewhat related to my previous post.

Blogjects is a neologism short for "Objects that Blog." The term was coined by Julian Bleecker to denote objects that tell the world about their experience, and also react to comments or messages left to them. Julian has a nice paper that sums the subject; see "A Manifesto for Networked Objects: Cohabiting with Pigeons, Arphids and Aibos in the Internet of things". On top of the blogging characteristic, Julian says Blogjects manifest three (rudimentary) characteristics :

• They track and trace where they are and where they've been.
• They maintain histories about their experiences.
• They have some form of agency; they can participate, stir up action, and show some form of autonomy.

Julian then takes this toward the idea of "things" that blog. He describes an example done by Beatriz Da Costa using pigeons that were equipped with GPSs, air pollution sensors, and wireless communications, mashed up with a map service to track air quality.

The reason I am telling you about this, however, is that if we think about the characteristics above and the EDA-enhanced SOA mentioned in my previous post, you can see that there are distinct similarities.

The services concept in the previous post has services that "blog" their state, know their history and have increased autonomy , and manifest the participation and action taking characteristics mentioned above. I think that is an interesting approach to take when you think about your publication contract for a service.

By the way, if you want to make your objects literally blog, take a look at RSSBus which, as its name implies, is a service bus implementation that utilize RSS.

Posted by Arnon Rotem-Gal-Oz at 05:26 PM  Permalink |

In the previous post I talked about the misalignment between SOA and BI. I briefly explored three options to incorporate BI and SOA without breaking SOA boundaries, then added that they all fall short. The aim of this post is to look at the fourth alternative which, as you probably guessed by now, is my favorite approach.

In my opinion, the best way to tackle BI in SOA is to add publication messages into the contract. By "publication messages", I mean that the service will publish its state either in a periodic manner or per event to anyone who listening. To make this complete, you can add additional requests/reply or request/reaction messages to retrieve initial snapshots. Following this approach, you get an event stream of the changes within the service in a manner that is not specific for the BI. Having other services react on the event stream can increase the overall loose coupling in the system, but that's another story.

Why is this better than the other approaches? For one thing, you can get a good picture of what happens within the service. However the contract is not specific for the BI and can be used by other services to cache the service state (thus increasing their own autonomy), for reporting (you can see an early draft of the aggregated reporting pattern), and for BI purposes. By working against a steady stream of events, the BI platforms can analyse treands, keep history and get the complete picture they need.

The approach above is sometimes referred to as "Event Driven Architecture" (EDA) and while I (and others) see EDA as another facet of SOA, not everyone agrees. Gartner, for instance, sees EDA as another paradigm and SOA just for request/reply, or client/server. That said, Gartner recently published a paper that calls this "Advanced SOA". I tend to agree more with the "advanced SOA" definition and don't see a contradiction with EDA and the SOA definitions. We are still using the same components and the same relations only adding an additional message exchange pattern into our toolbox. (I guess it might be good to post my definition of SOA. Maybe I'll do that soon.)

A note on implementation: If you are implementing SOA over an ESB that is rather easy to implement, (most ESBs support this out of the box) using WS* stack, you have WS-BaseNotification, WS-BrokeredNotification and WS-Topic set of standards. When using REST you need to implement publish/subscribe by yourself. Once you generate the event streams on the BI side you may want to look at CEP (complex event processing) tools to analyze and react on the events.

Posted by Arnon Rotem-Gal-Oz at 11:07 PM  Permalink |

Service-Oriented Architecture is about autonomous loosely coupled components. These traits give you lots of benefits but it also means that services have internal data. Data that you don't want to expose to the outside as exposing it will decrease autonomy and increase coupling. This is why services only expose data and processes via contracts and not their internal structure.

That is all fine until you start to think about business intelligence. The cornerstone of any business intelligence initiative is gathering, collecting and consolidating data from all over the place. Once you have the data, you can use tools to analyze it, data mine it, slice, splice, aggregate, and whatnot. Traditionally BI builds on ETL (Extract, Transfer, Load) which goes directly to the database of the involved sources.

And here lies the problem: On the one hand we have services that want to keep their data private, and on the other we have a datamart or warehouse that wants that data badly.

• If you go with traditional ETL, you introduce coupling into your service.
  
• If you only rely on contracts that were constructed for business processes you may be missing out on important data.
  
• If you build a specific contract that exposes "all" the data you are back at the point-to-point integration -- solving point-to-point integration is one of the reason we want SOA in the first place.
  

The second option seems to be the most reasonable choice of the three -- but it also has several problems. One problem is that the BI needs to know about all the contracts. The second was already mentioned -- important data might be missing. The third problem is that the BI system need to fetch data from the services which means it may miss out on data in the intervals between request. On the other hand, too frequent requests and you can congest your network.

Clearly we need a fourth option -- and I'll talk about that option next time

Posted by Arnon Rotem-Gal-Oz at 03:48 PM  Permalink |

I came by a presentation (via Shahid Sah's blog) by Ron Jacobs on the Software Architect's Role. In this presentation, entitled Architects and the Architecture of Software, Ron compares the architect's role to that of an explorer, advocate, and designer.

While I can go for the designer bit, I don't like the heavy analogy to building architects (I know, I know I have that as well in my software architecture presentation -- but it is no longer there in the next version).

However, I would personally replace "advocate" with "mentor", and "explorer" with a "polymath" or "Renaissance" man. I'd also add a leader and visionary (although Ron mentiones that as part of the discussion on explorer).

An advocate is someone who observes, listens, and gives advice. On the other hand, a mentor is someone who helps others reach the right decisions and help them learn and evolve. I think that has much more value. I want a Socrates, not an Alan Dershowitz, on my team

An explorer looks for new grounds and is a bit of a visionary. However, a Renaissance man is both knowledgable and inventive. As a development manager, I rather have someone who knows what he is doing, understand the wider perspective, and find solutions to my problems -- not someone who would take me on a road to uncharted territories. I'd take Leonardo Da Vinchi over Columbus ( who accidently gave the competitive edge to Spain and didn't even know it) any day.

A visionary and leader is also important. You want someone who is able to look far and that can help your team get there. I guess that is somewhat akin to an explorer (in the sense Ron mentions) but again I'd rather have a Martin Luther King than a Columbus (though being lucky wouldn't hurt either).

But hey, that's just my opinion :-)

Posted by Arnon Rotem-Gal-Oz at 03:41 PM  Permalink |

Can you develop software without having an architect on the team. Sure you can, but how would that affect your software quality? I guess the more interesting question is can you develop quality software without an architect?

The way I see it there are three types of projects that can work without architects:

• Simple systems. That's the most obvious type. You just have talented developers (for a simple enough system you might not even need that) and you just develop a solution. For example, there are many small businesses that have custom solution that fill all their needs and while these architectures might not win prizes (you know, five users VB6, and access systems where all the logic is embedded in the forms and the like) -- they just work and they are good enough for their purpose.
• Implicit architect. In this type the development team may not have an architect, but they are influenced by the architectural decisions of the architects of the tools they use. A good example for this is Rails. When you choose rails to develop your web site you may not spend too much time thinking about the architecture of your solution, but you are get all the architectural decisions made by David Heinemeier Hansson, e.g. using ActiveRecord for O/R mapping.
• Reference Architecture. This is similar to the previous type in the sense that someone else does the architecture for you -- only this time the decision is more conscious. You decide to use an architecture someone else designed (We're using "3-tier" architecture is an example for relying on a reference architecture.

All these options can save you time. The question is will you regret that saved time later when your solution crash and burn (or you'd have an architecture that is more complicated (vs. what you really need)). On the other hand, if the architecture you use is close enough to to what you really need you do save time and money.

It is best to have qualified architects (even if they only use is make the recommendation to use one of the methods above and the architect is dismissed). Architects should be able to help deciding on the alternatives. Without an architectural perspective you are just guessing that a solution is right, Again, whatever your decision is -- don't wait too long until you prove it with working code.

What do you think?

Posted by Arnon Rotem-Gal-Oz at 06:45 PM  Permalink |

I am attending a "transition to agile project management workshop" by Johanna Rothman. It seems most of the attendees are from large corporations -- Cisco, CA, SAP and the like -- which in my opinion is a good sign. I see a lot of benefits in agile values and it good to see interest from large companies.

Many of the projects me and the other attendees seem to have are rather large and include all sort of challenges for introducing agility; e.g., few participants mentioned off-shoring some of the development, another mentioned hardware integration, etc.

To me, that is just a reminder why JEDUF is important. I find that in projects that are large or overly complex "sacrificing" one, two, or even three iterations for handling technical risks and forming a candidate architecture goes a long way (and I don't care if this makes my project not agile. I am fine if it is pliant, lagum or what-not). Some things are harder to add or change at later iteration. prototyping the user interface (or more importantly the overall user experience) is one example, others are related to availability, security and basically many of the system's quality attributes. This doesn't mean that you "implement by architecture" as Johanna mentions in a recent blog. You still want to implement full features going through the different layers and/or components of the architecture since evaluation in code is probably the best way to prove your candidate architecture actually fits.

Posted by Arnon Rotem-Gal-Oz at 03:20 PM  Permalink |

My editors at Manning think that Chapter 1 of my SOA patterns book is not up to snuff.

Basically they say that the chapter talks about too much theory vs. the other chapters, which contain more down-to-earth stuff (e.g., Edge Pattern, Aggregated Reporting Pattern, Decoupled Invocation Pattern .

You can also take a look at the full (draft of) table of contents here). Also they’ve said that I spend too many pages explaining what architecture is or talking about distributed systems before I get to SOA -- which is the topic of the book.

The way I see it, understanding architecture and distributed systems is essential to understanding SOA (from the development side anyway; for instance, when you want to design and build services). For example, the discussion on quality attributes explains how you can use scenarios to find architectural requirements (and each pattern then has a section on relevant scenarios to help you find if the pattern is applicable to your needs)

I would be interested in hearing what you have to say (either as comments here or e-mails to me) about the chapter’s structure and content (considering that most of the book will be patterns like the Edge pattern).

Posted by Arnon Rotem-Gal-Oz at 01:05 AM  Permalink |

I started blogging here at Dr. Dobb's back in April, and now in December I've hit 100 posts. I guess it is a good time to highlight some of the posts I think were more notable.

What is Software Architecture (and a second part) talks about some of the characteristics of software architecture and provides my definition for it. The second part highlights the importance of quality attributes.

Should Architects Code (Part 1, Part 2, Part 3 and the related The Architect's
Role: A Holistic View). You already know the answer -- it depends.

Architecture Styles, explaining,well er, what are architectural styles.

I had a series on the Fallacies of Distributed Computing which I combined into this paper (already downloaded more than 10,000 times).

I had a series of posts on Object-Oriented Principles (Single Responsibility Principle, Open Closed Principle etc.) which was initiated by the 7 Deadly Sins of Design post, again, you can get all the articles combined as a single paper.

Another series that ended as a paper was on a discussion on the architectural dilemma of O/R Mapping. It got a fourth part after Ted Neward posted on how O/R mapping is the Vietnam of computer science.

I have an unfinished series on the Architect's Soft Skills (hopefully I'll end it before I'll hit my 200th post :). I already posted the Introduction, Leadership, Systems Thinking, Strategic Thinking and Organization Politics. The next post will talk about communications skills.

I also started to publish (unedited drafts) of SOA patterns from the book I am currently writing. The first pattern I published is the Edge Component Pattern. I think I'll add a new one every month

A few other interesting posts (in my opinion, well you know) included an application of Liskov's Substitution Principle for SOA contracts, Tier Is A Natural Boundary and The Ultimate Answer.

So far I am having fun with this blog, and I hope you are not too bored either.

P.S.
This is probably a good time to remind you that you can send architecture or design questions to ask@rgoarchitects.com and if I think they are general enough to interest this blog readers, I'll give you my view (and hopefully the answer) here on this blog,

Also, I am very grateful to everyone who has taken time to send me feedback (good or bad), keep up the good work :)

Posted by Arnon Rotem-Gal-Oz at 11:11 AM  Permalink |

I’ve received a few comments regarding my post about how optimizing serialization speed can be a good option for making the communication between two tiers (client-to-server or server-to-server) "better".

My claim is that in most cases optimizing things like serialization is a poor excuse for not designing the communications between tiers/services etc.

For one thing, networks are slower than memory and CPUs, and serialization is just one of the things you need to do. You also need encryption, signing, logging, and the like. And if you think about the whole cycle from a request to a reply, there’s also authentication, authorization, possible transformation (from external formats to internal) and what not. Is optimizing serialization really the answer to all these?

I guess that in most cases it all boils down to the problem of distributed objects. A lot of developers and architects are used to do object-oriented design -- and for a good reason. It has proven to be a very good paradigm for developing systems -- until we got to distributed systems where it didn’t really work well (which is why we need newer architectural styles like SOA).

Objects are not meant to be distributed. OO assumes locality. In classical OO you even control the lifetime of the objects you use (techniques like Dependency Injection allow us to lift this barrier). You assume the other object is there, so you can just call on its methods and read its properties etc. if every method call translates to a call on the wire. Then yes, you will soon find yourself optimizing stuff like serialization, but as Miagi once said in The Karate Kid the "best defense is no be there" don’t get to these situations. If you consider that a tier is a boundary , then you understand that the better approach is to carefully consider what data needs to be moved back and forth. If you design what data will travel, in most cases you can use the verbose XML format and not JSON or proprietary binary serialization. You won’t mind serialization time that much as it is just one link in the chain that takes more time.

One of the biggest advantages of SOA (as I see it) is that it explicitly makes you think about the distribution aspects of the system (vs. OO which doesn’t).

If you are building a simple solution, then it might not be worth your while to build a scalable distributed system. You may not need to integrate your solution with other system so you can serialize datasets, expose internal structure or any other behavior that will inhibit future system growth. But in this case you would probably won’t need to optimize serialization as well since you have modest requirements anyway.

Granted, there may be extreme cases where it just isn’t enough to think about what data should be transferred. In one system we built we had to transmit data through something like a 1200-baud modem, and no matter how much we tried to minimize the traffic, eventually, we had to find some smart way to compress the data -- even in this case the bottleneck was the amount of data we can push over the wire in a given time and not the time it takes to prepare the message.

So in my opinion, optimizing serialzation is a good answer -- but to the wrong question

Posted by Arnon Rotem-Gal-Oz at 02:05 PM  Permalink |