THE SOFTWARE SIMPLIST

Web Services Wisdom.

November 2006

About a year and a half ago I wrote a short article on what ESBs are for. There are a certain family of problems that ESBs are designed to solve. If you were to use an ESB product that does not solve these problems but instead solves others, you might end up in the kind of trouble I described in the article.

The bottom line appears to have always been that it is up to you to understand your domain and its specific technical requirements, then to find the right mix of buy and build under your budgetary constraints to answer them. Turning to any solution, or mix of solutions, without going through that first analysis step will probably give you yet another overly-expensive legacy nightmare that doesn't answer your business needs. Well, that's what happened to me when I did that, and let me tell you, that's not a career-bolstering move.

When I read things about how ESB prevents SOA, it sounds to me like we're arguing over what tastes better - apples or oranges, when our stakeholders want to take the restaurant to the next level. Apples and oranges are probably part of the solution, but we should probably be operating at a higher level. If you want to call that Enterprise Architecture, fine. But let's not be rushing to close deals for SOA-Suites with vendors so that we can get all sorts of discounts before we really understand what we need to support where the business is going.

Just to sum things up at the technical level. In my opinion, the Bus architectural pattern can help reduce spatial coupling between the services you designed but won't matter much unless you handled the temporal coupling between those services as well.

Posted by Udi Dahan at 05:33 PM  Permalink |

Every time the dialogue around REST picks up I try to listen for anything new, but I usually end up being disappointed. I’ve heard about how REST is inherently scalable because the internet is, supposedly, based on REST. I’ve heard how it’s simple. What I haven’t heard is how I can use it to build large-scale distributed systems that are controlled by complex logic.

It seems that the point about scalability is only made in terms of GET operations. The problems I experience in terms of scalability don’t manifest themselves so much in the reads, but rather in the updates, and how those updates affect the reads. There are business rules that control which data should be returned and when. All this has been known for quite a while and has been modeled around transaction isolation levels in the OLTP sense, and needs to be handled at the application level when it comes to intelligent caching. I haven’t heard anything about REST that deals with this.

The topic of simplicity that is raised when moving towards a Resource-Oriented Architecture (ROA) leaves quite a few loose ends for me. If, as a result of updating one entity, other entities are affected how is that information propagated back to the caller? Is a PUT (or POST, I never know what to use when) on one resource allowed to PUT (or POST) on another? If it does, whose responsibility is it to send the total results to the initiator? If not, does the caller need to be aware of all the possible interactions between resources and when they are relevant?

Let’s say I have a simple rule: When the sum of all orders for a customer in the past quarter increases above X, the customer is to become a preferred customer. Do I PUT the new order resource to the customer resource, or should I have a generic orders resource that is “partitioned” by customer Id? Once the change to the customer occurs, should I PUT some change information somewhere so that the initiator can find out where it should go look for the change? And how should I be handling transactions around all these resources?

If REST wants to conquer the enterprise, it had better get down from its “internet-scale” high-horse and start talking in terms of OLTP. Furthermore, some case studies would be nice. I’d like to see the architect who’s willing to bet his career on an architectural style that has yet to be proven in his domain. It’s not going to me, let me tell you that much.

Posted by Udi Dahan at 02:22 AM  Permalink |

The other day I ran into a blog post titled “Master Data Management – just another load of inaccurate data?” and got this feeling of deja-vu. This idea of having a single location where the authoritative definition of an entity lived is not new. Enterprises have been going back and forth on the continuum of centralization-distribution for years. The whole SOA and BPM fads that have taken the world of IT by storm appear to have stirred up this topic yet again, this time incarnated as “Entity Services”. [Originally published here.]

For those of us who went down the path of centralization, and failed, I probably won’t be bringing any words of wisdom. But for those who haven’t, maybe this warning will save you some grief – I only wish I knew where to look back when I was doing this.

In the example shown in the blog post, the problems that occur when customer data is duplicated in multiple back-end systems are discussed, as are those of employee data. The source of my failure rested in the belief that there is such a thing as a coherent entity. Only later did I find out that for sales different data was important than for billing. It was not so much the issue of data that got us in trouble, we just created a super structure for all the data, but that the business rules were so different – at times appearing inconsistent and contradictory. We finally had to ditch the centralization effort – there were too many worlds colliding that couldn’t be reconciled. Entity services appear to be directing us down that same logical, yet misleading path.

The author quotes some popular beliefs about SOA that appear to make matters worse:

“The problem is that I don't see how MDM helps SOA. SOA needs to work with disparate backend systems largely intact, benefiting from the logic they already provide. It should not be trying to replicate or rebuild the business logic in underlying systems, since if you are going to do that you might as well rip and replace those systems, not duplicate the logic in the integration layer.”

This view of SOA being bound to reuse at the expense of quality and effectiveness is misguided. Proponents of SOA tout reuse as a benefit that will occur in the future, but never is reuse introduced as a constraint in designing an appropriate architecture. This view of SOA is understandable given the EAI history so many bestow up SOA. Personally, I do not view SOA as an integration technique. Rather, I look to the principles of SOA to help in finding the inherent fracture points in the business domain so that services can be designed to handle each section of the domain. Services designed in such a way will be as loosely coupled as is possible.

I would suggest that we as an industry focus more on learning from lessons of the past than jumping on the latest bandwagon and betting the future of our projects and companies on unproven, vendor-driven acronyms. Maybe then, we’ll be able to codify those lessons in such a way that we’ll have repeatable successes worthy of acronyms.

Posted by Udi Dahan at 02:08 AM  Permalink |

Here's another question I received quite some time ago that has some food for thought in it. Wes asks:

"I’ve built up a service framework in which various service definitions (capabilities) are exposed as web services which are connected via a service bus. I have two public interfaces that communicate with the service bus, an asp.net 2.0 web application and a api web service that business partners use for service access and system integration. All capabilities and both public interfaces talk to the service bus via a service bus proxy. I currently have two service bus proxies … one internal and one external. The difference between the internal and external proxies are the internal one exposes all methods of all capabilities and the public proxy only exposes methods that do not require service orchestration (coordinating calls between various capabilities). Currently this means that the public interfaces are “forced” to go through a workflow in order to do certain things, whereas internally, the workflow capability can call any method of any capability. The goal is to try and reduce the number of “wrong things” that a client could do and to reduce the burden of the client to know which order to call things in and having to implement compensating logic when errors occur.

Any advice, experiences, or resources are most appreciated."

Like I said, lots to digest. I don't think that I understood his scenario entirely, but I think that the following bit of advice is quite pertinent, originally posted here.

In my opinion, the service bus should not have different APIs. The bus I use has 3 main methods:

Send(IMessage msg);
Reply(IMessage msg);
Publish(IMessage msg);

The specific messages that are sent are defined by what the service provided to its consumers. If a service wishes to expose only certain messages to a set of consumers, all it has to do is give them the appropriate subset of messages. The consumers won't know what other messages they can send, so chances are they won't send anything else. If security is a requirement, then you'd have to check, in your service, where the message came from before handling it, so that if someone shouldn't have known they can send you message X, but they still did, your service will reject the message.

Posted by Udi Dahan at 11:28 AM  Permalink |

I received the following question from Houman and I thought the both the question and answer deserved some more public discussion:

"Hello Udi,

I really enjoyed reading your article on MSJ, thanks.

I have been trying to get a handle on the trade off between consistency and availability in distributed environments. I feel that it is closely related to autonomousity. I was wondering if you could shed some light on it and how one can take advantage of WCF, WWF to address the problem.

Regards,
Houman"

And my response, originally posted here:

--
Consistency is a domain level concept – for instance, when you order a vacation including hotel, flight, and car rental, if one of those orders falls through, you don’t want to “rollback” on the others. From these domain level definitions, you can often run a sanity check on your service boundaries. Areas of consistency should fall within the boundaries of a single service. I don’t see myself, or anyone else for that matter, trading off on this – it is fundamental to the correctness of your system.

Availability is, or at least should be, orthogonal. You should be able to be consistent AND available at the same time. Service availability is not much different from "regular" availability that we’ve been doing all along. Of course, if we’ve divided up our services like I said before, than when one service becomes unavailable for some reason, the availability of other services should not be affected. This is definitely a part of service autonomy. If you have a poor division of responsibility between services, you can see it manifesting itself when the unavailability of one service causes other services, even though they may still run, to be incapable of providing service.

WCF is simply a communication mechanism. By understanding the above concepts, you will know which part of WCF to employ and when in your specific application. Workflow Foundation (WF) doesn’t really relate to any of the above.
--

Posted by Udi Dahan at 08:39 AM  Permalink |