Our long-time listener Bill asks the following:
]]> Hi Udi,Thank you for your podcast clarifying the concept of autonomous components. It certainly cleared up a lot in terms of how you define an autonomous component and helped position the concept better in my mind.
I am considering building our Policy Administration service with 3 autonomous components as you previously suggested - one for each insurance product family. As you previously mentioned, these autonomous components could either share a database instance or have separate database instances, even though they may share schema (being a part of the same service).
What concerns me however is that there is some data that must be shared between these autonomous components. A perfect example is Region. If a new Region is defined somewhere in the enterprise, a RegionCreatedNotification is published onto the bus. If I have 3 autonomous components sharing the same database, is it best I just choose one to be the subscriber for this event? Or should I create a new separate queue to receive these notifications and put a message handler for it in a separate MessageHandlers assembly? Or should I subscribe all 3 autonomous components and have them each check if the Region has already been created in the database before inserting a new one?
Option (1) seems presumptuous because it assumes all 3 autonomous components are sharing the same database. Option (2) seems a bit strange because it is like creating a 4th autonomous component only to receive the RegionCreatedNotification messages. And Option (3) seems wasteful since there is redundant code and the system is doing extra unnecessary lookups. At the moment it looks like Option (2) is my best bet, but I'm very interested to get your take on it.
Another question is whether you would say that autonomous components should be hosted in different processes. Since I don't need to run these autonomous components on separate servers at this stage, it would seem wasteful to create a service host for each component. I could always create separate service hosts at a later time if I needed to host one or more autonomous components on other servers.
And the final question I have is in terms of solution structure. I was contemplating the following structure:
ProductFamily1
----------------------
Company.PolicyAdministration.ProductFamily1.Domain
Company.PolicyAdministration.ProductFamily1.MessageHandlers
Company.PolicyAdministration.ProductFamily1.Messages
Company.PolicyAdministration.ProductFamily1.Persistence
Company.PolicyAdministration.ProductFamily1.Persistence.Implementation
Company.PolicyAdministration.ProductFamily1.Etc...
ProductFamily2
----------------------
Company.PolicyAdministration.ProductFamily2.Domain
Company.PolicyAdministration.ProductFamily2.MessageHandlers
Company.PolicyAdministration.ProductFamily2.Messages
Company.PolicyAdministration.ProductFamily2.Persistence
Company.PolicyAdministration.ProductFamily2.Persistence.Implementation
Company.PolicyAdministration.ProductFamily2.Etc...
ProductFamily3
----------------------
Company.PolicyAdministration.ProductFamily3.Domain
Company.PolicyAdministration.ProductFamily3.MessageHandlers
Company.PolicyAdministration.ProductFamily3.Messages
Company.PolicyAdministration.ProductFamily3.Persistence
Company.PolicyAdministration.ProductFamily3.Persistence.Implementation
Company.PolicyAdministration.ProductFamily3.Etc...
Common
----------------------
Company.PolicyAdministration.Common.Domain
Company.PolicyAdministration.Common.Persistence
Company.PolicyAdministration.Common.Messages
Company.PolicyAdministration.Common.ServiceHost
Company.PolicyAdministration.Common.Etc...
Do you have any thoughts on the above structure?
Once again thank you very much for your extremely valuable advice!
Best Regards,
Bill
Or download directly here.
Additional References:
Want More?
Check out the “Ask Udi” archives.
Got a question?
Send Udi your question and have him answer it on the show: podcast@UdiDahan.com.
]]>...
Harry asks me to consider using a 'logical name' of the receiver. The sender contacts a logical end point, the addressing infrastructure turns that into a physical end point, and we still have decoupling.
Honestly, I like it but I think it is insufficient. What if we need to contact 20 downstream systems in a complex workflow, but I don't want a single "orchestration coordinator" to be a bottleneck (or single point of failure). I don't want to hand the orchestration off from my app to a central orchestration hub.
Let me propose a different approach.
When we use SOA/EDA (the same thing as far as I'm concerned), the top-level building block used is the Service. A service may make use of a number of existing systems to perform its work. The business-level events that we publish (and subscribe to) are done by the service, not the existing systems.
If there's any orchestration/workflow that needs to be done as a result of a service receiving an event, it is done entirely internal to that service. Inter-service orchestrations don't really exist, as in there is no orchestration coordinator that is not in a service. And the orchestration coordinators within a service don't touch other services' back-end systems - if anything, they publish other business level events.
Be aware: when just starting out on an SOA, you'll find that multiple Services make use of the same backend systems. This may be necessary, but not a desirable state to stay in for too long since it embodies the most insidious and invisible kind of inter-service coupling there is.
I want to go back to Nick's original question:
So what if no one picks the message up? Is that an error?
If a service publishes a business-event (message) and no other services currently care, that's fine. It's not an error. Actually, you'd probably have some kind of infrastructure "queue" where messages that haven't been received more than X time get sent to, so that the event isn't "lost". On the other hand, within a service - if an existing system sends out a message that needs to arrive at another system, and that message doesn't arrive or isn't picked up "in time", that is an error.
This is one of the advantages SOA brings to the table in terms of EDA (again, the same as far as I'm concerned). You get simple messaging semantics between services, while within the "sphere of control" of a service you need, and more importantly can do more complex messaging and orchestration.
Bottom line: you need higher abstractions than your existing systems to employ EDA effectively.
You might also want to check out my podcast on this topic: SOA, ESB, and Events.
]]>Here are some of the more popular ones:
If you are a new visitor and are looking for more information on NServiceBus, you can find it all here. You might also want to take a moment to subscribe so that you don't miss out on any of the updates. And, as always, please do feel free to ping me about these topics via email or in the comments below.
Thanks for dropping by.
]]>If you are a new visitor and are looking for more information on NServiceBus, you can find it all here. You might also want to take a moment to subscribe so that you don't miss out on any of the updates. And, as always, please do feel free to ping me about these topics via email or in the comments below.
Thanks for dropping by.
]]>I think WS-* will make it possible to have the ESB evolve from a vendor-product to a concept implemented in the operating systems an network devices that understand WS-*. Then you can leave the prefix "Enterprise" and we will be ready for an univeral asynchronous data bus over the Internet (or any other network you like). This will help breaking the current "services centric" idea of SOA into a "messages centric" perspective.
What are your thoughts?
Download via the Dr. Dobb's site
Or download directly here.
Additional References:
Got a question? Have Udi answer it on the podcast - podcast@UdiDahan.com
]]>Or download directly here.
Additional References:
Check out the “Ask Udi” archives
]]>Download via the Dr. Dobbs' site.
Or download directly here.
Additional References:
Want More?
Check out the "Ask Udi" archives.
Got a question?
Send Udi your question and have him answer it on the show: podcast@UdiDahan.com.
]]>Or download directly here.
And here's the question that gets us started:
Hello there,We are developing an application in .NET Smart Client application which connects to a WCF service and then to the data access layer. The issue we have here is on the type of data which can be passed between the client side and web service.
1. Can it be a dataset (No!), custom entity or XML or anything else ?
Which is the best one ?
2. Of course , a dataset can't be used coz it can't be used in non-.NET environment?
3. If we go for custom-entity or XML how do we ensure the changes happen to the data during client-side actions ? For example, binding the XML or custom entity to a data grid can be done. But if user makes any changes to the same, how do we identify ?
4. If we use custom entities / XML, the same can not be passed to the Data Access Application Block which invokes the data objects.
Note : All the above are based on assumption without using a translator in between any of the layers.
Please provide a better solution if you have come across any.
Additional References:
]]>E2E, a relatively unknown Swiss software company has created an ESB based on a virtual machine that runs UML. Not only that, but they're using their product for seriously large customers.
]]> I've spent over an hour on the phone with their executive team going into both the technical and business aspects, and they're the real deal. They will adapt their pricing model to whatever works for your business - per click, per CPU, per transaction, per hour, anything.I'm thinking that I'll be seeing these guys showing up on the short list of many of the projects I'm consulting on.
]]>Download via the Dr. Dobb's site
Or download directly here.
Additional References:
Q: Could you tell you your thoughts or preference for a distributed or centralized ESB?]]> This is the problem with a lot of the products that call themselves ESBs. They are centralized brokers which may be clustered for availability. But they are in no way an implementation of the Bus Architectural Pattern. Please check this before cutting a check to your vendor.DON: there is no such thing as a centralized ESB.
Also, understand that if you do security related things in your ESB, possibly as a part of your routing rules, that if the security infrastructure is centralized that means your ESB is too. Even if it really was distributed to begin with.
Buyer beware.
]]>What I find promising: an Acropolis part can essentially consume a SOA service, allowing the composition of process and activity services to be as simple as snapping parts onto a surface.
Since services which conform to SOA principles will expose duplex/asynchronous APIs, this will almost necessarily make the client-side interaction multi-threaded. I do, and have done quite a lot of work on multi-threaded, thread-safe, smart clients. They're hard. Here's a post describing how hard. There are other solutions as well, but very little of it has to do with technology.
Since both the CAB and Acropolis do very little in handling, or even describing in their docs these multi-threading issues, I don't think that they're going to make it simple or easy to work with SOA-style services. This is not to say they don't provide any value. On the contrary, I think that they bring the mainstream many steps forward. However, there is still a long way to go.
]]>string SomeFunction(char c);
But it didn't.
This was ridiculous. If Web Services couldn't give me interoperability at this most basic level, I swore that the whole thing must have been a conspiracy. After some ill-mannered googling, I found out that:
The Java char datatype is not supported because of an omission in XML Schema.
Bugger that.
They're all out working on WS-Make the coffee machine interoperate with Columbian coffe bean growers, and they couldn't be thorough enough to handle that?
The thing is, for this project nobody budgeted or scheduled time to go over all those APIs to find out what wouldn't work, change the component's interface and implementation, re-test it, and on and on... We were strapped as it was.
The thing was, I hadn't given this interoperability thing a thought. "Just use Web Services", is what I, and everybody thought. With all the wizards and stuff built in to the tooling, you can get a component "Web Service-ized" in next to no time.
I'm beginning to wonder what else haven't they been telling us...
]]>Here's the original question:
Hi Udi,This is Kumar, i am new to .NET and related technologies. Actually i am designing an application with few a components like a Socket Server(a service) which received input from payment terminal for loyalty application, which should communicate with Web Services deployed in an IIS application server, then the web services should communicate with an Oracle Database.
What are the patterns and best practices to use for the communication between the socket server and application server, and how do I scale for multiple clients in socketserver, and what are the security things that i need to consider.
Should i need to use .NET Remoting instead Web Services for my application server layer, and which is best for my socket server?
Thank you for your support.
Kumar
Additional References
]]>Before going too far ahead, you might want to take a look at my previous post "Space-Based Architectural Thinking, or listen to my podcast Space-Based Architecture for the Web. There's also a 30 minute webcast online describing SBA more fully here. I'm also going to try to stay away from things concerning Jini this time after already discussing the connection between Jini and SOA, and the tradeoffs between two general approaches: Tasks and Spaces vs Message and Handlers.
OK, so the issue of state-management is a big one. Everybody wants to work stateless, because it scales. The only problem is that the business processes that we are automating are long running, meaning that there are external systems or people involved. This makes these processes inherently stateful. So, we need a way to scale statefully - SBA gives us that. For some background on the "Shared Nothing Architecture", I suggest reading this post on inter-process SOA and this one as well.
Availability also has to be handled, not only in terms of having enough servers online to handle the required load but in having all the data required to process each request be accessible. This has often been handled by the database using ACID transactions - durability being that which solved availability issues, but also hurting latency the most. The problem with saving the state of our long-running business processes/workflows in the database is the load and the responsiveness requirements. In many verticals - telcos, financial, and defense to name a few, we need millisecond level latency on each stage of the workflow. This is what leads SBA to the in-memory, replicated data grid.
Note that SBA only intends to take these workflows out of the database, and not anything else - especially not Master Data. The lifetime of these workflows is incredibly short compared to that of master data like customers and products. It will have much different backup strategies as well. In terms of load, these workflows will be heavy on reads and writes together in the same transactions, but quite low in terms of just reads. If we have workflows that perform work in parallel, we easily end up with concurrency requirements that make DBAs cringe under the barrage of short transactions.
If you're worried that Workflow Foundation (WF) won't scale because of the above, you needn't be. You can (more or less easily) replace the persistence mechanism of WF with your own, saving your workflow instances to an in-memory replicated data grid.
By enabling the objects in the grid to call back into logic on your servers, you have, in essence, done messaging and more. The added benefit that SBA receives from this is a unification of technology between caching and messaging. This translates directly to savings when it comes time to cluster each of those technology's environments.
Finally, if we can find an attribute in the incoming stream of messages that creates a nice even distribution, we can then partition our load between our servers by that key. This will work up to the point where the load per key increases beyond a single server's capacity, and then we have to look at re-partitioning, a non-trivial problem. However, if we put objects in our grid that represent the master data, and tie them to our workflow instances with both of those tied to the key of our load, a smart infrastructure can make sure all that data is already resident on the server that is handling that piece of the load. That decreases latency even more since we no longer have to pay network roundtrips to collect all the data needed before we can process it. That's a substantial advantage for the above verticals.
But all of this has nothing to do with SOA.
Sure, it'll change how we implement our Services internally, but it has no impact on their interfaces or the top-level service decomposition. In the Java community, the word "service" is often used to describe the logic of a system. Great significance is placed on keeping these "services" simple, as in Plain-Old Java Objects. The fact of the matter is that the logic of the system should be simple and independent of other concerns like data access and communcations (a la Web Services), but that does not make it a service, not in the SOA sense.
For more information on what Services in SOA are like, check out this podcast on Business and Autonomous Components in SOA. Actually, SBA will probably have the biggest impact on the way autonomous components will handle service-level agreements.
So, it appears that even with SOA, SBA has its place. The former dealing with business level agility, the latter dealing with all the technical aspects of supporting that agility. If you're tasked with the designing the architecture of a scalable, available, high-throughput, low-latency distributed system, I'd strongly advise you to look at SBA - the technical value is overwhelming. Even if you don't utilize all elements of SBA and choose the Master Worker Pattern instead of load partitioning, you'll find the technologies supporting SBA to be quite flexible in that respect.
Will Space-Based Architectures be a part of your future? I don't know for sure, but they're a most welcome part of my present.