Queues Are Databases?

On a meeting I had with a few fellow architects a couple of days ago, (courtesy of Eliaz Tobias from Microsoft Israel), someone brought up an old (1995) thesis by Jim Gray called Queues Are Databases. In this paper, Jim argues that queue systems are just a private case of a database, so they might as well be implemented on top of a database.

In retrospect, this paper was probably one of the initiators to what we know today as "SQL Server 2005 Service Broker". The fact that Service Broker is built on a database gives it a few niceties which are harder to implement on other messaging platforms. For example, in Service Broker messages are delivered only once and in order. You can add readers to empty queues faster, and lock all related messages that are read by one of the readers (so you won't have a situation where one reader reads the order and another reads the same order's order-line before the first even updated the order into the database) etc.

One (minor) issue is that queues need a lot of insert/delete operations, where usually tables show fewer deletes and many updates.

Another issue is that databases have relatively heavy installation, while messaging middleware has a much lighter footprint.

The main and real problem is that Queues Are Databases only when you need reliable messages. Messaging middlewares support few other types of messages express or uncertified message, like multi-cast messages (using protocols like Pragmatic General Multi-cast (PGM)) and publish/subscribe semantics all of which are not very database-like. (Imagine you create an application that sends out stock tickers to hundreds on users on your LAN you don't want to do a transactional uni-cast to each and everyone of them, now do you?)

To sum up: Queues are not databases. They can be implemented on top of a database for scenarios where
transactional messages are important. I would still stick to non-databased queues for most scenarios.

By the way, as a side-effect of having Service Broker in the DB, you can now perform asynchronous invocations (have a reader stored proc listen on a queue) and other nifty tricks like a derived transactions (having one transaction trigger another transaction which is not in the scope of the first one -- i.e if the second fails it will not the fail the first one)

Posted by Arnon Rotem-Gal-Oz at 09:14 PM  Permalink