
Uses For A Message Queue - paddyforan
http://blog.iron.io/2012/12/top-10-uses-for-message-queue.html
======
praptak
Not listed: ability to route through very restrictive network setups
("galvanically isolated" networks.) MQs, being latency insensitive, can even
go over protocols like e-mail.

~~~
carimura
Nice one. #11.

------
jjm
Side note, I think the picture are of White Geese and not ducks.

~~~
paddyforan
I had the same thought. However.

<http://www.flickr.com/photos/david-hilgart/4142396713/>

If the photographer calls them ducks, who am I to argue?

------
KeyBoardG
Great stuff. We used message queues when rewriting major components from an
ancient VFP Client heavy application to newer C# code running on an
application server. It enabled both asynchronous work as well as ordered
operations.

------
Spearchucker
In the freezing cold at Clapham waiting for a train, so not much time here to
dig deeper. However, the elasticity part is interesting, and I'd like very
much to know what that _really_ means in context.

Other than that I've worked with messages queues a lot over the years,
'specially in the late '90s/early 2000's. Cutting to the chase - I have a
problem with message queues, in that they tend to add a lot of bloat (in terms
of both deployment size and integration cost) and in return don't solve a very
difficult problem.

First and foremost, "message queue" is really just store and forward. This is
hugely relevant in today's mobile world. Being able to edit a row in a
database while on the London Underground, and knowing that it will just sync
up the next time I connect to a network is a big thing.

That said, other than elasticity I'd like very much to know how this is
different to MQ Series or MSMQ. Things like in-order delivery and guaranteed
delivery are not a big deal to achieve.

Guaranteed delivery is super easy. When uploading, you call and the server
with your payload, and inspect the server response. If the response is an
acknowledge/success you've guaranteed delivery. Downloading a message requires
two calls - the first to get the payload, the second to confirm back to the
server that you've received it. Again, guaranteed delivery, done! If either
the upload or download fails then just keep trying until it works. It doesn't
matter if the server gets the data 20 times, it only needs to record one
receipt. "Only once" delivery is the same - the server just discards
subsequent uploads from clients that didn't get the appropriate response, for
whatever reason.

In-order delivery is solved by adding a sequence number to the batch/payload
packets. You can actually send them in any order you like -- the server simply
re-assembles the messages in the required order when the whole batch has been
delivered. Where messages are spaced apart in time this is even less of a
requirement.

What's really interesting to me, and what isn't mentioned at all, is whether
they do push from the server. Everything there can be done using client pull.
MSMQ and MQ Series both fall down with push, because you can't see through a
NAT'd network.

Finally, store and forward, by definition, provides resilience and buffering.
The same concept of store before forward means that any half-decent
implementation will handle spikes (hence my question on "elasticity"). Async
comms are always a good idea, because they help you scale. Not sure how others
do comms, but I haven't used a synchronous call since .NET first came out in
2002.

I think real innovations around store and forward/message queues come from
performance (relevant especially on mobile networks, where I prefer to avoid
chatty formats like XML or JSoN), and security (encrypting a bitstream without
requiring SSL). Because of this I wrote a library that I simply drop into
every one of my mobile app projects. It took me a week to build and stabilise
(I add that to demonstrate that this is not a difficult engineering problem -
and I'm no rocket scientist by any stretch of the imagination).

The other innovation that's a fair bit harder to solve, is when a single item
is edited by two parties while one is offline, and then merging the updates
when both versions sync up.

[Edit] I've had a look now, and elasticity means little more here than the
caching nature of message queues (the store part of store and forward). As an
architect I see yet another middleware vendor, albeit with hipster terms like
spike and elasticity. As a programmer I imagine they solved this problem for
themselves the same way I did - and if they can make some money from their
efforts then they've done more with it than I have, and I think that's awesome
for them!

~~~
moe
_Guaranteed delivery is super easy._

Oh, then why do you propose a wrong solution right there in your next
paragraph?

 _you call and the server with your payload, and inspect the server response._

Wait, what if the response gets lost on the way? Are your messages idempotent?
Do they depend on state on the other end? How long are they valid? Is each
message atomic?

You may want to take a look at
<http://en.wikipedia.org/wiki/Two_Generals%27_Problem> and
<http://en.wikipedia.org/wiki/Atomic_commit>

Guaranteed delivery can become quite tricky depending on which constraints you
can and cannot enforce in your system.

~~~
ojilles
Completely agree. One small correction: "Guaranteed delivery can become quite
tricky" should roughly be: "Guaranteed delivery ARE quite tricky, if not, you
have not run into all contraints yet."

~~~
Spearchucker
Can you actually list these constraints? If not all, at least _some_ examples?
I've not encountered any I'm not handling [1]. My code has been running on
various device types and under all sorts of loads for years, and have yet to
encounter something not handled as described above.

[1] If you're referring to guaranteed delivery in the sense of guaranteeing
that your message is delivered to _a person_ that is guaranteed to action the
message, then this is not the same problem that message queues attempt to
solve. Such a scenario requires non-delivery timeouts and re-routing along a
routing table, culminating in what the military world calls a "guaranteed
action point", which is staffed by real people 24 hours a day.

I built a prototype based on Exchange and SharePoint once that tries to do
this. It's difficult, but not impossible. The real issue we faced was
cultural, in that a commander sending a message just "felt better" when
sending a courier on a motorcycle, as opposed to clicking send in Outlook.

------
bobinator30
old news

[http://blog.pasker.net/2008/06/16/you-might-need-
messaging-i...](http://blog.pasker.net/2008/06/16/you-might-need-messaging-
if/)

