

Ask YC: Why the excitement over XMPP on the backend? - gstar

I'm developing something now that implements a push client, and I keep seeing people getting excited about XMPP for publish-subscribe type things.<p>The slide deck that keeps getting press lately is here:
http://en.oreilly.com/oscon2008/public/schedule/detail/4359<p>It really seems to me that using XMPP (which is a stateful chat protocol) for RPC might be the wrong approach.  Have the XMPP advocates just re-invented message queues?<p>In the OSS world, ActiveMQ exists, so does AMQP.  MQ theory is very old, having been used for decades for things like financial systems.<p>Am I missing something?
======
briansmith
Message queuing is old, boring, and well understood. The same kinds of people
that are trying to convince us that REST is a huge improvement over WS-<star>
(despite the fact that once-RESTful protocols are becoming increasingly
similar to WS-<star>) are trying to convince us that XMPP is to message
queuing as REST is to WS-<star>.

More cynically, XMPP vendors generally give everything they have away in hopes
that somebody with money will buy support for it. Since customers of message
queuing products are people with money, it is only natural for people to hype
XMPP as the "worse is better", cheaper alternative to "enterprisy" MQ products
in hopes of having some money fall into their pockets.

Less cynically, the people hyping XMPP understand it very well from their
personal experiences with chat systems and "chatbot" like applications
including Twitter. It is easy to say "hey, if a computer can IM me, I bet a
computer could IM another computer." It is a little tricky to abstract that
into "hey, that is the same thing as that old, boring message queing stuff,"
and anybody that gets that far realizes it's not that exciting to blog about.

~~~
bct
> once-RESTful protocols are becoming increasingly similar to WS-*

How so?

~~~
briansmith
It is pretty common for a RESTful protocol to end up with batch-fetching and
batch-updating operations. Once you add those operations, you've already
violated the "uniform interface" constraint and removed all the benefits
(primarily caching, authentication, authorization, and parallelism) that
derive from the uniform interface constraint. Once you have batch-fetch and
batch-update, those two operations can generally do everything that you
usually did via individual PUT/POST/DELETE operations. So, why do you need the
separate PUT/POST/DELETE operations at all? It is just more surface area for
you to maintain.

Now, maybe you want to combine your batch-update and batch-fetch operations
into a single "batch-update-and-fetch" POST operation. Well, that is exactly
what WS-* does.

On the flip side, sometimes instead of batching you want to fetch or update
only parts of individual resources via some extension to partial GET requests
and or a PATCH method. There is currently no uniform interface for those so,
just like in the case of batching, you lose all the uniform-interface
benefits.

Now, let's say you want to combine your partial-resource operations with batch
operations over multiple resources to reduce the number of requests you
receive and/or the reduce the aggregate size of requests, so you add the
ability to do patching and partial-fetches to your batch-update-and-fetch
mechanism.

Now take a step back. Doesn't your batch-update-and-fetch-with-patch-and-
partial-fetch look a lot like SOAP with WS-ResourceTransfer? It will probably
look a lot like SOAP; not even the newish SOAP 1.2 that actually is somewhat
HTTP cache friendly, but old school RPC-only SOAP + WS-ResourceTransfer.

Look at Google's GData and OpenSocial's RESTful API for examples.

And, that is totally ignoring anything to do with transactions.

~~~
briansmith
OK, I want to mention one more thing.

Take any WS-<star> specification, and you can probably find one or more ad-hoc
proposals for adding similar features for RESTful protocols. For example,
AtomPub is adding a mechanism that is almost exactly like SOAP-with-
attachments, but more limited. The various HTTP message signing mechanisms
like those found in Amazon's web services or the one in OAuth (and OpenID? I
forget if there is a different one for that) are analogous to the various WS-
Security stuff. As another example, I've seen a couple RESTful protocols that
tried to define something like a simpler WS-Policy.

I don't think that the WS-<star> stuff is that great either; a lot of it is
really complex. But, WS-<star> stuff is all decomposed in ways that make it
relatively simple to combine just the parts you need together. SOAP 1.2 isn't
complicated at all. WS-Policy and WS-Security and other stuff is more
complicated than it needs to be, but it isn't so complicated that it needs to
be totally redone anew. And, a lot of it can be (and is) subsetted pretty
easily.

------
aconbere
I'm not sure this is an either or proposition. I haven't seen many message
queue protocols with tools for identity, user authentication, federation, etc.
I'm also pretty sure that xmpp doesn't even begin to replace message queuing
on the backend, but when you look at what folks like EngineYard are doing with
it (basically distributing messages), it seems like the value for them was
more along the lines of 1) easy ui integration, 2) well understood problems
and tools by developers, 3) well understood servers by admins.

More on the proposal of mixing the two, the rabbitMQ folks have done some
interesting work integrating with xmpp. (See their xmpp transport
([http://www.lshift.net/blog/2008/07/01/rabbitmq-xmpp-
gateway-...](http://www.lshift.net/blog/2008/07/01/rabbitmq-xmpp-gateway-
released)), and rabbiter (<http://github.com/tonyg/rabbiter/tree/master>).

So there we have an interesting mixture of the two technologies, you get the
insanely fast message routing and throughput of RabbitMQ while leveraging the
tools that xmpp brings with it (users, federation, etc.)

~~~
aaronblohowiak
The merging of xmpp and rabbit mq is fascinating -- i've bookmarked this and
will follow-up on it. I'm reading through the documentation now, but from what
I can see this is great tech. (mod parent up!)

------
pelle
Almost none of the current generation of large scale web applications
including twitter are doing anything remotely new.

The investment banking world has as you say had to deal with huge feeds as
well as mashups from multiple sources.

<http://www.zeromq.org/> for example is coming out of the financial world and
looks like it's going to be the nginx of amqp systems. It looks like it's way
faster and "scalable" than creating something over xmpp.

~~~
newton
Also check out this Erlang/OTP AMQP server: <http://www.rabbitmq.com>

AMQP is very interesting. Check it out if you are building a distributed
system or if you have a requirement for reliable asynchronous messaging. When
I last looked into it about a year ago, the implementations (all two of them
that I found - apache qpid and rabbitmq) were still a little rough but it
looks like a lot has happened.

It's wierd how few people in the web world are aware of asynchronous
messaging.

------
eugenejen
No. You didn't.

But the context is different. XMPP as MQ mechanism may work better in Internet
scale distributed system. Just like HTTP works better as a client server
mechanism on Internet scale distributed system than other protocols before it.

I worked in a startup called PubSub from 2003 to 2006. We in fact used XMPP in
2004 internally to relay our error message log of our cluster of machines in
real time to our office's archive server. It was an experiment but it seems
pretty useful. We called it as fuselet before in our company.

~~~
briansmith
"XMPP as MQ mechanism may work better in Internet scale distributed system."

Why do you think so? What advantages does XMPP have over other MQ technologies
at "internet scale"?

~~~
eugenejen
Is there any existing MQ system that is decentralized? I said "may works
better" because it is a conjecture limited by my ignorant mind. Internet is
essentially decentralized. So if any MQ system can be modified to worked in
decentralized environment. Then they may work better than XMPP. I believe the
performance of throughput of those system may be higher than XMPP.

On the other hand. If I only have an internal cluster, I will choose to use
existing MQ system instead of XMPP. I may want higher throughput, and I don't
care authentication of endpoints because they are all trusted. Why do I need
all the extra stuff that XMPP demands?

Regarding the case I mentioned in PubSub: It was just an idea. In fact, I was
the one who mentioned the idea then. And our CTO and VP of Eng thought it was
an interesting idea because in this case, we just wanted to see how does it
work out and did a test implementation. And it was pretty flexible for us to
use it.

Of course XMPP is not the first MQ system on Internet. SMTP is the first one,
right? In SMTP, we don't know anything of endpoints. We assume senders and
recipients exist and are taken care of by SMTP servers. I think in XMPP,
because the original application is chatting, then it adds an new protocol to
notify "presence" of endpoints.

Edit: I think when I said "internet scale", I actually meant "a system of
billion nodes that are loosely coupled together"

~~~
melo
"Is there any existing MQ system that is decentralized?"

Yes. <http://softwarelivre.sapo.pt/broker/> for example.

But "decentralized" does not mean "federated" which is I think your point.

Best regards,

------
neilk
You're missing that they don't intend it as an RPC or REST interaction. It's
to "stream" updates out of services, that can have multiple listeners.

This is already happening, but in isolated cases. Behind the scenes, Six Apart
streams all the blog posts on its services directly to Google Blog search.

There are quite a few well-known services that get more API traffic (search
engines, etc, pinging for updates) than actual human traffic. So something
like this is ultimately necessary, whether it's XMPP or something similar.

I think they are aware of message queues, didn't they allude to them in the
talk? Anyway they billed their talk as two guys who build web services making
a tentative foray into the message-queue world, so if you think you can help,
send feedback.

~~~
puppetsock
The OP understands that XMPP provides pub/sub message queues. He's asking what
XMPP provides over the other MQs.

~~~
melo
Federation.

I read a bit over ActiveMQ, RabbitMQ, and the recent and lovely ZeroMQ. As far
as I could find, none of them provide federation on a Internet scale.
Specially Open Federation using TLS.

I'm a XMPP developer for 6 or 7 years now, and if someone thinks that XMPP is
going to replace MQ systems, they are delusional. But for Web integration,
XMPP beats the crap out of current MQ systems because of Federation.

Sure, you can get 2.5M messages/sec on ZeroMQ. So what? I've seen personally a
XCP XMPP server doing 1M messages per second also. But for internet scale
stuff, even slower servers like OpenFire would work just fine _for the current
loads_ they are seeing.

It also helps that for scripting languages commonly used (like Perl, Ruby,
Python), XMPP bindings are commonly available, while AMQP are still rare.
Please please prove me wrong :).

My recommendation for startups? Use a MQ system internally from day one to
develop your solution, but use XMPP to talk to the outside world.

RabbitMQ (and I'm suggesting it based only on what I read, I don't use it
myself, I use another one) is the best placed MQ system right now, because it
offers the bridge between local MQ system and XMPP world.

Best regards,

------
bjclark
"which is a stateful chat protocol"

You're missing that it's not stateful chat protocol, it's stateful xml
protocol (that has, so far, been use mostly for chat).

People are excited that it's fast, stable (some of the implementations
anyway), and they don't have to reinvent the wheel.

~~~
puppetsock
"stateful xml protocol" makes as much sense as "stateful ASCII protocol" or
"stateful bits protocol".

~~~
neilc
"Stateful XML protocol" makes complete sense, it just isn't very informative.
It describes a protocol that is both stateful and XML-based. Those are
independent attributes of a protocol; obviously, you can have both stateful
non-XML-based protocols and XML-based stateless protocols, but I don't see why
the OP's statement is nonsensical.

------
wmf
The MQ people don't seem to be explaining their value to the Web 2.0 kids.
When I read the ZeroMQ site the other day I kept thinking "but what is it used
for?"

~~~
newton
It probably doesn't help that books on the subject have desperately unhip
titles like "Enterprise Integration Patterns". The introduction to that book
is actually a great essay on when and how to use asynchronous messaging, and
it is online here:
[http://www.enterpriseintegrationpatterns.com/Introduction.ht...](http://www.enterpriseintegrationpatterns.com/Introduction.html)

------
aaronblohowiak
you have a categorical error when you compare ActiveMQ to AMQP; they exist at
different layers of the MQ stack (AMQP is a transport protocol.)

[http://activemq.apache.org/how-does-activemq-compare-to-
amqp...](http://activemq.apache.org/how-does-activemq-compare-to-amqp.html)

~~~
gstar
Noted, but my intention wasn't to list products rather to name some
technologies that are available and open for MQ.

------
bct
Why not? It's extensible, it builds on tech people are already familiar with,
and it's fast enough for most purposes.

It's also got lots of good libraries for many languages, and neat tricks like
bridges between it and the web browser.

------
joshu
Nobody's proposing using it for RPC. They're suggesting it as a replacement
for polling.

------
jfarmer
Nope.

