Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: What should replace XMPP?
75 points by tylorr on Jan 28, 2016 | hide | past | favorite | 55 comments
Let's assume for the sake of discussion that XMPP has died or is at least on its last legs. What could we learn from XMPP that would help us find its replacement?

There are a few key questions that need answering. What did XMPP do right and could be copied? Why did so many big players ditch XMPP? What would it take to get big players to adopt it? What are the existing resources for creating protocols similar to XMPP? Is it possible to simplify a chat protocol or is it complex in nature?

I briefly tried working on an existing XMPP server implementation but found the protocol and all the extension overbearing and very hard to grok.




What XMPP did right:

  - Any chat protocol must be extendable
  - Federation is good
What could have been better

  - Regular users don't want to run their own server
  - Overcomplicated specs with optional bits (MUC, Pubsub)
  - Encryption should be mandatory from the start
  - Logging/history is important
Other lessons

  - Device sync is important
  - If you have multiple devices or clients; commit log style is best (see new MIX spec vs MUC spec).
  - In the modern world, people are afraid of XML
      - 95% of people still don't understand XML namespaces
-------------------

But XMPP isn't dead (and doesn't have to be). New interesting XEPs are being working on:

  - MAM
  - MIX aka MUC2
  - OMEMO


  - In the modern world, people are afraid of XML
      - 95% of people still don't understand XML namespaces
I guess that's one way of saying that something is too complicated :)


It isn't much different from namespaces in programming languages, which most developers do not have a problem understanding.

They are in XML to avoid naming clash, and they do that well. I haven't really seen a different mechanic to solve that problem.


I think the problem with xml namespaces enters in a few ways:

  - Non XML namespace supporting parsers: these are xml parsers people throw together without paying attention to the fact that attributes starting with "xml" are special and *must* be rejected.
      - This results in people trying to shim namespaces in after the fact; which rarely goes well.
  - There are two syntaxes: `<foo xmlns="urn:example"/>` vs `<myns:foo xmlns:myns="urn:example"/>`. Most people get caught up in the latter without understanding it, and you end up with *lots* of implementations that depend on `myns`, which is meant to be arbitary and can change between serialisation/deserialisation
  - The W3C style for namespace inheritence forces each element to keep a reference to it's parent: a nightmare for simple data structures.


The concept of namespaces in XML is fine, but the implementation... there's an awful lot of twiddly options and details and implementation/API quirks that come up. Throw stuff like XPath and XSD schemas into the mix, and the pitfalls multiply.


Nice list.

> What XMPP did right:

> - Any chat protocol must be extendable

That is a theoretically desirable technical property, but there is a strong argument to be made that extensibility, and the interoperability issues that result, is the Achilles heel of XMPP.

Of course, there is risk the other way around too: by over-specifying, you constrain your use cases, which may hurt adoption.

I'm skeptical that there will (or should) ever be "one protocol to rule them all." There is such a diversity of use-cases, anything that delegates its raison d'etre across multiple specs is going to encounter the same flavor of problems that XMPP has.

> - Federation is good

Definitely another desirable property. Unfortunately, the choice whether or not to federate is often a policy decision, not a technical one. And those policy decisions are not always made simply to keep users siloed: there can be a sound technical basis for them. For example, to mitigate abuse, to immediately extend the service with new features, and to have end-to-end control of user experience without concern for interoperability. Interoperability with arbitrary network elements is often the most frustrating and most expensive part of standards implementation.

For federation to be taken as the default, you need critical mass, and no single IM service has ever achieved critical mass. While it would certainly be nice to have IM standardized like E-mail, telephone calls or IPv4, that comes at the cost of glacial progress forward, and you will be stuck supporting the lowest common denominator for a long time. So there will always be operators trying to short-circuit that baggage by introducing new (and often proprietary) services. The problems of standardization (or lack thereof), in their broadest strokes, are never going away.

> What could have been better

> - Regular users don't want to run their own server

> - Logging/history is important

Those seem like implementation details and not anything intrinsic to the protocol.


> That is a theoretically desirable technical property, but there is a strong argument to be made that extensibility, and the interoperability issues that result, is the Achilles heel of XMPP.

> Of course, there is risk the other way around too: by over-specifying, you constrain your use cases, which may hurt adoption.

IMO anything that doesn't have extensibility built in, will gain it after the fact whether you want it or not.

Better to have clear deliniation of duties and a standards body than a hodge-podge of uninteroperable things.


> Better to have a standards body than a hodge-podge of uninteroperable things.

XMPP has both in spades -- that's where the theory doesn't meet the reality. And XMPP is hardly alone; this is a chronic issue with messaging, because the problem space is so ill-defined.

The most successful protocols have been extremely focused in scope. IPv4 is essentially unchanged from RFC 791. TCP has a very carefully curated set of options that has stayed small. For ~15 years, RFC 2616 was the only spec you needed for HTTP. Notice the layering here -- those protocols don't try to be everything for everyone, unlike XMPP's catch-all design.


What you clearly forgot were features basically everyone is using today that don't work with XMPP: sending pictures, sending voice messages

> - Federation is good

I think it actually isn't. You're locked in to one server with your account. The server might shut down totally. The server might shut down federation (just like google did). Registering is another step hindering usage, Peer to Peer should be the way to go.

There are two things you need servers for: collecting messages while you're offline and later pushing them to you, having a registration service for a nickname.

At least the first one doesn't require accounts to be bound to servers at all. A model could be that message bouncing servers are run by the app developers. Friends computers that are running could act as bouncers.

Which brings me to xmpp addresses. They're there to be able to initiate contact with someone you haven't contacted before. Nobody is using the username scheme anymore, because it really sucks. Facebook uses a) real names, where you can search somebody with the real name and more importantly uses b) the friends of your friends in suggesting who you could add. You can easily implement the latter feature also for p2p messenger protocols. Other messengers often use other verified usernames like phone numbers, that users already have in their contact book. I don't see how this is possible for a p2p messenger protocol.

> In the modern world, people are afraid of XML

It's just mapping very badly to native data structures in any programming language except xslt. This makes working with XML lots of work. Sometimes this is justified, as XML is a really good fit for the data structure of the document. That's generally the case when you have some text with markup, like HTML or for word processor files. Otherwise it's just an unneeded strain on developers.


> that don't work with XMPP: sending pictures, sending voice messages

Any files, pictures, voice, real-time streams (jingle) are all known extensions of xmpp/jabber. I'm not sure what you mean by "don't work".


What do you mean by "commit log style"? I couldn't really figure it out when looking at the MIX spec.


I really don't want XMPP to die, but I'm worried that it can only go down hill after the big players ditched it.

Is it possible for XMPP to make a major come back without their support? Would big players ever come back or are they too concerned with walled gardens?


I don't think even more XEPs will solve the problem that there are already too many incomplete / incompatible clients (and servers).


While I agree with you I think there is some good that can come of more XEPs. Eventually the good XEP concepts can be taken to create the baseline for a new protocol such as multi-user, multi-device encryption, push notifications, sync, etc.

XMPP as it stands provides a good test bed to find out what the minimum requirements are.


I think it's good: it allows XMPP to evolve as people learn what is good vs bad.

Even if you don't have full compatibility between 2 given clients; it's a hell of a lot more than two completely different protocols.


I think the issue of running our own servers would be non-existent if we could run XMPP in inexpensive $5 a month digitalocean servers. It is my biggest drawback for XMPP.


It's really easy to. Just install e.g. prosody

I run my XMPP server, and IRC bouncer (weechat in relay mode) + a few other misc things all off a $5 VPS.


Where do you host at? Curious what the spec differences are between your VPS and mine, if you don't mind answering


I use vultr, here have an affiliate link: http://www.vultr.com/?ref=6824304 I feel like vultr's $5 VPS is what DigitalOcean should have been...

But you don't need good specs. I used to run prosody off a microcontroller at home; I only moved to a VPS cause my net connection wasn't stable.


Digital ocean!


There are also plenty of free servers - you don't have to run your own. DuckDuckGo, for example, offers free @dukgo.com accounts.


Prosody runs on the original RaspberryPi, so it should run on even the crappiest VPS. It probably doesn't scale to thousands of users and is missing a fancy web GUI, but that doesn't really matter for a private server.


https://matrix.org/ — Matrix is increasingly looking like the only sensible player; they get so many things right:

* Federated

* Fault-tolerant

* Cryptographic attestations of messages


I'm a fan of Matrix.org and very slowly working on an implementation of the client-server API in Rust[1]. In the process of doing so I'm also doing my best to help them improve and clarify the spec. I'd love to see nice native clients for Matrix implemented for the various major platforms. Right now there is vector.im, but I'm personally not a fan of using a web browser for applications I leave running all the time, like chat.

[1] https://github.com/ruma


This is the only other alternative to date I've found that gets those things which I want. It's quite amazing all they're trying to accomplish as it is.


As you know, there's a lot of good in XMPP. The part that I think causes the most difficulty is that it's based on XML. I don't mean this in the typical knee-jerk XML bashing sense. It's just that XML is ill suited for a stream-based protocol.

XML is amenable to extensibility, with its innate namespacing features. But this also made it difficult for beginners to use well and was easy to get wrong. The XML decision was really just a consequence of the times in which it was developed. It was a natural choice. If it were done again, it would be REST-based and use JSON as the data format.


Would it be worth it to try to replace XMPP with something very similar but using a JSON based protocol and clean up its worts in the process?


I absolutely think it would be worth it. The data format change would be one thing. There are many other factors which are worth considering. For example, the long lived connections of XMPP don't work well in the mobile environment. There are XEPs which address things like this.

I think to do it right, you have to build a company around it, and build it in the open. Also, no one is going to jump on a protocol just because it's open. It has to be built around a service which is compelling in its own right, and gains user traction regardless of the protocol. Geeks can get it going, but it has to be self sustaining.


No. One of the problems of XML is that it's binary-intolerant, which means everything that isn't pure text is a second-class citizen. That means pictures, sounds, documents, compressed stuff and most importantly, any encryption. We've all agreed that encryption is now considered to be mandatory and must be included in a protocol right from the start, so any serialization that doesn't allow easy binary usage can't be used.


I'm using JSON for my new socketIO server. You can put anything on the wire with socketIO, so messages are small and simple. Plus the clients seem really good too. They'll auto reconnect, etc. if the server goes away. I haven't tested whether they'll queue messages while offline yet... guess I should.

I'd definitely recommend it. This is a problem I've had multiple stabs at, but I finally feel like I've got a long-term solution now. But yeah, no federation, etc. My need is single-site realtime chat.


I was under the impression that XMPP was alive and well, it's federation that's dead. Because as it turns out, the big players aren't really interested in interoperability, they want their own walled gardens because that's where you can actually make money.


XML was probably a mistake. Strong, deniable, end-to-end encryption should be mandatory. The Axolotl ratchet is the current state-of-the-art: maybe it does asymmetric things we don't need, or maybe that's helpful.

Looking forward: metadata protection. This is a much more difficult-to-solve problem, but existing tools such as Tor are partially successful.


> Strong, deniable, end-to-end encryption should be mandatory.

Strong end-to-end encryption with perfect forward secrecy should be mandatory. Deniable authentication (https://en.wikipedia.org/wiki/Deniable_authentication), however, seems like a potentially interesting option but not one that the protocol should mandate. Sometimes you do want authentication that remains valid after the conversation ends, so you can subsequently authenticate the messages in it.


Pond uses Tor for metadata protection. Vuvuzela is another recent messenger that protects metadata.


It you need that, why wouldn't you just tunnel your new protocol over Tor?

Edit: metadata protection, I mean. I'm not disputing the need for encryption.


Encryption would be a massive thing to figure out. More precisely, end-to-end encryption with proper multi-device support.

That's also something all the other established players haven't solved, so it would be a great advantage to get users.


Putting the format and security aside, isn't the main issue that XMPP was so good that we tried to bite more than it could chew?

- Local Federation is great, but Global Federation doesn't benefit the big players

- Extensibility is great for domain-specific problems, but it leads to more fragmentation

- It tried to handle data streaming, but it's inherently suboptimal at it

I mean, it was a great protocol to explore the IM design space, but it was bound to be replaced once we clarified our needs (for the most part, multiple devices and media streaming).

As a side note, I often get showerthoughts about mixing IM and blockchain techs. I'm not sure where that would lead us.


We are developing an open protocol, which take the best of XMPP and improve it :)

We are for decentralization, the extension of the protocol, but without harm for the users.

For web and native clients use thin client connecting to the server via websocket and CBOR. Why not JSON? This allows you not to have problems with sending binary files unlike XMPP.

Also thinking about using WebRTC for the transmission of all messages, files and audio/video, but today WebRTC is not supported everywhere.

The server has the ability to transport other protocols.


Can any of the new secure-text-messaging protocols, such as the one used by Signal, work in a federated manner? Many of those protocols currently use a central server; which ones actually require that, and which ones don't?


A few weeks ago I replaced a XMPP chat architecture with socketio. It's so much better for my needs (website with realtime chat).

There were 2 main reasons in the end:

- No way to log all messages sent to a database so that users could retrieve them on subsequent logins - No way to edit/delete messages, especially in multi-user chats.

Maybe these things are possible, but I couldn't find how to do it. Now, with socket IO on nodejs, I have a < 400 line chat server with DB integration that is much more suitable for my needs.

I guess XMPP's focus was too narrow.


> - No way to log all messages sent to a database so that users could retrieve them on subsequent logins

This is what MAM does.


Perhaps the ejabberd support is bad or poorly documented? Either way I just couldn't find how to do it. All I could find for MAM was logging to flat files.


This post [1] seems to indicate that ejabberd's support of MAM is complete. Note that you also (obviously) need support from the client, which most but not all clients do. Conversations [2], on mobile, does.

[1] https://blog.process-one.net/ejabberd-15-06/ [2] https://play.google.com/store/apps/details?id=eu.siacs.conve...


That sucks. I searched for ages but found nothing. Oh well, it only took me a couple of days to replace it, so it's not the end of the world.


Would something like ProtocolBuffers[0] or FlatBuffers[1] be a good replacement for XML? The data could then be transmitted in either binary or JSON formats.

[0] https://developers.google.com/protocol-buffers/

[1] https://google.github.io/flatbuffers/


BEEP on TCP - RFC 3081

Any day now...

https://en.wikipedia.org/wiki/BEEP


IRC - because dead simple & extendable

Tox - because fully decentralized & encrypted

SIP - because rock solid for voice & video

Chat & IM is not the web. It shouldn't be running on HTTP with HTTP APIs.


SIP is a signaling protocol (and XMPP can also be used that way). It doesn't really have much to do with voice & video other than it gets used to initialize a connection.

Also SIP is horrible.


May not be 100% replacement, but JMAP ?


SIP?


The IM protocol that runs over SIP is called SIMPLE and it's anything but.


It's of course possible to simplify a chat protocol. For instance, I'm not sure why the XMPP designers decided to go with XML; it's much more verbose than most formats, and typically slower and more complicated to parse in my experience say versus JSON.

What advantage did it have? Unless someone is sniffing unencrypted XMPP packets and finds that XML readability handy, I don't really see the point. And that's an edge case. Not to mention XML's main advantage over many other formats is having hierarchical relationships--but how does that help in a chat protocol?

I'm sure someone could very well have counter points to the above, but those are the things off the top of my head that made me question why XML was the top choice here for an IM protocol.


XMPP predates the first general articulation of JSON by at least 3 years.


XMPP development started in 1998 and so predates JSON. And while it's true that XML is more verbose than JSON, I don't think it's worth redesigning the protocol over.


> Not to mention XML's main advantage over many other formats is having hierarchical relationships--but how does that help in a chat protocol?

Extendability. Transporting Messages in other messages. Transporting hierarchical data instead of only text.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: