Hacker News new | past | comments | ask | show | jobs | submit login
Ask HN: Why XMPP failed and SMTP didn't?
104 points by kertoip_1 on May 26, 2022 | hide | past | favorite | 170 comments
I was born too late to witness popularity of XMPP as a decentralized instant messaging protocol. When I started to be a "conscious" user of the Internet, such services already became centralized, so I don't really remember those times. As I follow the development of projects like Matrix, I wonder what has made SMTP such a stable protocol that has stood the test of time and what is the real reason we don't use XMPP (I mean, we do using Whatsapp, but as a part of closed ecosystem).

Is it only that XMPP was overcomplicated and if managed better it would survive? Or is it something else?




XMPP was actually fairly successful. It was just embraced, extended, and extinguished by the major companies that provided services using it. Google, Facebook, and Atlassian all used XMPP for their chat clients. You could freely interact with all of their chat solutions using XMPP clients. XMPP support for each one of them went away one by one.

The environment outside of major providers was (it definitely is) kind of janky. If you can set up a server application it's not hard but it's not exactly easy to get started, or get people on it without handholding (from my own experience).


We used XMPP as our work chat back in 2010's. We run our own server and everything. I am not sure why -- was it a bad setup or an bad protocol? -- but experience was horrible compared to other chats.

The biggest problem I remember was that XMPP was (is?) about message delivery, while what we really wanted was "synchronized message log".

So multi device did not work: One device was designated "active" and would receive the messages, others would not.

There was no history sync of any sort: If you had checked messages from home, they would not appear in your work computer's history next morning. If you replied from home, you won't be able to see your own messages at work PC.

Anything mobile (mobile phone, laptop in coffee shops) was also unusable -- you cannot start app and catch up on all missing messages. You had to be online to receive them.

For mobile access, I ended up doing custom Pidgin plugin (in perl!) which would export messages to text documents served over webserver, and then I'd visit that page from my phone to read the messages.

---

I still don't know what was wrong with that system, was this our admin not spending enough time on the server, bad documentation, or bad clients -- but this did not really matter, this experience really destroyed XMPP brand. We used Google Talk and Slack later, and they were "just working", no batteries required, no messages lost.

The funny thing I believe that Google Talk might have used XMPP under a hood at some point -- but it worked really well. So it is definitely possible to build working, feature-full things on XMPP... you just had to extend it and use it as underlying layer. Kinda like TCP and IP in a way.


Google Talk was XMPP from start to finish. At its peak, Google folk participated in the standards process. They also contributed various protocol extensions, including the one for audio/video calls. Their open-source implementation of this (libjingle) still lives on as a part of their WebRTC stack in Chrome.

You're spot on with your memory of early XMPP's behaviour. While it was one of the first messaging protocols to support simultaneous login, the focus was originally about routing messages to the right device. In the context of that period in tech, that was excusable and even desirable. As people got more devices connected, more permanently yet intermittently (i.e. smartphones), it stopped making any sense. We switched to a broadcast-and-sync approach, but it took a long time (years) for all the long tail of smaller internal deployments to roll that out (if they ever did). Another big issue was that one of the most popular clients was Pidgin, which ran into developer resource issues around the same time and still hasn't caught up yet. To many people though, this was XMPP.

I'm hoping a new generation of XMPP implementations can solve these kinds of issues, which was what inspired my "Products vs Protocols" [1] talk/article a couple of years ago, and is why I work on Snikket (and we have many happy users who are both new and not-so-new to XMPP).

[1]: https://snikket.org/blog/products-vs-protocols/


I don't know about when the extensions to the protocol came out, but having synchronized message logs you can catch up on with multiple logged-in devices is now a thing that reasonably works.


And, if I understand correctly, the text chat in Zoom is XMPP.

All of these companies chose to kill federation for their own advantage, making the world poorer.


Not really. Google killed XMPP federation because virtually nothing used it except spammers, who were becoming a big problem.

Controlling spam inside a closed network is drastically easier than controlling spam in an open network, because in a closed network you have lots of ways to make bans stick, whereas in an open network you really can't. Faced with the choice of building a massive Gmail style heuristic spam filter in which basically all the messages it was processing would be spam, or, just turning off federation, they went for the latter. It was probably the right call. Very few people even noticed, let alone cared. The idea of an SMTP-style global network of federated chat servers was dead by that point.


That's manipulation of history. XMPP federation was actually still working on the Google Talk servers even as recently as a couple years ago. However, Google outright refused to update their XMPP server capabilities. I am not talking about refusing to implement the latest XEPs (which they don't), I am talking that they didn't even bother to implement SSL! https://xmpp.org/2015/03/no-its-not-the-end-of-xmpp-for-goog...

Google was kicked by the other servers out of federation due to their abandonment way before Google turned it off. And it was always flacky to begin with, specially since they also started dropping 3rd party clients (and for that you can't blame spam...).

And even if one argues that Google ignored federation because of spam and/or low usage, OP's claim that they did that "for their own advantage, making the world poorer" would still be valid.


They killed it mainly because there was a dictate to unify around the emerging Google+ (I'm pretty sure there is a public source for this, but I fail to find it right now, otherwise I would link it here).

I'm not saying they didn't have spam problems, but that's the kind of thing they certainly had resources to solve. They already had such systems due to Gmail, and even independent XMPP operators have a pretty good handle on XMPP spam these days without even a fraction of the resources available to Google. It was more a matter of priorities du jour for the company.


Well, I was there at the time and this is what I remember. The Gmail spam filter is very email specific, it's not like you could just point it at a stream of chat messages and get good results.

Independent XMPP operators don't have anywhere near the spam problems Google has/had. They're just far less appealing targets.


A company I used to work at had an in-app chat feature that ran entirely on XMPP, specifically using ejabberd which scaled really well. Can't say it was super intuitive to setup and run though, there was definitely a learning curve.


I suppose this includes irc in a way too - AFAIK slack used to be (mostly) just a propiatary IRC network and client?


They had IRC gateways that you could use to connect your IRC client into slack, but I don't think their architecture started like the IRC protocol. It appears to have at one time just been a webapp, with databases for storage for messages. Not anything like IRC's protocol/architecture.

https://www.slideshare.net/InfoQ/how-slack-works

It may have subsequently turned into something more like how IRC servers work, but early on it doesn't appear to have been.


Twitch still is


The short answer is network effects. The longer answer:

SMTP is an old protocol; it really predates much of what we'd think of as the Internet (e.g., it predates DNS or even IPv4). This means that during the big initial growth phase of the Internet, SMTP was already an established standard that could be used to transfer email. Indeed, for email routing on the internet, you kind of had to support SMTP anyways, which makes it difficult for any other protocol to gain traction.

That's not to say that there weren't alternative protocols. The biggest of these was X.400, which many in the 90s saw as the eventual replacement of SMTP. But this was hampered by the already existing install base of SMTP. Other failures leading to the failure of X.400 was its reliance on the OSI stack, which lagged behind in implementation compared to the TCP/IP stack, and the rather cumbersome addressing model of X.400 compared to SMTP. The value added in supporting X.400 in addition to SMTP wasn't worth the cost of implementing X.400, and it wasn't really feasible to support only X.400 given the already widespread use of SMTP.

In this model, XMPP actually works closer to X.400 than it does to SMTP. It came about after the protocols it aimed to replace were already in existence--and wide use. (See also IRC, which is still alive and active). So implementing XMPP means you need to justify the value-add of XMPP over existing legacy protocols. If you were implementing your own, new chat service, it might make sense to build it on top of XMPP instead of a custom protocol. But replacing existing chat protocols with XMPP was again a costly move with benefits rarely justifying the move. The further federation goal of XMPP would have been an anti-goal to many of the chat implementations.


XMPP shares another feature in common with X.400: implementation complexity. Where XMPP had numerous implementation options and XML, X.400 has ASN.1. It's probably straightforward to write another solid explanation that phrases SMTP's success entirely in terms of its simplicity


As an old sendmail hacker, it amuses me to see SMTP called simple. I suppose it is though, at it's most basic level. It's the extra layers added over the years (MIME, text encodings, security, ...) and sendmail itself that make it complicated.

Email address routing used to be ugly too, but the consolidation on the <user>@<host> standard really helped there.


SMTP was simple then. It grew organically via grafting of new things. It's not simple now, but nothing would be.


Email has grown without fragmentation, because interoperation is seen as a basic requirement by the users and operators.

Also strictly speakng MIME and other content layer things aren't part of, and didn't need changes to, SMTP. This is of practical importance because it means email server infrastructure doesn't need to know about email content format evolution, only end user email software does.


Yes, this.


I mean, it's in the name :P


I recall reading that XMPP also wasn't always fully implemented / federated between services.

E.G. wasn't advertising basic client online / busy / etc janky between different siloed implementations?


Online/busy/etc. are part of the core standards and very basic functionality. All software has occasional bugs, but I never encountered widespread issues with statuses such as you seem to be describing.

There was a period of time where if someone signed in using the Google+ chat client, they would appear online but wouldn't receive any messages you sent them from the XMPP side. That was entirely a Google implementation thing (they had stopped maintaining XMPP interoperability at that point).


The big growing pain for XMPP was behavior when you had multiple clients connected simultaneously. Extensions were added to the standard that unified behavior around which clients received messages (today we'd assume all of them but that was more up in the air at the time) and tracking of read status across clients in order to highlight new messages. But during perhaps the heyday of XMPP the extension was not universally supported by even some popular clients, so multi-client functionality could be confusing and inconsistent, e.g. messages delivered to only one client for no clear reason. This is a pretty well solved problem today but, well, no one is using XMPP.


Tooling is always an issue in the beginning, when you pick a new technology like XML or ASN.1. Tooling is not an issue now, but now is too late. Turns out that writing specs -and implementing them- around simple textual protocols is easier than building new binary and textual structured encodings.


> it predates DNS or even IPv4

I don't know about IPv4... IPv4 was RFC 760 (Jan 1980) while SMTP was RFC 821 (Aug 1982). DNS was certainly later, RFC 1034 (Nov 1987).


The first SMTP RFC is RFC 788, not RFC 821. RFC 780 and RFC 772 are RFCs for MTP.

IPv4 is defined by RFC 760 and RFC 791, although I don't know if RFC 760 actually implements something that looks like modern IPv4 (one of the challenges with old RFCs is that they were actually requests for comment, and I don't have a good sense of when this practice actually stopped).

I was going by Wikipedia's dates, which give the actual use of IPv4 on ARPANET as January 1983, which I believe postdates the actual use of SMTP on ARPANET.


> I was going by Wikipedia's dates, which give the actual use of IPv4 on ARPANET as January 1983, which I believe postdates the actual use of SMTP on ARPANET.

This is documented on RFC 801 (https://datatracker.ietf.org/doc/html/rfc801), which literally just mandated use of the previous NCP until 1982 and then switch to the incompatible TCP/IP protocol stack in 1983. Imagine doing that with IPv4!


I can't confirm nor deny any claims about protocol age, but still want to point out that the RFC numbering only indicates the order in which these protocols were standardized (or at least proposed for standardization), not when they were first used.


Sure, and they both had several predecessors. But everything sort of blends together if you don't pick a specific point to define when a protocol 'begins'. Standardization through RFC seems like the best choice for the core internet protocols.


Nope, SMTP is the older one. RFC 821 standardized SMTP, RFC 760 created IPv4, and in fact if you read RFC 801 it required a switchover from the previous Network Control Protocol (NCP) to then-new TCP/IP on the start of 1983. As the grandparent points out:

> RFC numbering only indicates the order in which these protocols were standardized (or at least proposed for standardization), not when they were first used.

Even today, many things are de-facto implemented before standardized so I personally don't think that it's correct to look at the RFC date as the date to consider.


> Other failures leading to the failure of X.400 was

naming. The biggest failure of X.400 was naming. X.400/X.500 style naming was and remains an unqualified disaster. Not that the pre-DNS alternative (UUCP) was great, but that name@domain turns out to be infinitely better than X.400.


If I recall, Exchange Server back in the day was built on an X.400 stack and SMTP was a extra connector yiu could buy. That all got turned around when Exchange 5.5 hit and it was clear that even the US Government, who were a massive motive force behind X400/500, were moving on.


That said, the initial protocols that were too challenging for XMPP to unseat (MSN, AIM, Skype...) have been unseated by newer tech anyway (Whatsapp, iMessage, Telegram), so it clearly wasn't a perpetually insurmountable challenge.


I was a weird XMPP nerd in high school and tried to switch friends from AIM. So here's my experience.

* Onboarding was difficult. There was no obvious choice of server or client to use.

* Adding friends was difficult. You needed to send a subscription request to a contact, and they needed to send one to you. If anything happened during this process, you couldn't chat.

* Popular XMPP clients, like Pidgin, also supported the other chat services (AIM, ICQ, MSN, Yahoo, etc) so people just continued using those.

* Network effect. You need to convince a mass of people its better, otherwise nobody's using it because no-one uses it.

* No obvious benefit to the user. It's decentralized sure, but there weren't many improvements over AIM that people actually used.

* A lack of good iPhone XMPP clients.

In 2005 Google added XMPP support to Google Talk/GMail Chat and they were federated, but nobody federated back and they closed off its successor (Hangouts).


> A lack of good iPhone XMPP clients.

As someone whose company used XMPP prior to the iPhone, this is what killed it for us, and I suspect it’s a major contributor to federated XMPP losing any public inertia it had.

iPhone push notifications could only be sent with a signing key tied to the same developer account as used to publish the client application.

That meant it was impossible to send push notifications from your own XMPP server to a generic XMPP client written by someone else.


FWIW, this eventually got solved in XEP-0357, which, while it says it's deferred, is actually implemented by common mobile clients and modules exist for both common server implementations. This is the same design that Matrix eventually borrowed to solve the same problem.


I dunno how well that solved things as a practical matter. I recently began running a Prosody server to host a group chat (MUC) among myself and some friends. Some of us use Conversations on Android, and one uses Monal on iPhone. I also use Monal's macOS client.

The original setup was confusing. I ended up hacking some modules to debug why notifications didn't seem to be working, but eventually I got everything working. (I also wrote a module to sandbox Prosody using pledge and unveil: https://github.com/wahern/prosody-openbsd)

Then OpenBSD 7.1 was released, which upgraded Prosody from 0.11 to 0.12. It's roughly around that time that the iPhone Monal user stopped seeing notifications for the MUC. But the Monal app also updated around that time, so it's difficult to place blame. As far as I can tell notifications seem to be going out to the push server, but it's still a bear to debug and I haven't had the time to properly analyze things, let alone fix them.

I also installed macOS Monal on my young son's macbook so he could IM me while at work, but for some reason desktop notifications don't work at all for him, not even 1:1 chats, and neither pre- or post-Prosody upgrade. (They work for me on my mac mini, though.) We can still chat, but he has no idea if I replied unless he's monitoring the Monal chat window. It's most likely a client-side Monal or macOS issue, though, because Monal doesn't even appear in the Systems Preferences -> Notifications & Focus application list, and side-channel push notifications shouldn't even be involved.

Years ago I used Adium on macOS and ejabberd server-side and things seemed to work more smoothly, even when chatting with Google Talk users. But now Adium is no longer maintained and has fallen behind in extension support. In some ways the XMPP ecosystem seems to have regressed even for the simple stuff, let alone the more complex mobile and MUC support.


MUCs are sadly the absolute worst part of XMPP we see on a day-to-day basis. I don't think mobile notifications in them are actually solveable - while you're not connected, the MUC has absolutely no idea you exist, so why would it send you a push notification.

I'm very hopeful that MIX eventually solves the problems we have with it.


Members-only MUCs know who to notify because of the affiliations list: https://modules.prosody.im/mod_muc_offline_delivery.html


Fascinating. Is this problem solved for Mastodon?


This restriction that push notifications must only come from the developer’s server is a restriction imposed by Apple and their App Store.


A solution would be E.G. subscribing to a community (an endpoint URL, or what Discord likes to brainwash consumers by calling a 'server') should trigger background actions that allow that community's chosen point of contact to have push notification capabilities, and to maintain that forward until your subscription is expired.


> should trigger background actions

Because of the way iOS works nowadays, you cannot reliably do this for notifications because you can't run apps in the background reliably for this purpose.

> allow that community's chosen point of contact to have push notification capabilities

This is what push notifications are?


Not device background. IE submit requests to the iOS mothership and to the endpoint URL to talk to each other and work things out.


I apologize I'm not sure what you mean by mothership, I'll assume you mean the core API's the OS presents. I mean that's how requests already work? You ask the system to handle the network for you and it passes the app data?

In the context of notifications this isn't an issue for when the app is in the foreground. But when it's in the background the app's event loop isn't ran consistently, so you might not get the notification for hours. To my knowledge the ONLY way to consistently push notifications for apps in the background is through apple's push gateways.


An __off device__ request that is processed server to server.

Mobiles aren't "real computers" in the sense of the Internet. They are not intended to be always on; nor always connected from a stable address. They are portable (very smart) terminals, often with poor text input (but usually with reasonable quality camera and microphones), and often a large local cache.


The original comment was

> iPhone push notifications could only be sent with a signing key tied to the same developer account as used to publish the client application.

> That meant it was impossible to send push notifications from your own XMPP server to a generic XMPP client written by someone else.

and you were replying to

> This restriction that push notifications must only come from the developer’s server is a restriction imposed by Apple and their App Store.

So I'm confused as to what you mean.

> An __off device__ request that is processed server to server.

> ... They are portable (very smart) terminals ...

How does this relate to methods of reliably pushing notifications to an iPhone?


By this definition, doesn't it describe any laptop just as well?


Depends on how it's used. Laptops are more like portable computers and often the end user expects them to take a moment to startup fully even after a resume.


Presumably you can do like Matrix, and have the developer of the app run a push gateway.


Exactly, that's how XMPP does it too. Since authentication to the OS vendor APIs are generally linked to the developer's account, that's basically the only practical way to do it.


It is solved for XMPP as well: https://xmpp.org/extensions/xep-0357.html


> In 2005 Google added XMPP support to Google Talk/GMail Chat and they were federated, but nobody federated back and they closed off its successor (Hangouts).

I don't think "lack of federation back" was a big driver in this decision. Everyone who used XMPP was able to chat with gchat folks and despite some weird changes Google made to their integration, it worked reasonably well. You didn't need to do anything special to federate with another XMPP server. It just worked like email does.

I expect either product complexity (like having to support non-gchat addressing which then requires full JIDs instead of short names) or de-prioritizing features that weren't directly driving their growth thesis were probably more relevant. Ie, why spend two engineers to fix integration concerns and deal with federation everywhere, when you can just retask those headcount onto some new feature that will drive growth.


I really mean that no other big chat service federated back. The global network of federated XMPP servers was dwarfed by Google's userbase. Maybe if AIM/MSN/Yahoo added an XMPP bridge they would have kept it.


I seem to recall there was a brief period of time when gchat federated with both XMPP and AIM, and I was using Gajim to chat with users from both of those services without a bridge (or at least, the bridge run by Google/AOL was invisible to me).


Wasn’t Facebook Chat a big one?



Was Facebook messenger ever _federated_ though, like GChat was? You could use an XMPP client with messenger, but you couldn't talk to people on other XMPP servers like you could with Google Talk, from my recollection.


> A lack of good iPhone XMPP clients

And no server based push, though I'm sure someone invented an xmpp extension to fix that eventually :)


This is a dumb questions, please excuse the ignorance: Why were/are there not (or are there?) adapters so that you can speak whatever protocol you prefer (smtp, xmpp x.400 etc) but it still gets through to the user on the other end?


XMPP has explicit support for "transports"/"gateways" for that purpose, so you could plug in AIM/ICQ/whatev support into your XMPP server. See https://spectrum.im/documentation/about.html for example.

IRC has bitlbee (and probably others) to fulfill that role https://www.bitlbee.org/main.php/news.r.html

Matrix has "bridges" to interop with other protocols https://matrix.org/bridges/


There is a long history of this including multi-protocol clients. You see this now for example as people want a single "chat" for the many places they publish live streams (twitch, youtube, discord, etc). Unfortunately, because you must ultimately support the lowest common denominator you end up with much of the nice features of particular platforms being unavailable or things like the youtube chat filled with comments like `@comment-copy-bot: James (Discord): That was a cool video!` and things like direct messages and emotes don't really work.


That's exactly what XMPP was originally created for - a single open protocol that would let you interface with all the others.

But it's very hard, technically, to maintain interfaces to a changing set of evolving third-party protocols, and provide a good user experience. Especially if those people don't want you bridging to them. Open networks tend to fare better in the long term, but generally the user experience of an app bespoke for the target protocol is nearly always better.


* A lack of good iPhone XMPP clients.

Not only that. I'm putting the downfall of XMPP right about where most people started having a smartphone (2009-12), so it certainly didn't help - also on Android there weren't any great ones, the ones we know today are all younger.


I thought it was pretty easy to use using Pidgin.


Go on...


I'm glad that you have learned of the brief golden age of online communication. I had friends on 6 different services and used Trillian, Pidgin, etc to talk to them all. Somehow we have to escape these walled gardens and get back to that borderless world.

At work (operations for a space mission), we use an XMPP-based chat system for tactical communications along with several other systems, but Slack was introduced and more and more communcations are being handled over Slack. I'm advocating for replacing several systems with Jitsi, which is XMPP-based, because it allows us better control and customizability. It's an uphill battle.


>, I wonder what has made SMTP such a stable protocol that has stood the test of time and what is the real reason we don't use XMPP

There are probably multiple reasons and we can't replay history to know which reason contributes the most but one key difference is that email accounts (SMTP) were pushed onto the regular non-techie consumers whereas XMPP was more of a geek tool that users had to pull and opt into.

E.g. New students at a university automatically got a ".edu" email address. Residents got a free email account (ISP) with their cable service. Employees got a corporate email address. In other words, millions effortlessly got the utility of email/SMTP even without installing AOL CDROMS.

To further reinforce email/STMP, if consumers want to order something from Amazon, they needed an email address to create an account instead of an XMPP address.

Other forms of communication like XMPP/IRC don't have that widely disseminated self-reinforcing utility cycle so they stay a niche tool. What critical service in normal life requires an XMPP address? I can't think of any.


SMTP achieved critical mass before the age of modern startups / companies.

All of these federated protocols: XMPP and even Mastodon to an extent, are smaller groups of humble nerds running instances. They aren't massively advertised, large-scale billion dollars of investment companies like Discord or Facebook, trying to build network effects and/or advertising revenue.

-----------

When competing against profitable companies, its not sufficient to merely exist. You're competing against advertising and eyeballs. Facebook is big because its big, because it advertises, because it is constantly pushing for more-and-more users. Same with Twitter, same with Discord.

This leads to little advantages: a $10-million+ UI overhaul every few years. Designers to simplify the interface and onboard faster. Paying Google/Apple absurd amounts of money to access the push-notification APIs on phones. Etc. etc.

Smaller XMPP instances already fail at the push-notification thing. Who will pay for that? And without push-notifications, do you really have a modern chat platform? Other companies can afford the costs.


For the record, access to push notification APIs only needs to be done by the developer of the app the users are using. Small servers never need to communicate with Apple/Google themselves - any necessary notifications are routed through a gateway operated by the app's developer (which is also necessary because only the app developer is given the API keys and permissions to send push notifications to their app).

At the time of writing this, 90% of 363 tested XMPP domains support push notifications.

I'm not disagreeing with your overall point, mind. There is lots to compete with, and due to the lack of a good business case for open networks, most XMPP projects are by open-source volunteers and the majority of public XMPP services are similarly operated by volunteers.

There do exist projects with funding though, and there are multiple companies actively engaged in paid XMPP projects and deployments. But these all tend to happen behind closed doors, they're typically not deployments for the general public, or the XMPP is quietly under the hood of a larger service/product (such as happened in Zoom, WhatsApp, and many online games such as EVE and Fortnite).


Is there a publicly available report of this test of 363 XMPP domains? Thank you.


Yes, you can view this and other results at https://compliance.conversations.im/tests/


There are two reasons that I see.

One is that XMPP is an overcomplicated nightmare. The protocol was verbose, flabby, and hard to implement, and the server software was very hard to set up and run especially for novices. A person could not just install a server and start chatting, even if the server was just stand-alone let alone linked to anyone else.

The second and IMHO more fundamental reason is that the Internet is a dark forest. Any open system that becomes sufficiently popular will be destroyed by abuse.

If you offer a chance to make any amount of money whatsoever online millions of hustlers will rush into the void like gas molecules invading a cracked vacuum bell and will scramble all over each other to suck every last fraction of a penny out until nothing of value remains. So far only centralized managed systems have been (somewhat) successful at keeping the barbarians at bay. I'm not saying a decentralized system could never succeed here, but I don't think it's been done yet. (Cryptocurrency isn't an example as it's already been throughly destroyed for its original vision and use case by scammers. It's a great case study in exactly what I mean.)

(Edit: the goal is not always money either. Read money as "value." Political propaganda, cult recruitment, weaponized disinformation, or just trolling for lulz all count as extracting value of some kind at the expense of the commons.)

SMTP along with Usenet was one of the first casualties of this phenomenon, and I would argue that it did fail as an open system. You can run your own SMTP server but it's not for novices or people without time on their hands. You'll have to fight constantly to keep your IP out of blacklists and to keep spam away from your users. Spammers are in an arms race against huge companies with massive training data sets and entire teams dedicated to spam filtering, so this only gets harder over time. An independent SMTP server is easy prey.

99% or more of users use one of several large mail hosts. These are mostly Google, Microsoft, and Apple in the USA. E-mail hosting is cheap to free and the vast majority of people would rather someone else deal with the pain of defending them from spam.


> If you offer a chance to make any amount of money whatsoever online millions of hustlers will rush into the void

It's even worse than that because just a very small group (dozens, or even a single individual) can really screw over an entire system because doing things digitally is so cheap and easy. For example the world has 8 billion people, and sending every single one of them an email message (assuming everyone has an email address, which they don't) is easy, cheap, and can be done relatively quickly. With just 100 spammers on the entire planet your email system would still be screwed.


> Any open system that becomes sufficiently popular will be destroyed by abuse.

This is a harsh truth that I'm always surprised that so many HNers haven't accepted yet.


We are about to see it happen with reasonably open software package repositories. Prepare yourself for a coming deluge of increasingly stealthy and sophisticated software supply chain attacks.

It's not that most people are bad. It's that it doesn't take very many people behaving badly to ruin a common platform on the Internet since one bad actor can use cloud compute and bots and other resource pools to launch huge attacks. Add anonymity to the mix and you've got a recipe for easy high-impact low-risk abuse.


Underrated truth bomb right here.


XMPP didn't fail. It's used everywhere. I can look at my firewall traffic any time and I always see XMPP traffic from the various Big Tech companies you already know.

But, like commenter tetraca mentioned, XMPP has been embraced and extended, and it's not very interoperable, as implemented.

SMTP has been and will always be around forever, albeit eventually viewed like FAX is today. I personally view it like that today.


XMPP was a nightmare. Too configurable. The exact opposite of HTTP's "just work" mentality. All the interesting functions were in XEP's and some of them (chat rooms, e.g.) were just insanely complicated. Classic death by committee.


As a rabbit trail/pet peeve, I think there needs to be _some_ standard in this space. It is silly to me that iMessage/SMS/Google Talk/Facetime/etc. can't all just interop on some open standard. I understand there are marketing forces pushing against it, but we do it for email, why not chat & talk? Totally ridiculous that I have to think "is this friend using Android or iOS?" when I text or call them. Even worse for group texts, plus means that windows/linux users can't easily message through their iphone. I haven't dove into it, but XMPP seems like a good candidate.


A common mistake tech people make is assuming that things that bother them must also bother a large number of other people enough to incur sizeable switching costs to solve it.


Nah, the other people are just wrong.


XMPP is already a RFC Internet Standard.

It's just that getting federation, interoperability and backwards compatibility is much harder than inventing your own new thing + users gain the freedom to move to a different provider (without loosing all your contacts).


The web killed the internet.

As developers we look too often for technical reasons for a limitation or failure but usually the social reasons are more significant.

Getting a client installed on a computer was socially difficult like convincing some one to install an app on their phone. Having a browser installed on every computer and phone made it trivial to get a client on a computer the only down side is being forced to use http and websockets as the protocol.

DNS and WHOIS used to be protocols in their own right but both are being replaced by http, I wouldn't be surprised if http came for smtp next.


> The web killed the internet.

This is an under-appreciated observation, and adds useful context to a lot of the other answers given here.

> I wouldn't be surprised if http came for smtp next.

I hate to be the one to tell you, but:

https://jmap.io/


Tbh I wish SMTP had died a long time ago. It's the reason email is so terribly broken. We can't trust a sender is who they say they are. We can't stop spammers, we can't hide personal info, there is no e2e encryption. As a result everyone is afraid of clicking on links and companies no longer send confidential info through it. It's just a fancy notification service now ("there's a message waiting for you at our portal" crap). I understand why they do it but it's really poor UX having to log into portals just to read a message.

Of all the original internet protocols, SMTP is the ice that needs replacement the most IMO.


In spite of all the valid criticism, we still got lucky that we ended up having this nonproprietary, decentralized, universally supported medium called email. A large number of businesses still (and will continue) to run on email, as does a certain subset of open source and other projects (using mailing lists).

> companies no longer send confidential info through it.

It depends, B2B absolutely still do.


> In spite of all the valid criticism, we still got lucky that we ended up having this nonproprietary, decentralized, universally supported medium called email. A large number of businesses still (and will continue) to run on email, as does a certain subset of open source and other projects (using mailing lists).

What I meant was that Email as a whole should be overhauled, though SMTP is by far the weakest link there. All the kludges and workarounds like DMARC, SPF etc don't fix the issues, and this is why so many companies are inventing the wheel. For example, Microsoft won't let your server deliver mail to them if you don't have enough 'reputation', meaning you must send a certain number of emails that are not spam. Otherwise you will find yourself in the doghouse (I've commented on this more detailed before). Even if you never sent spam in your entire life and your server isn't on any blocklist!

SMTP is decentralised yes and that is a great thing. But such measures like MS are doing totally undermine this decentralisation. And it will continue to do so, it's already super hard to run your own small-time server for the issues mentioned below. This decentralisation is already disappearing. It's time for a new protocol that is ready for the future.

> It depends, B2B absolutely still do.

They do for now because people involved in sales are usually hungry for business and don't care about the risks. They want to have low barriers for the customers, which is understandable. However there are significant risks depending on how the customer does their mail. We're on O365 and comms with other companies that are also on that are obviously pretty secure because they never go unencrypted through the public internet. Again a point that Microsoft (and Google) make into a selling point for their services, further undermining decentralisation.

We in the cybersecurity realm are imposing rules, e.g. any confidential info must be shared through secure filesharing. But it's hard to avoid this happening.


I'm generally wary of suggestions to overhaul a working long-lasting sociotechnical system. Everything that has been around for decades is patches on patches.

The the world wide web, ipv4, IRC, the US constitution, the telephone system, email, were all developed in older times, and as times change, they develop flaws which need (ugly) patches. That's a sign that they're successful, that no one company has come and replaced it and then killed it when things got inconvenient.

Think about the opposite situations: Google Groups replaced Usenet, and now it's completely dead; Google Reader took a lot of marketshare from RSS and almost killed it when they shut it down.

So let's not hastily destroy the only online communication system we have left, that's not controlled by a mega tech company.


I agree normally, but in this case email is just not fit for the original purpose anymore. It's undermining its own relevance in today's world. We don't trust it anymore and its key aspects like decentralisation are being undermined.

Personally I think google groups was not a replacement as such, more an extension of usenet. Usenet also had some very serious problems that made it hard to maintain and more specialised discussion fora took over. I don't think it was google groups as such, that was more one of the many ways to interface with usenet.

I'm absolutely not advocating replacing it with something a mega tech company has their hands in though! Any replacement should be decentalised. I would love to see an email replacement tech along the lines of what Matrix is doing for the chat world.


> It's time for a new protocol that is ready for the future.

People have been saying that for 30 years (maybe more?)

The reason it doesn't happen is that every bright idea to create a new email is built on removing vital functionality from what email is. So you end up with something which solves some of the headaches of email, but is crippled to the point it is either not interesting, or becomes simply yet another walled garden so what's the point.

Email is what it is because it is completely open. I don't need to rely on any particular entity or be approved by anyone to allow me to create an account or log in (under threat of being locked out). It is 100% decentralized, nobody owns it. It can be forwarded (try that with most walled gardens), piped to a shell for unlimited possibilities, it supports creating ad-hoc mailing lists, automation in any way you can imagine, archiving, unlimited searching, there are no possible restrictions on which client you use to read or send or process, or different ones for each. It works on everything under the sun.

Any replacement is either crippled by removing some of the important characteristics, or simply becomes the same thing but nonstandard.

That's why email has been around since the 70s. While imperfect, it's far better than anything else.


SMTP doesn't really prevent end-to-end crypto for email. There's a lot of reasons why that's hard that have nothing to do with SMTP:

  - multiple devices sharing a mailbox
    - store in plaintext?
    - if store in ciphertext, how to index, search?
    - how to share decryption, signing keys among devices?
    - how to enroll devices?
  - how to exchange keys?
These problems are easier to solve for IM because it's interactive, not store-and-forward.


It doesn't prevent it, no. But the protocol for email as a whole should incorporate it in a simple manner.

In terms of "End to End" I would consider the user the endpoint, not the device. There's many techniques that can make that happen. But this is more on the receiver side of course.

PGP is obviously not simple. S/MIME is also not simple. This is why nobody bothers using either (though S/MIME is used internally within a lot of corporations).


The only way I can see of retrofitting end-to-end encryption into email is to use DNSSEC / DANE. But this will require new MUAs.

DANE for SMTP, however, has created pretty good email security -- hop-by-hop, yes, but it's much better than before.


If all you care about is hop-by-hop security, you can accomplish that without DANE (which requires a forklift upgrade of DNS records across the Internet to work) --- this is why Microsoft and Google came up with MTA-STS.


If there is one practice that I'd like to see more encouraged, it would be for websites to start giving other forms of account verification beyond "we will send you an email". Zoom bought keybase, why couldn't push for a "login with keybase"? Why can't sites send a link via matrix/xmpp/whatsapp/signal message? The only service I've used so far doing it is njal.la.


I prefer email but why can’t it be encrypted. Why can’t I log into my bank and optionally upload my pgp public key?


Because given the rest of the software ecosystem available, maybe 0.01% of their customers would ever use such a feature.

It's way easier to just properly secure website logins with any number of 2FA methods, and rely on good old TLS for your encryption.


Regardless of SMTP and XMPP being from different eras, we can make these high level observations: SMTP was something designed for and deployed at the enterprise/institutional level. Whereas XMPP relied on word-of-mouth spread and adoption among individuals.

I tried XMPP early on, and found the software to have a user experience that was utter garbage, so after that I didn't give it a second thought.

At the enterprise level, though, garbage UX is the norm. SMTP is hard to deploy, but you have paid full-time sysadmins whose job it is to figure out the necessary parts of sendmail.cf or whatever else. You tell them "make it so" and they do.

I suspect there was no enterprise level push for XMPP anywhere.


> I suspect there was no enterprise level push for XMPP anywhere.

At the about the time when federated XMPP could have gained momentum at the enterprise level, 2005-2010, SaaS and then cloud began to take off. Enterprises began to shy away from hosting their own services. Even though many were still hosting their own e-mail and web servers, it was already too late for XMPP, especially given all the other factors--IM wasn't yet a killer business app, etc.

Without a large, diverse group of players effectively forcing federation, there was nothing countering the incentives for big players to create walled gardens.

I worked at company, Barracuda Networks, that acquired an enterprise XMPP appliance product around 2005. They also hired the main developer of one of the most popular open source Windows XMPP clients to help them integrate enterprise features (LDAP, etc). After a few years they basically put the project on life support. If there was more enthusiasm, and Barracuda worked harder with other vendors to build a large installed base, improving the overall value-add and "creating" a market, maybe things would have worked out differently. But that's a big ask for a bunch of medium-sized, short-term profit-seeking companies. Eventually startups like Slack ended up doing something like that, but using proprietary, walled-garden, hosted services, a model where they could capture entire markets--i.e. a big enough potential pay-day to make it worthwhile. :(

Another alternative scenario: maybe if anti-trust regulators came down hard on Google when they began to close-off federation. GTalk could have been (and briefly was) an important anchor tenant for a nascent federated IM ecosystem. Not sure it would have been legally viable, though, to restrain Google that way given the state of IM at the time.


> I suspect there was no enterprise level push for XMPP anywhere.

Gmail and Facebook adopted XMPP. But it seems like it wasn't a good fit for their business model of shutting users in and collecting as much data as possible, so they shut it down eventually.


Before the pandemic hit and management decided to move to MS Teams, we have been using Cisco Jabber which — by the name — obviously seems to use XMPP.


How can XMPP have failed when I and many others use it daily? When many things like Snikket and Zoom and WhatsApp are built on it?


I think the argument for failure is because barely anyone is using it directly with an XMPP client. But as a back end layer for other platforms, it's a smashing success.

According to the XMPP offical website to add some numbers to what you mentioned:

    ~800 million    WhatsApp
    ~200 million    Zoom
    ~4 million      Grindr
Matrix is also (one of) the more popular decentralized chat platform, which is built on XMPP.

I wouldn't at all be surprised if there's chat services that have you connect your local app with XMPP to, and get a simple UI layer that only connects to specific servers and other provides other config settings.

I've also heard of other uses besides chat, like for server management or video game multiplayer games for example. It's a great protocol that's provides delivery logic, accounts, etc. but keeps the actual messages simple and flexible for whatever you want to do.

I think SMTP "won" from being focused on email management instead of a more general protocol for multiple uses. (not that you can't do wacky things with SMTP, of course)


XMPP is in actually in Cisco VOIP too, and that Cisco VOIP Unified Communications [1] stuff is incredibly popular.

The client side Cisco Jabber app is Cisco's version of it that works with their VOIP.

Believe it is both Client & Server, and that is an interoperable implementation that looks like a reskinned pidgin somewhat in older versions, but has a great deal more functionality now, and looks capable of doing VTC and other things [2]

Considering how popular Cisco VOIP is, I think lots more people have XMPP functionality than they realize

[1] https://www.cisco.com/c/en/us/products/unified-communication...

[2] https://www.cisco.com/c/en/us/products/unified-communication...


> Matrix [...] built on XMPP.

Matrix is not built on any existing Internet Standards for messaging and is not compatible with XMPP in a meaningful way.


>Matrix is also (one of) the more popular decentralized chat platform, which is built on XMPP.

It most definitely is not built on XMPP in any way.


Huh, I thought it was. My bad, thank you for the correction!


It's built on the experience of the XMPP ecosystem ;-)

But tech wise you're probably right.


They may use the protocol but it's not interoperable - you can't chat with someone on Zoom from WhatsApp; that's what people mean by "it failed".


His other example, Snikket, is self hostable and interoperable with the wider xmpp world. The pattern is largely related to companies wanting to build walled gardens. On that basis, most open source tech has "failed" in the sense of having created greater value in VC backed companies with marketing departments than in communities without them. It isn't really surprising that it works that way, and IMHO it isn't failure.


That's really the key and the end of it - SMTP was big enough that all the companies had to work with it, even the ones that originally wanted to walled garden (Lotus Notes and Microsoft Exchange are two big names, even now Exchange still has X500 stuff in there).

The closest thing to that which chat has is text messaging, and I'm honestly surprised that MORE platforms don't have a text messaging "bridge" of sorts.


WhatsApp switched to the Signal protocol several years ago.


Signal protocol refers to a cryptographic protocol not a messaging protocol. XMPP also has an Signal protocol implementation called OMEMO.


Only for two-way chat, plus the default is still not encrypted (using Signal protocol) for WhatsApp.

In short, data-at-rest still can remains unencrypted for majority of the messagings on the WhatsApp’s server for all those who may be interested in them … UNLESS you enabled Signal/Encrypted option for two-way.

Group chat for WhatsApp?, it’s still the proverbial pants’ down.


No. All communications in WhatsApp are encrypted end to end, whether between 2 persons, or inside a group [1].

I believe you are referring to Telegram which does not use encryption by default.

[1] https://www.whatsapp.com/security/WhatsApp-Security-Whitepap...


I must be having a brain fart moment. Make that Telegram.

My bad.


Signal as in the chat platform? Fantastic if so.


XMPP is only kind of decentralized.

As far as any individual user is concerned, loss of the server(Which their identity is tied to) would be a hassle. Just like email.

SMTP got really entrenched really early. It's what gmail uses. They can't really do a proprietary mail protocol, SMTP was around way before them and nobody wanted to switch.

Instant messaging for the masses was proprietary from the start, as far as I can tell.

There was IRC, but it seems like by the time the internet was everywhere, people were already on AOL and ICQ and a bazillion other proprietary ones offering various extras and integrations.

None of them seem that interested in open protocols.


SMTP was a way for email systems to communicate - and so things like CompuServe jumped on the bandwagon early.

Universities and others using mainly UUCP also migrated because of the benefits.

There wasn't really a huge benefit for the big instant messaging players to integrate, and the dirty secret is cross-platform "instant" messaging has always existed - it's SMTP! Many people still use email as a form of a chat tool.

XMPP also suffers from the "oh shit security might be important" that SMTP originally didn't deal with (it's horribly insecure by modern standards) which greatly increased the complexity.


> SMTP originally didn't deal with

More so, absence of security is what allowed SMTP to proliferate.


XMPP was never that popular... it was IRC that was the (relatively) widely used open protocol.

SMTP was early enough and good enough to get widespread adoption. IRC in its original form was never good enough (esp. re: scaling), so services like AIM were able to step in and were good enough and open enough[1] solution while the open source world stuck with IRC for too long because it 'works for us'. So open standards like IRC never got entrenched the way SMTP did in proprietary solutions and by the late 90's the open source world had become pretty weak on developing new Internet standards that would even get widespread adoption in the open source world.

It's worth noting that there's nothing etched in stone to say that SMTP has won indefinitely. Try setting up a purely open source SMTP server on the Internet these days and see who you can talk to. Google has been making it harder and harder[2] to use non-Google clients with their servers. Slowly but surely, business interests seem to be enveloping SMTP with their own walled gardens... it's just taking longer.

[1] Well, sort of. They actually seemed to want to be proprietary about it. But every time they changed the protocol, the apps that worked with it were able to reverse engineer their changes quickly and AOL didn't fight too hard to prevent them from doing so.

[2] Or at least enough of a chore so that most people won't bother trying.


AOL didn't fight too hard?! AOL fought the hardest of any of them! AIM interop was by far the most notoriously difficult challenge for alt client devs back in the day because they explicitly fought alt clients, whereas MSN Messenger, ICQ etc didn't bother.

The trick I remember the most was really quite clever - the server started sending hash challenges to the clients i.e. a random byte range into the client binary. The client had to respond with a hash within a timeout or else the server disconnected them. The trick was, because the challenges were randomly generated you couldn't make a database of them, but also, the copyright license on the client forbade people redistributing it. Eventually it was worked around of course, people set up servers to answer the oracle queries that clients could relay to, but it really disrupted the alt messenger scene for quite a while. And in turn that meant it screwed people who wanted to use Linux because AIM was the number 1 social network in the USA during this period (though it's all forgotten now and was never as big in Europe, where MSN dominated).


IRC was a different usage model too. While you could do direct messages in IRC, it was a side feature and not the primary use case for it. XMPP, AIM, and the like are focused far more on direct messaging with chat rooms being a clunky add-on.

Anecdotally I experimented with an XMPP server for awhile and found it far more difficult to start up than an IRC server. It was far more enterprise oriented, assuming you had control over the DNS (only one server per domain of course) and with lots of complexity around LDAP or AD integration. This is as opposed to an IRC server where you downloaded the software, optionally exchanged keys with a peer, and started it up.

Using XML as a message format was also a mistake, but not one so bad that it would sink the entire project. ASN.1 would have been a better choice.


I used to chat on IRC between 2005 to 2014. We tried to switch to Jabber several times but it was simply too much work (we had elaborate mIRC bots, for example) and it simply wasn't as good - bloaty clients (this was in times when we had 512 MB RAM and single or dual core), chat servers were frequently down while IRC always worked (with the occasional netsplit, but oh well)...

Then we tried Slack but most people didn't come there too much. Today we have a Telegram group and it's so much better UX/UI wise...


Telegram is backed by xmpp ;)


Cool, but the coolest thing about Telegram is that IDGAF, it just works


XMPP wasn't adopted by large organizations who valued interop and federation. They valued the implementations, but almost invariably used it to build a walled garden.

I'm not entirely certain why this was, but I suspect that valuing extensibility over interoperability (or the perception of it) was part of it. Optional extensions and interop with hosts that used a different set simply gave hosts/operators too much to think about in terms of deploying the service for users.


It has nothing to do with the protocol, it's the penetration of the services that used it.

In the 90s email addresses along with domain names became part of professional fashion. Everyone wanted @mycompany.com addresses, and that created one of the obstacles for walled garden email. And ISPs often offered email-addresses to subscribers.

"The Microsoft Network" was an attempt at a walled garden that failed.

Hotmail etc had no choice but to support the existing open protocols. Hotmail was just one of many free webmails.

No instant messaging network used "global identities" (or hierarchical identities/addresses like email). The early-mover ICQ was a walled garden, and it died quickly when it became outdated. ICQ logins/identities had no relevance to anything else.

If instant messaging platforms that wished to form a network had existed, then adopting some protocol would have been trivial. Creating new a network of services/operators competing with already an established platform is not trivial.


Forget all the stories about GAFAM implementations, network effects or complexity.

It's 2022, XMPP is 23 years old, and I still can't use it as a primary solution for my community. There is no client/server environment available that meets the most demanded purposes and runs on all common devices.

In 5 years I have tried to set up an infrastructure 3 times to have a look, I always come back to Matrix (just because I have more hope that it will turn out better in the future).

XMPP is ready to do too many things, that's a good point, but the most common practices that everyone finds on centralized solutions are not implemented on all platforms, clients, and servers.

The server documentation doesn't give any advice on which clients to use, and the clients doesn't advise which server to use. Everyone has their head in the sand.

It's been at least 15 years since it's easy to set up an email infrastructure by yourself, that everyone can use on all their devices.


Matrix is also incredibly complicated at this point. A chat room in matrix is like an append only log with rules about precedence to ensure even when federated the resulting state of the room makes sense.

That might be the biggest contribution matrix has to offer.

The json encoding is large and seemingly under defined. Many things only seem to work with element and the python server.


> A chat room in matrix is like an append only log with rules about precedence to ensure even when federated the resulting state of the room makes sense.

Doesn't sound very complicated, yet?

> The json encoding is large and seemingly under defined.

Examples?


SMTP was already entrenched in business uses and thus will survive for some time. XMPP was used by some chat servers. I used XMPP/Jabber in a company but we could have easily switched to something else such as IRC or NNTP. Moving off SMTP would be a non starter for most businesses.

If there were an alternative to SMTP I believe it would be NNTP given it is similar in concept and just introduces threaded messaging which SMTP clients try to mimic now. There are several potential rabbit hole discussions of why moving away from SMTP are highly unlikely to ever occur.


SMTP was there early and with significant adoption - it's not inherent stability or quality of design. Early success and adoption tends to both entrench and freeze protocols despite their deficiencies. XMPP never got the sort of adoption to make it entrenched and its deficiencies were noted far earlier in its lifecycle. Moxie Marlinspike's piece on some of this is well worth reading:

https://signal.org/blog/the-ecosystem-is-moving/


Daniel Gultsch's response is also worth reading: https://gultsch.de/objection.html


Seven years haven't been particularly kind to that (fairly slight, I think), objection. The specifics change but the principal points Marlinspike raises haven't. Of course, XMPP is more dead than ever, SMTP is still around and taking on the piece by moving the goalposts from protocols to something else hasn't become more effective.


> XMPP is more dead than ever

What are you basing this on? The opposite is the case: https://takebackourtech.org/xmpp-comeback/


It's declared dead by the people where every single one of your 20 nerd friends in your xmpp contact list has shut down their server, and that happened like 8 years ago in my case. We were on IRC before that and we're still on IRC, that's why IRC is alive.


That link says XMPP advocates think XMPP isn't dead.


From that post:

> because someone’s choice to use an XMPP client or server that doesn’t support video or some other arbitrary feature doesn’t only affect them

It was actually even worse than that, because clients also have complete freedom in which audio and video codecs to support. So you would end up with client 1 and 2 both supporting video, but not the right protocol, so it didn't work, and it was hard to figure out why.


Isn't XMPP almost ubiquitous?

I thought it's what everyone used for messaging and push notifications. I know Google extended it beyond compatibility around when Hangouts came out and they started doing browser notifications. Apple stopped providing sources for ejabberd in the last few years, so I assume they moved off that for their iChat/messaging backend, but isn't iChat still an XMPP client?

Aren't most SMS messages routed via XMPP?


Wikipedia tells me that iChat was XMPP-based, but was discontinued in 2012 and replaced with iMessage, which is based on Apple Push Notifications and not XMPP.

Google has also discontinued Hangouts and who knows what you're supposed to use now. Their chat stuff has been so chaotic I really can't keep track. (Allo? Duo? Plus? Google Talk? Wave? Chat?)


> isn't iChat still an XMPP client

That's really cool. I would love to send XMPP messages to my friends who use iPhones without having them register on a public chat service and downloading a client.


XMPP is popular in military and public safety applications.


SMTP is not such a great success either. It has fading into irrelevance for a long while already. Private comms have been shifting towards facebook/whatsapp/whatever, and corporate which held to email longer is now shifting towards slack/teams. Even with email SMTP is becoming less relevant with the rise of Microsoft-Google duopoly. Come to think of it, SMTPs decline and XMPPs failure coincide pretty well.


SMTP came before corporations offered their own email solutions. XMPP came after AIM and others had already staked out their claims on the chat space.


SMTP got a toehold before the eternal Corporate Age of the Internet practically ended open protocol development, because it was far more useful on intermittent connections, and on terminals one only used a few minutes a day.

Chat didn't get super-useful until everyone had a screen in front of them or on them just about every waking hour. People used it before (I know, I was there, and I did) but it was more of a "see who's on and chat with them" than what it is now.


Chats have always been super-useful for a more technical crowd, IRC is still alive somehow.


I'd say it was the opposite. In 1990 there were numerous non-SMTP email services such as Lotus Notes, uucp-based email on Unix systems, the email system in VAX/VMS, etc.

SMTP made these all interoperate and ultimately replaced them all.


Like Lotus Notes, Microsoft Exchange comes with its own protocol for internal communication and it is more popular than ever, and there is nothing stopping Microsoft from slipping in some ‘superior’ new proprietary protocol for mail transport between installations, apart from that it’d be a lot of work and not very useful for them.


XMPP failed like RSS failed - the original vision for the protocol effectively died, but the protocol is in wide use despite that.

Essentially, XMPP's federation goal was antithetical to the vested interests of messaging providers so it was ignored. But the protocol itself was relatively well designed and so formed the basis of a lot of what we use today.


Alternative take: all protocols die a slow death, being replaced by centralised services. XMPP was replaced by WhatsApp. SMTP is being replaced by Gmail. HTTP will probably get replaced by some binary Chrome protocol, etc. At some point network effects kick in, and you just don't need a decentralised protocol when most users are on the same system.


HTTP was already replaced by HTTP/2, which is mostly SPDY rebranded - a protocol developed by Google. They weren't satisfied with that so now HTTP/2 is being replaced with QUIC, a binary protocol implemented first in Chrome.


around 2008 a german mass email and hosting provider holding chose every email address being a valid xmpp address, too. It was about 10 millions each for GMX.de and Web.de alone.

They never managed to promote pleasant onboarding and finally shut it down ('nobody wants such') months before WhatsApp rocketed - with ejabberd.


SMTP is simpler. XMPP is extensible.

SMTP was first so one registered accounts with email and not XMPP. Companies send stuff like bills via email.

There were various popular providers -- today Gmail is the most popular one but others like AOL were popular before. It's hard to change one's address.

Sometimes, your ISP gives you an email address.

Email is formal, so job application happens also over email (at least in not very technical jobs).

Probably because email was so important, there were and are still more people who host themselves, including companies (companies with their own xmpp setup are rare).

The rest of email is today (mostly) at Facebook, Twitter or LinkedIn.


IIRC, It was successful for a long time, but then when Google started supporting it, they captured a lot of the market because gmail was so popular, and it had xmpp chat support built into it with gchat.

Eventually, for various reasons, Google decided that an open xmpp based system didn't work for them, and they closed it and extended it. This was effectively the end of xmpp as a distributed tech that many people use today.

Interesting, in a lot of ways SMTP is no longer an open system. It is extremely hard to operate a mail server in such a way that your valid emails get delivered correctly and not marked as spam. So for example in the name of fighting spam, it is extremely difficult to participate in this "open" system.


I’ve always assumed it was because it came after IM chat clients were already popular? I think ICQ for instance was measuring hundreds of millions of users before the first jabber/xmpp announcement?

I vaguely recall there being something else as well as ICQ but we’re talking back to high school for me so well out of the range of good/clear memory.

I think messenger also came into existence at around the same time as jabber so concurrent development presumably meant completely independent protocol, so it wouldn’t even be subject to embrace/extend/extinguish - this was the height of MS monopoly so presumably it meant essentially “everyone” instantly had messenger?


Goddammit I forgot AIM, that was the other one.

So by the time jabber and xmpp came around there were giant existing networks: AIM, ICQ, Messenger. Obviously none of them had any business reason to adopt an open, etc protocol.

Additionally even if they wanted to, actually doing it would likely have been a significant engineering challenge, due to the need to maintain compatibility between client versions.


XMPP was difficult for most to understand. Hardly plug/play.

Same issue with Mastodon, in my opinion. People want to grab something from the Store or Play, sign in, and be where everyone else is.


Because SMTP was absolutely necessary for everyone and XMPP wasn't - and moreover, there was active interest in making XMPP fail. At first, all competitors supported XMPP to lure users, and in the end they dropped the support. Same old story as always, see Slack shutting down their IRC gateway.


It's mainly because XMPP is overcomplicated, but also they're for different use cases.

XMPP:

  - is designed for chatting and P2P and VoIP over persistent HTTP connections
  - requires everyone to be online
  - servers have to implement 50 different specs
  - works at the frontend and backend
SMTP:

  - is designed to send one large message in bulk over a regular short-lived TCP session
  - is a store-and-forward messaging system, highly resilient to slow or inconsistent networks and server issues
  - is simple and the ecosystem is layered, so the core server doesn't have to implement a dozen specs
  - is practically backend-only
The difference between "Can you hear me now?", and putting a letter in the mail.


This isn't really accurate: xmpp isn't p2p, it is hub and spoke. All comms (except optionally some VoIP and file transfers) go through the server. XMPP can also do store and forward pretty easily with common server addons.

I'd not really describe it as fragile or bloated either, having used it for twenty years or so with no real problems except the now-solved absence of e2e encryption (OTR, OMEMO).

Still, clients could be better.


What an XMPP server supports:

  - XML
  - XHTML
  - vCard
  - Bookmarks
  - Message Archive
  - P2P
  - VoIP
  - Multi-user Chat
  - File Transfer
  - Data Forms
  - Service Discovery
  - Pub-Sub & Personal Eventing
  - Bidirectional Synchronous HTTP Streams
  - WebSockets
  - Push Notification
  - Protocol Gateways
  - TLS
  - OTR
  - PGP
What an SMTP server supports:

  - SMTP
  - TLS
175 XMPP specifications (https://xmpp.org/extensions/)

24 SMTP specifications (https://en.wikipedia.org/wiki/Simple_Mail_Transfer_Protocol#...)


But SMTP also failed in a sense (and for the same reason). Big mail services may use it internally, but try to spin up your own mail server reliably.


cheogram has an SMTP<->XMPP gateway now (along with others.)

Aren't open protocols and interoperability great?


If anyone is wondering how to use it see https://smtp.cheogram.com/ .


Timing and demand. For many, the cost does not outweigh the benefit. Migration is a b**.


Google is killing SMTP. Try sending a large file these days.


The classical Betamax vs VHS question?


Here's a slightly different take.

SMTP dates from the earliest days of the internet and you don't really see new decentralized/federated protocols after about 1995, which is when IM started to be developed. But that isn't by itself an answer, it just raises the question of why open protocol development mostly stopped in that era.

I think a big reason why was the advancement of networks and hardware made it feasible to actually build and run large internet connected datacenters for the first time. In turn that meant you could run a server farm to which everyone could connect to, without running massive modem banks and the like.

In the early days of the internet that wasn't the case. Servers were machines running in people's offices and the ecosystem was extremely fragmented. There were lots of different kinds of network and email system, SMTP was really a sort of hack to bridge different systems together. The internet was congealing, being patched together out of lots of existing stuff, and big companies mostly didn't care. Networks were often only intermittently connected, the internet was for hobbyists and researchers, not 'serious' companies like AOL or Microsoft, who didn't "get" the internet at all until some famous memo by Bill Gates (they were pushing their own closed network). SMTP reflects this world with its handling of offline servers, lowest-common-denominator simplicity etc.

Once big companies realized the internet mattered and datacenter / networking technology got far enough along, the era of decentralized federated infrastructure died out pretty quick. This is in many ways very sad, but it's also in some ways inevitable. The decentralized cooperative standards-based approach sounds romantic, but it can be described in other much less noble ways e.g. as a form of committee-controlled pure communism in which nobody owns anything.

Phrased that way it's maybe obvious why that model of development has failed: there's no incentive to do anything good. SMTP isn't good. It's like an MVP email protocol. XMPP isn't that good either in many ways; you certainly wouldn't design a protocol that way today. And of course the issue here is not just the protocol but the entire stack - the servers, and most especially the clients. Every federated protocol system always suffers badly from a fragmented and min-viable client landscape, which certainly held back XMPP a lot.

Who is incentivized to make something good? Well, people who have an ownership stake in the results. The prospect of a big payout is what lets you attract teams of the most skilled people who will work crazy hours to build a service people love. It's the capitalist model - wall off the garden and charge an entrance fee. Suddenly finding gardeners isn't a problem anymore, because you can just pay them instead of asking them to volunteer.

The lack of ownership causes all sorts of subtle problems in the federated protocol landscape, beyond the obvious ones like a proliferation of low quality clients written in people's spare time. Probably the worst is that there's little innovation. People become motivated to build a federated/open protocol mostly when they see a successful proprietary product and they'd like to have the benefits of that product without the loss of freedom, dependence on a single vendor, having to pay for it etc. These communities are always going where the puck was, not where it's going now. By the time they finish cloning one cool feature their proprietary competition has, it's too late, that competition already moved ahead and added something else. This really nailed XMPP because networks like MSN were constantly adding features that XMPP couldn't easily respond to, and you see it again with stuff like stickers in Telegram. No open source dev is going to spontaneously decide to add stickers if they haven't seen that feature before, because it'd require recruiting a ton of artists and would seem trivial. Proprietary software companies are much more likely to come up with popular stuff like that. They find it much easier to build multi-disciplinary teams.




Join us for AI Startup School this June 16-17 in San Francisco!

Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: