Hacker News new | past | comments | ask | show | jobs | submit login
XMPP: A Communication Protocol for the Internet of Things (briskbraintech.com)
89 points by liotier 12 days ago | hide | past | web | favorite | 72 comments

> XMPP supports federation, which means that devices from different manufacturers connected to different platforms can talk to each other with a standard communication protocol.

The idea behind federation with XMPP for humans (aka, Jabber) is that my friend can have an account on server A (friend@a.com), and I can have an account on server B (me@b.com), and we can tell each other our addresses (because we know each other in real life), and then get messages to each other, which is cool, because we're friends, and both know English, and so will be able to meaningfully communicate. Not bad!

The idea behind federation with XMPP for IoT devices is...what? That my smart lighbulb can be talking to a control server run by Phillips, and my smart fridge will be talking to a server run by Samsung, and...they have no idea each other exists, have no ability to address each other, and have no ability to generate messages the other can consume, because they're completely different devices, from different manufacturers, with different protocols? Even if the fridge wanted to tell the smart lightbulb to turn on when the door is open, they need a shared "lightbulb control protocol", no? This feels like claiming email lets any two people on Earth hold a meaningful conversation; just because you can transport the message to the recipient doesn't mean they can understand the language of the message.

How does federation help here? Or any other XMPP feature? There's just no attempt to tie any XMPP feature or characteristic to the nominal use case.

It'd be cool if the fridge just announces "my door has just been opened", and a controller decides what to do with the lightbulb (or should the logic be on the bulb with some parsing like if (device == "refrigerator" and location == my.location and component == "door" and state == "open") { ... }?)

It'd be cool if the sunroof can open when it gets the message from the weather monitor that it's a nice day out, and the AC can switch off when it gets the message that the sunroof is opening...

Federation won’t solve tha.

Heck, the best way for your house is to use multicast, with a unique IP for each endpoint type, and a unique port for each distinct endpoint within the type.

The endpoint IDs could even be self assigned, with the idea that you’d only really need a gateway for use outside the home. Standard but extendable protocols can be developed too, so everything can interop but still include custom stuff beyond the standard.

Apple’s Bonjour protocol?

Bonjour is actually an XMPP extension. https://xmpp.org/extensions/xep-0174.html

Bonjour is Apple's name for their implementation of mDNS service discovery. It doesn't involve XMPP.

XEP-0174 is a peer-to-peer extension to XMPP which uses mDNS ("Bonjour") to discover other clients on a local network. For a while, Apple had an implementation of this in iChat.

Yes, you are correct. I was trying to say the same thing. (English is not my native language)

I think they still do, in Messages

Not anymore. I believe XMPP support was finally removed from Messages in macOS 10.14. (Almost nobody noticed.)

Interesting! Thanks for sharing this!

I just run a giant Javascript file on a Pi that does this. Devices just need to send data through LAN as dumbly as possible.

The federation does help here. Device manufacturers might provide built-in XMPP addresses for devices (fridge124529@samsung.kr, bulb981232545@phillips.nl, etc), and they can interact with each other and with home automation hub over a reliable standard protocol. And if you're not fine with the idea of having your lightbulb having a Phillips account you can configure everything to work with your home router.

And of course, you can get notifications from all these devices right in your chat.

> Device manufacturers might provide built-in XMPP addresses for devices (fridge124529@samsung.kr, bulb981232545@phillips.nl, etc), and they can interact with each other and with home automation hub over a reliable standard protocol.

Yeah, but they haven't. And almost certainly won't. And even if they did, in order to interact with each other they'll need to agree on a common language, which would be an enormous task.

But if that enormous challenge was solved, then yes, federation would be one of the ways you could let devices find each other, but there's others. For example, the home router you suggest could configure all your smart devices could also, say, act as a simple registry.

Saying "federation does help" oversells it greatly, I think. It's more like "federation could help, one day in the far future, if we solve a lot of difficult problems".

Well, I agree with you that currently federation does not help, cause it is not used. However, XMPP is actually very easy to use protocol. It only gets hard when you're building a chat application that does all sorts of things that are expected from a modern messenger. But receiving a message, replying, is easy, fast, and requires surprisingly low power. It is no coincidence why both apple and Google use a modified XMPP to power their push notifications. So supplying every 'smart' device with a capability to talk over xmpp potentially solves all interroperability issues. You just need to agree on message format, all other infrastructure is already in place.

Right. XMPP is merely a transport protocol. That solves one very minor problem which can be solved dozens of other ways. The contents of those transport packets have to make sense between devices. How do you do that?

Say you introduce a new device to the network. Whatever magic transport protocol you are using discovers it and your control hub takes ownership after some handshake process. Now, how does the hub—and by implication every other device on the network—know what that new device is and what it can do? It’s a lightbulb? Okay, that’s just on and off right? Oh, no it might have dimming capability. Maybe it’s a candelabra with multiple bulbs? Maybe the bulb can show different colors. Maybe it has a wake-up gradual brightening process built in. Maybe it has a sensor built in to adjust brightness and color temperature based on existing light sources. Maybe you want to disable that but also set the color to purple. How do you exchange all this capability information? And this is just a lightbulb.

XMPP federation doesn’t solve for that.

You're not wrong but any solution has to start with sending a message.

XMPP seems to have failed, sadly, for its primary purpose. No major vendor uses it for (federated) messaging, even those that used to. And that's where there's a clear benefit to users.

IOT vendors have a less clear use case and they've shown they're not particularly interested in openness or compatibility (or security). Why would this be any more likely to succeed for IOT than it did in messaging?

You're right that it has to start with sending a message, but there's no incentive for companies in this industry to ever take step 2 here.

Most major vendors did use it, and likely still do internally, but with federation turned off.

If you have a billion “users” using a working chat system and the concern is “with federation they can chat with outsiders and we can bleed users” you don’t just drop the whole thing, you turn off federation and prevent third party client logins.

> ... and have no ability to generate messages the other can consume, because they're completely different devices, from different manufacturers, with different protocols? Even if the fridge wanted to tell the smart lightbulb to turn on when the door is open, they need a shared "lightbulb control protocol", no?

That is what a message broker [1] is for.

[1] https://en.wikipedia.org/wiki/Message_broker

I assume that, assuming you can get custom software on the fridge, you could get an iq stanza over to the toaster (that also assumes that the toaster and the fridge each contact respective services that allows server-to-server/federation).

It's highly unlikely to pan out in the real world, there's a lot of assumptions there that Philips, Samsung + co are highly unlikely to satisfy.

People who worked extensively with XMPP in the past told me that it just wasn't cut for the current paradigm, with real-time and asynchronous communication, network hopping (mobile, wifi, etc) and just the way things evolved since... 1999. There were some changes planned but apparently the whole thing would be better off rewritten in a new form, and the effort never really kicked off, much like the protocol itself faded from use as we moved towards more and more hybrid networks and uses.

It's a sad realization, I always thought XMPP would save the day some day, but apparently there's a lot of work between now and there. And it might just as well be called something else. I think the latest move towards HTTP/3 enforcing SSL1.3 is one way to envision a fully secured web, 10 years from now.

We work extensively with XMPP, so I can speak with some authority on this topic.

It is true, the protocol was invented in 1999 for then-relevant conditions. There are good things - everything that is related to federation works remarkably well, and bad things - multi-device usage, reconnection, etc.

However, the very thing that is mostly panned here by Matrix proponents - modularity and extensibility - allows to uplift it for the modern world. Actually, we recently did just that, by developing a bunch of new protocols and implementing them on an XMPP server. I'll spare the verbose details, but in short, we redesigned reconnection process from a resumption of an XML stream to restoration of messenger state, and it works rather well. That set of protocol extensions greatly helps to improve the performance of our upcoming XMPP client.

(also in relation to your other response w. details) This is very, very promising! I'm so glad to hear XMPP is alive and kicking; I'll definitely explore these packages of yours. Thanks for the work and for sharing.

It's been about a decade since I had the oh-so-common billion-dollar idea (read: extremely costly but nowhere near profitable) of "hey, how about we used private servers/clients in peer to peer to communicate? Like MSN and descendants but without a central server?" I think me and the 100,000 others with the same idea felt the challenge of improving XMPP + working on conversion from existing platforms was too tall an order.

Now, if guys like you have been/are improving XMPP 'enough', then I can see a path for the open ecosystem to grow anew. And that, provided alternatives like Matrix etc. provide a path for data (API, URLs etc), is just what we need to massively disrupt the basic communication space.

Really, thanks. Spark re-ignited for social apps and a new social paradigm fitting our times.

Who are "we" and what client/projects are you working on? Are these changes new public XEPs that others can/will implement?

"We" are makers of Xabber, a popular XMPP client for Android. It had gotten a bit stale lately: a few years ago we figured out that XMPP improvement should start on the server-side, so we diverted all our resources towards creating a set of protocol and implementation it on a server, an alpha version of which was released last month, it has a built-in an insanely great web version of Xabber, which likely offers the best XMPP UX ever, on any platform. We also plan to release an iOS version of Xabber soon - this month or next.

Our protocols are public, and we plan to submit them to XSF, once we're done translating them from our native language. Hopefully, these XEPs would be supported one day by other clients. We believe they are crucial for XMPP future.

Very interesting. Which server are you enhancing? Do you provide xmpp hosting?

We forked ejabberd because it's the best existing XMPP server. We do provide XMPP hosting (at xabber.com/xabber.org domains), but currently, it's just standard ejabberd, a transition to our own build will come later, hopefully, this year. We also plan to provide domain hosting.

> apparently the whole thing would be better off rewritten in a new form

It seems to me that this is basically describing Matrix [1]. Funnily enough, since its current transport is HTTP, it looks like by the time HTTP/3 becomes a thing, network hopping would be solved essentially for free.

[1]: https://matrix.org

Was wondering what you mean by network hopping. Assuming that by http3 we mean quic, does it provide some functionality for message routing that I’m missing?

I’m most likely either misunderstanding what you mean by hopping or completely missing a feature if quic.

Thank you

From what I understand, QUIC allows for IP roaming, much like how mosh does.

Matrix is a fundamentally broken protocol held up by the prowess of it's UI design team.

How is it fundamentally broken?

Can you explain why? (Not looking for any great detail, just a starting point.)

As someone who has worked extensively with XMPP in the last ~6 years let me tell you that it works fine with network hopping, multiple devices and everything else you'd come to expect from a modern IM experience in the likes of WhatsApp and Signal.

I have written a lengthy article in 2016 in the state of mobile XMPP: https://gultsch.de/xmpp_2016.html

The situation has only improved from there.

I'll read it with much interest, thanks a lot.

You know, I like the sound of you and the makers of Xabber. The "myth" of XMPP being unsuited to mobile / IM is clearly overblown if you're only even half-right about it all (because then that means there's no "fundamentally broken" paradigm). It's great to hear the community is active nonetheless! ~

Ever looked at internet bill for things like GPS loggers using http?

Even most barebone request often eats more bits than data

That's why nor HTTP nor XMPP will be going anywhere

It’s funny... I have, sort of. A place I used to work at had a bunch of old serial devices connected to the internet via Ethernet-serial gateways. To query them, we connected over Telnet (heh). The original devices had a CLI, and the polling was done by sending a series of keystrokes to the devices. Each keystroke was one byte, and was wrapped in an entire TCP packet (~40 bytes, if I recall).

We were blowing through our 1MB (yes...) SIM cards quite quickly, and it was 98% just packet overhead. I can only imagine what it would have been like over http.

My GE Microwave uses XMPP for its control. It sends its status using a JSON object of hexadecimal addresses and values: https://brianhenryie.s3.amazonaws.com/2019/door_opened.xmpp....

I started reverse engineering it but haven't quite finished. The plan is to connect it to Home Assistant and negate any need to communicate outside my house. To get the above output, I wrote a tool for logging the communication in iOS: https://github.com/brianhenryie/xmppframeworklogger

For some reason the article doesn't mention it, but there is a few retracted XEPs for IoT (sensor data[0], provisioning[1], control[2], etc). I think it would be neat if they've incorporated Semantic Sensor Network Ontology[3] and were actually used. But it seems that in practice there are mostly custom and unspecified binary protocols, with inherent TOCTOU issues, awkward integrity checks, and with no security measures (anecdata, based on 7-8 years of dealing with various devices that can be classified as IoT).

Regarding the use of XMPP for it, as pointed in other comments, it indeed looks like unnecessary overhead, but I think it is quite similar to using HTTP for transfer of things other than hypertext: not a perfect fit, but already has good software support, standards, and additional features that are useful. And perhaps more importantly, no wheels to reinvent poorly and hide from customers.

[0] https://xmpp.org/extensions/xep-0323.html

[1] https://xmpp.org/extensions/xep-0324.html

[2] https://xmpp.org/extensions/xep-0325.html

[3] https://www.w3.org/TR/2017/REC-vocab-ssn-20171019/

Why XMPP? Seems a bit of unneeded overhead.

XMPP, a solution looking for a problem for over 20 years. :(

That's a little harsh. If it wasn't for Matrix, XMPP would probably be the best currently existing solution for a chat protocol.

I fear that's more an indictment of available chat protocols, than it is a defence of XMPP. :)

Indeed. XML itself is nontrivial to parse and the "infinite" extensibility is just overkill for probably 99% of the time I've seen it used, XMPP included.

As an aside, this article seems quite devoid of actual content, bordering on spam IMHO. The title mentions IoT but the term appears only once in the entire article at the end, in a vague "XMPP is for messaging and IoT also needs messaging, so.... XMPP and IoT!" way. This is the sort of page that clogs search engine results with useless non-content.

XMPP's own extensibility is a huge stumbling block as well. Almost every major feature of the protocol is implemented as an optional extension, so it's very difficult for clients to rely on anything being consistently available.

Well you have a choice. You make a protocol extensible and live with the fact that people are eventually going to extend it. Otherwise you choose a different fixed protocol for each and every task and live with the fact that you will never get any level of interoperability at all.

If one complains about extensible protocols, one should be willing to come up with some other approach that works.

You frame it like a bad thing, while in reality it's a good thing that makes a protocol more resilient and capable to evolve.

What exactly is nontrivial in parsing XML? Any popular computer language has plenty of libraries for working with XML.

When I'm building firmware for an ATMega or a Cortex-M0 or Tensilica device, the idea of dedicating the required space in my firmware for an XML parser, burning the CPU overhead for doing the parsing, losing the network overhead for the verbose text encoding... all this is absolutely absurd. Just so I can use XMPP? Why? That's my battery life getting wasted, that's my features getting displaced.

There's no place for XML in iot, and definitely no place for XMPP. Even json is luxury you often can't afford, not if you want to make a reasonable cost target or battery life.

The cases you describe do not require really hard xml parsing. It's trivial. Composing and sending an xmpp stanza is extremely easy. Parsing is easy too.

Why you might want it? Well, maybe for the same reason any home router has a built in web interface (despite all the overheads of having a kinda webserver on such device) - for user's convenience and interoperability.

And yes, stating that JSON is somehow better than XML for such cases is just... wrong.

I never understood the desperate attempts to use XMPP, and the inevitable failure of EVERY attempt to run a popular messaging and/or presence application on it.

If you want to IoT, why not just use NATS or something?

I am quite happy with xmpp being unpopular, as it means governmental pressure against E2E encryption will most likely never reach this place.

Any IOT protocol should be lightweight and ideally UDP, 3 ways TCP handshake takes too long, XMPP is supper verbose. Use HTTP like the mozilla IOT thing for that... MQTT QOS 0 is pretty ok.

Ideally you absolutely don’t use UDP if you care about data delivery, or you just reinvent a poor implementation of TCP. I want my smart light to come on, my doorbell to ring, and my dryer to tell me the cycle is complete 100% of the time.

yes, if you care about delivery you also dont use MQTT QOS 0, use 1 or 2. FOr sensors that report often (and you can loose some packets in the middle) or idempotent actions that you can repeat should be fine though.

At this point, please just let XMPP meet it's long overdue death.

Rumors of XMPP imminent death are greatly exaggerated.

I really wanted XMPP to work, but it was a hard protocol to like. It ticked a lot of the right boxes on paper, but in practice it didn't always lend itself to efficient or neat implementation. It didn't feel like it was designed by programmers for programmers.

(A feeling I'm very familiar with these days as I work in the telco business - an entire business that knows fuck-all about how you create standards that lend themselves to clear implementation)

It kind of started off on the wrong foot by using XML. XML is too complicated. There is too much freedom, and freedom to screw things up. It also puts too much flexibility and semantics in the wrong place. Complexity, gnarly performance characteristics and design dilemmas abound.

And not paying attention to important detail such as cheap and predictable message framing is such a boneheaded mistake - you have to pay attention to things that make implementation awkward. It is the first problem you need to solve when creating a somewhat performant implementation and you find yourself stumbling out of the gates having to compromise. Sometimes by having more than one XML parser - one to do framing and one to deal with messages.

It has been many years since I worked on anything XMPP-related. I have no idea if XMPP has improved (probably a good idea to spend a weekend playing with XMPP again to have a look). But if it still depends as heavily on XML, it is not for IoT.

Please do not put it in IoT devices. Please, please, please.

IoT spans from devices with systems more powerful than your desktop computer (in kilowatts and MIPS) to small systems that have perhaps hundreds of BYTES of RAM.

While microcontrollers are getting bigger, with some having megabytes of flash and even hundreds of kilobytes of memory, the most interesting growth areas are in devices that use very little power. Ofte as little as a dim LED. And when they sleep, which many do most of the time, they need to draw so little current you'll spend many sleepless nights poring over datasheets to figure out if there's a voltage regulator out there that doesn't blow your entire energy budget.

For the big systems you can do whatever you want. Go crazy. For constrained devices, not so much.

If you need some infrastructure for communication that is to be useful, you have to be able to scale down. Paradoxically scaling up is easy - scaling things down is hard. It tends to make systems grow ugly warts when you create adapters all over the place to translate between domains.

And it isn't just about size. It is about asymmetries in communication patterns as well.

More primitive protocols are quite popular (like MQTT and CoAP). Things that can be done with more or less constant memory and CPU requirements. But they're not necessarily great for everything. And people do not necessarily understand how to design with them. They tend to exist at the edges of systems and concern themselves with hauling data back and forth between constrained devices and the luxury world of more powerful computers.

Then there's the traditional embedded world where it feels like you are always dealing with hermits that got cut off from the rest of the world 25 years ago. Things are getting better as the embedded world is gradually being invaded by more mainstream programmers, but it is painful to watch. People who aren't very good at software versus people who aren't very good with constraints.

What is almost shocking is when you get a peek inside the kimono of different embedded companies / chip manufacturers and you realize the stark difference between where they are in a developmental sense. Some understand where they need to go, and mainly struggle with getting there - others just have no clue how far behind they are from where they need to be.

And then mix in politics, companies that are hysterically protective about their IPR, and the time pressures that exist, and you have multiple culture collisions at once.

I can understand the attraction of XMPP. I really can. But if you ask me we need to do messaging over again. And it has to do less - not more.

Far too verbose to take seriously. Small devices don't have enough memory to store all those strings, or do complex parsing. Then again we are discussing 'IoT' which is sort of like the tooth fairy... so basis in reality is optional.

XMPP was too heavy even for its intended purpose - chat.

MQTT is the way to go for IoT.

MQTT at the edge, Kafka at the core.

Do you know a better "less heavy" solution for federated chat? Matrix? That matrix which server can handle 50x fewer users than ejabberd on similar hardware?

Yes, Erlang was designed for fault-tolerant distributed scalable concurrent chat :)

There is a good Erlang MQTT broker: emqtt / EMQX.

XMPP for this purpose feels like an overkill. Open Sound Control might be an interesting alternative. After all, like MQTT, it originated in a world where equipment needed to talk to each other.

Offtopic: what are your thoughts on PJON (pjon.org)?

I guess if you have one really bad idea, you can't make it any worse by mashing it up with another really bad idea, right?

Ugh, is there a reason your web page doesn't work with my PAGEUP, PAGEDOWN or arrow keys?

There's a circle numbered "8" in a very bad place, for web developers who modify browser shortcut keys and scrolling behaviour.

There's a circle numbered "9" in that same bad place, for browser developers who refuse to allow end-users to easily stop keystroke-hijacking and scrolljacking with a simple preference checkbox.

Explaining what I'd like to do to either group would probably land me in circle #5, and carrying it out would land me in #7.

Works in Edge. All one of us is happy.

Works fine in Firefox, but Chrome they don't work at all.

What browser are you using? Works fine in Mac Safari.

Ironically the fault is due some code intended to make scrolling smoother but some bug is making it so instead it doesn't scroll at all (source: https://i.imgur.com/XIewJXS.png)

In practice, "smoother" almost always means laggy and sluggish. It's one of the things I always turn off upon a new install of a browser.

Guidelines | FAQ | Support | API | Security | Lists | Bookmarklet | Legal | Apply to YC | Contact