However, may I suggest you add to description on GitHub page some more info? Performance is only a part of equation and description only talks about that. Did you give any thought to reliability and robustness? How about security? It is written in C++ - was it fuzzed? Does it support TLS? Authentication?
These are much more important questions (to me) at this time (disclosure: using Mosquitto and happy with it, didn't hit bottlenecks yet). I have no doubt you will address them sooner or later, but it might be good to start working on some of them as soon as possible - or, if you already have this covered, to write about it.
Of course, I might not be your target customer, at least at this stage, in which case please disregard this advice. :)
I was very excited to see this post since I happen to be looking for a scalable high performance MQTT broker.
The first thing I noticed when I opened the GitHub page was the lack of documentation. Then I opened the source files. No headers describing what each module does. No method headers. No code comments. Nothing that would let me get a feel of how well this could fit into my project without doing a lot of code reading...
Maybe I'm just an old fart and need to accept that this is the way code is written these days, but with my own projects I always tried to make sure that at least minimal documentation was available and that the code was easy to read and understand before releasing it to the world.
I'm sorry but HackerNews thinks they know better so they mod the title into something misleading.
Original title was "I'm developing a new and faster MQTT broker" with extra focus on "developing". It is a 7 day prototype, not a finished product. The naming is not even known to be final at this point.
If you need a demo use case, we provide an open MQTT feed of some 1500 public transport vehicles, soon once per second (plus per-stop departure prediction feeds): host mqtt.hsl.fi, port 443 and 1883, topic /hfp/#
That's a pretty nice use case, especially with MQTT supporting persistent messages (which are messages that "stay around" and which the client will receive immediately upon subscription, no matter how long ago they were published). That way, you can publish the messages as rarely as they come, and a client that subscribes will still get the latest time available without having to wait for the next publish.
(The reason is that we use MQTT topics creatively: instead of one topic per vehicle, the topic changes dynamically to provide more filtering criteria such as the route and the next stop of the vehicle. Thus, most of the per-topic persistent messages would be obsolete.)
We try to be as open as possible nowadays. However, these backend systems are from an old proprietary contract. There's a computer with a GPS receiver and driver UI in each bus and tram sending the MQTT messages to mqtt.hsl.fi (IBM MessageSight hardware for now) over 3G cellural data connection.
The subway can't use GPS but the trains read RFID tags installed along the tracks and there's wifi coverage. The suburban trains have Raspberry Pi's listening in promiscuous mode in the maintenance Ethernet ports and relaying the vehicle data to us.
The REST API is part of Digitransit, which is a fully open passenger information platform and journey planner. Docs and code here: https://digitransit.fi/en
If you want to have a quick look at how the feed looks like, there is a web client for mqtt at http://www.hivemq.com/demos/websocket-client/. Don't leave it open for too long though, it might kill your browser (Chrome struggled a bit on my old laptop).
It's great to see a message broker that's optimised for performance - having looked at the source of a few of the more popular brokers it's clear that there's a lot of overhead in many of them (object allocation/boxing, multiple copies being made of the data, inefficient serialisation, verbose wire protocols and so on).
I'd be really interested to see how uTT performs compares to Aeron[0], which, while not MQTT, as far as I'm aware it is the fastest message transport around[1,2].
Please note my original title was "I'm developing a new and faster MQTT broker".
HackerNews changed it and I cannot edit. Point being (a) I do not even know the naming is final, (b) this is in no way a finished product, it is a 7 day prototype.
Interesting thread, and it is always good to see performance pushed by OSS messaging. NATS is lightweight and performant as well. Both are TCP based - any thoughts around other transports underneath (LPWAN, BLE, etc)?
I'm really excited about this. It's by the creator of uWebSockets[0], alexhultman[1]. It looks like it builds heavily on his work there. Based on the performance compared to redis, this is already an impressive start. Excited to see where this goes! I also second an attempt at comparing it to ZeroMQ, even though they're obviously somewhat different.
Are you considering support for MQTT-SN, or willing to take pull requests to support it? The ability to handle lightweight clients could be a very important differentiator.
This is very useful if it's faster than the competition. MQTT is one of those things that's so useful that you tend to overuse it, and extra speed/reliability on the broker is always welcome.
Nice work! I'd be curious to see a comparison w/ RabbitMQ's MQTT extension. By any chance have you done such a comparison? Also I assume the comparisons are done at QOS level 0?
I have in fact tested RabbitMQ but it was not at all impressive. The two fastest brokers I have found are emqtt and Mosquitto, both good at either large or small broadcasts but not both.
> The two fastest brokers I have found are emqtt and Mosquitto
What about vernemq? I [ran across][1] a comment a while back about some tradeoffs with emqtt's usage of mnesia. Haven't gotten around to trying verne myself yet though.
We can have a big bench-off with more brokers later on. I have tested maybe 6 and from this I have decided to write my own. I wouldn't spend time reinventing the wheel for free if I didn't have a reason for it :) Then you also have memory usage which is impossible to optimize in VM/GC languages.
I am happy for you, hope this grows!
However, may I suggest you add to description on GitHub page some more info? Performance is only a part of equation and description only talks about that. Did you give any thought to reliability and robustness? How about security? It is written in C++ - was it fuzzed? Does it support TLS? Authentication?
These are much more important questions (to me) at this time (disclosure: using Mosquitto and happy with it, didn't hit bottlenecks yet). I have no doubt you will address them sooner or later, but it might be good to start working on some of them as soon as possible - or, if you already have this covered, to write about it.
Of course, I might not be your target customer, at least at this stage, in which case please disregard this advice. :)