
RCP 11 – Stream data type proposal for Redis - neo2001
https://github.com/redis/redis-rcp/blob/master/RCP11.md
======
judofyr
Yes, please. I've never understood why someone wants to do pub/sub and only
base the reliability on TCP/WebSockets. The concept of "fire and hope everyone
who wants the message still has a connection open" always seemed fragile to
me.

This has been the reason I've recommended implementing long-polling instead of
WebSockets for real-time applications. And every time I see a real-time
solution which _only_ uses WebSockets I try to steer away from it. Once you
have a reliable data model (which includes log position, retrieving old
messages etc.) it's just as simple to implement long-polling as WebSockets.
With WebSockets-only solution I can't help but think they base all the message
delivery reliability on TCP.

This proposal looks like a clean (very Redis-like) solution, and I immediately
see use cases for it.

~~~
luchs
Did you look into the EventSource API? It's basically automatic long-polling
with a simple file format built into the browser. It supports pretty much
every "real-time" pattern: regular polling, long polling, and streaming.

The browser handles the log position for you (via the id field and the Last-
Event-ID header) and automatically reconnects when the server closes the
stream or the connection is lost.

The proposed Redis API seems to fit extremely well to this model. My previous
usage of EventSource with Redis worked by sending the entire state whenever
someone (re-)connects and using PubSub afterwards. This works well for me, but
likely doesn't scale very well.

~~~
bradgessler
The only big barrier to adopting the EventSource API is the complete lack of
support from Microsoft's browsers:
[http://caniuse.com/#feat=eventsource](http://caniuse.com/#feat=eventsource)

~~~
lobster_johnson
Fortunately, polyfilling is fairly easy, e.g.
[https://github.com/Yaffle/EventSource](https://github.com/Yaffle/EventSource).

------
dunkelheit
That's cool. In case you haven't seen this blog post [1], it provides an
extended discussion of the uses of this abstraction.

One comment about the groups API is that while it is very convenient, it seems
a bit fragile - if a consumer is nuked immediately after it gets a fresh batch
from redis then this batch is lost forever.

[1] [https://engineering.linkedin.com/distributed-systems/log-
wha...](https://engineering.linkedin.com/distributed-systems/log-what-every-
software-engineer-should-know-about-real-time-datas-unifying).

------
sintaxi
Salvatore is a machine. Since 2012 Redis has been a main tool in my preverbal
belt and it has always seemed to have forward momentum.

~~~
smt88
Have you paid for Redis? If so, was it direct donation, Redis Labs services,
or committing code?

Edit: Not asking to be a jerk or insist that s/he donate. I use lots of things
heavily that I've never contributed to. I'm just curious how the Redis
community works.

~~~
sintaxi
No, only used the open source library on my own.

------
thesorrow
Apache Kafka is a beast. Would be awesome to have a light alternative !

~~~
jon-v
Agreed. I'm currently considering Kafka for a project but am hesitant because
ideally we need a zero maintenance solution, and Kafka seems to always end up
requiring some handholding.

I'm not familiar with the Redis dev process at all. How long might one expect
it to take for a feature of this complexity to make it from RCP to production-
ready?

------
louthy
This looks very interesting. I use Redis as a backend for an actor-system [1]
as part of a larger functional framework for C# [2]. I use RPUSH to add
messages to an actor's queue along with PUBLISH for saying 'new message' to
the actor that should check its inbox, this is then followed by LINDEX 0 to
peek at the item at the head of the queue, before calling LPOP when the
message has been successfully processed. If I'm understanding this correctly,
that process could be wrapped up in a single stream?

[1] [https://github.com/louthy/language-
ext/tree/master/LanguageE...](https://github.com/louthy/language-
ext/tree/master/LanguageExt.Process)

[2] [https://github.com/louthy/language-
ext](https://github.com/louthy/language-ext)

~~~
lobster_johnson
Yes. Instead of multiple operations, you would get this from a single
operation. But instead of modifying the log on every "pop", the operation
would merely update a "read position" on behalf of the client. The position
semantics would allow you to jump backwards in the log if needed.

~~~
louthy
Excellent. This is very exciting! (well, for me at least)

