

Show HN: Redis-pipe – Use Redis lists like Unix pipes - morgenkaffee
https://github.com/lukasmartinelli/redis-pipe

======
fprotthetarball
See this for more inspiration (using pub/sub):

[https://github.com/whee/rp](https://github.com/whee/rp)

~~~
morgenkaffee
Ah that's the same idea! Wished I found that before. Thanks man.

~~~
fprotthetarball
The interesting thing about rp is that you can use it as a pipeline in your
other Go applications, as the Redis pub/sub "pipe" is exposed as a normal Go
io.Reader/io.Writer interface. Might be something worth expanding on somehow.

------
nissimk
Why are you using LPUSH/LPOP instead of PUBLISH/PSUBSCRIBE?

I guess you get storage that way, but only one consumer of each message. It
would be nice if it was optional in case you had a different application where
you need to have multiple subscribers to the same topic.

~~~
morgenkaffee
Yes you're right. I can create a flag --publish and --subscribe.

------
waitingkuo
What's the benefit to use Redis pipe instead of Unix pipes?

~~~
bytesandbots
I guess it might be very useful in a cloud like environment where you use this
set up to pipe data across scaled up servers.

~~~
morgenkaffee
I use it as a poor man's job queue.

cat jobs.txt | redis-pipe jobs

An then on several workers: redis-pipe --count 10 jobs | python do-work.py |
redis-pipe results

An then in the end: redis-pipe results > results.txt

Or you can use it as simple logging mechanism: tail -f /var/log/syslog |
./redis-pipe logs

~~~
eddd
Poor man's job queue is celery with redis backend i guess.

~~~
morgenkaffee
Mine is even more poor :)

