
Verk – An Elixir job processing system backed by Redis - geordidearns
https://github.com/edgurgel/verk
======
mootpointer
I don't understand why we're rebuilding Resqueue/Sidekiq in Elixir. This seems
like a problem which is solved largely by OTP.

Yes there are issues with back-pressure and over-filling mailboxes, but I
can't help but wonder why you'd want to bring Redis into the picture when
asynchronous processing is built-in for you.

Similarly if you want more robust queueing semantics, isn't RabbitMQ there?

Am I missing something?

~~~
edgurgel
OTP won't out of the box persist jobs that you want to guarantee that it will
run. It won't also retry and it can't schedule jobs to happen in the future.

Unless you are willing to lose the jobs that are just in memory you can simply
use Task and send messages.

RabbitMQ also does not handle retries with backoff, scheduled jobs, etc.

Disclaimer: maintainer here.

Edit: I added the "with backoff" to not mix with automatic reenqueue if you
reject a message from a Rabbitmq consumer.

~~~
mootpointer
Fair point. Why then persist to redis, rather than using something like mnesia
and keeping it all in Elixir?

Is there a compelling reason to add an additional piece of tech in my stack?

~~~
edgurgel
That's a great question. I chose Redis for 2 reasons:

* Simple to use

* Compatibility with other systems like Sidekiq/Resque. Out of the box it will understand their clients.

My experience with Mnesia is close to none but I know it's not trivial to deal
with split brain scenarios for example. It's also complicated to connect
multiple nodes for my use case (using docker) and also to deal with moving
instances inside a cluster. I would need to care about the locality of the
data.

One can use any Redis service (AWS Elasticache, Redis Labs) and not care about
maintaining a database. The Verk user just needs to have a working Redis
database.

I'm pretty sure that having the database in the same space as the job
processing system would speed up things. I would love to see a project that
tried this approach. Probably not using JSON but simply Erlang terms.

~~~
mootpointer
Seems reasonable enough. I just feel like shouting from the rooftops "redis is
not a queue server" but you already know that :)

~~~
w-ll
Honest question, what is redis then? I've used it for all sorts of reasons, it
seams like a swiss army knife, caching, queing, pub/sub, persistence, non-
persistence all-in-one utensil. You can do a lot with it, and if it's already
in the stack, it's a lot easier for others to reason with.

------
out_of_protocol
Why not [https://hex.pm/packages/exq](https://hex.pm/packages/exq) ?

~~~
edgurgel
Here's a detailed explanation:
[https://github.com/edgurgel/verk/issues/2](https://github.com/edgurgel/verk/issues/2)

------
walterstucco
why Redis when Elixir supports the Erlang native ETS and DETS libraries?

