

Show HN: Martinet – Distributed Task Queue for Node.js - mpconlen
https://github.com/mathisonian/martinet

======
meritt
Generally speaking, using a database as a queue is a particularly bad anti-
pattern.

~~~
mpconlen
I guess this should be clarified more in the README - this uses zmq (and
potentially other brokers) as the queue, and uses the database for tracking
results, saving recurring tasks, etc

~~~
meritt
Good to know, so more like a crontab entry for the database.

Where's the actual queue though? zmq isn't a message queue, it's more akin to
a protocol. e.g. if I have N workers and M jobs, and M>N, where are all the >N
indexed jobs kept while the others are being processed?

Like it appears you're using PUSH/PULL pattern, which means the jobs queue up
in the publisher's memory (on a per PULL-peer basis..). So if the publisher
crashes, those jobs are lost. If the number of jobs queued exceeds the high-
water mark, those additional pushes are either going to discard messages or
block. If it blocks, it looks like your code is going to have lots of other
issues... If the worker crashes, I believe that queue is _also_ going to be
lost.

~~~
mpconlen
You are right about the push/pull pattern, in the case the publisher crashes
those jobs will be lost. That could be an issue depending on the use case and
would be a great thing to address.

If I am understanding the other notes correctly:

Neither the worker nor publisher should ever really block (even in cases where
M>N, N>M, publisher crashed, etc), mostly due to the async zmq bindings for
node. If there are a ton of messages being sent to a single worker this worker
will pull down messages as fast as it can and the rest will still be in
publisher queue.

If a worker crashes this is happening separate from the publisher and won't
affect the queue stored in the publisher's memory

------
fasteo
How is this different from beanstalkd ? From the README file persistence seems
like its main feature, but beanstalkd already has this (via a write-ahead
log), along with priority, delayed jobs, burying/kicking jobs, TTR, explicit
ACK (for distributed workers), etc.

------
bhouston
Just use the nodejs api for beanstalks.

