
Show HN: Dramatiq – An alternative to Celery - Bogdanp
https://dramatiq.io
======
adambrenecki
> Dramatiq is licensed under the AGPL and it officially supports Python 3.6
> and later. Commercial Licensing is also available.

Oh.

> I’ve done a ton of open source work over the course of my career. Companies
> have used that work to generate income for themselves. Inevitably, I’ve
> ended up supporting that software on those companies’ behalf for free and
> that is not sustainable long term.

This is totally 100% true and understandable. Unfortunately, a lot of devs are
going to go "hmm, I could battle to convince someone that can approve
purchases that it's worth forking out $2k/year for this, or I could just use
Celery".

~~~
Bogdanp
Yep, that's understandable. If someone makes a cost-benefit analysis and
decides that they prefer the cheaper option then that's perfectly fine with
me, I'll still be here a year later when they realize their mistake. ;)

That said, I do give out commercial licenses for free for companies that are
just starting out. My goal with this is not to get rich, it's just to get
people to contribute something back in whatever way they can.

~~~
nine_k
I'm sorry about your decision. AGPL is not a way to make corporations pay for
your code. It's a way to make corporations _stay the hell away_ from your
code. This may be a legitimate goal, but apparently not in your case.

Due to the viral nature of AGPL, they have hard time trying your code to
decide if it is worth using. Due to copyright assignment and patent clauses,
they have very hard time contributing anything back.

What more or less works is GPL + commercial license, the freemium model. You
need to get enough free users on your bandwagon to show corporate users that
the code is worth trying, and maybe paying for. You need to get corporations
hire you to provide support and develop special features they need in the paid
tier.

Not that this problem is completely solved, but likely you heard about e.g.
MySQL or Nginx who use the freemium model.

~~~
acemarke
I've always been kind of unclear on how the AGPL works, even after reading
various TL;DR-ish explainers. What happens if you link/import AGPL code in, vs
running an AGPL service as a separate process as part of your architecture?

------
bjt
RabbitMQ is not hard to use directly. I think most developers would be better
off doing that, and coming to understand the power of AMQP, rather than
adopting cute DSLs like this and Celery that put you into a box and narrow
your view of what's possible.

I started that way with Celery and Django, maybe 6 years ago. At one point we
found a bug with Celery not resolving 'chord' callbacks when all the parallel
tasks had completed. It was difficult to debug, going through Celery's layers
of code that try to make various backends present the same interface. We
weaned ourselves off Celery and started using just the RabbitMQ and Redis
libraries directly. It was definitely a shift worth making, allowing us to
make performance/reliability tradeoffs that were better suited to our systems,
and opened our eyes to possibilities with RabbitMQ and Redis streams that we
hadn't been able to see when looking through a Celery lens.

If there's a place for DSLs like this, it may be when you have very novice
programmers needing to write quick throwaway jobs without wanting to spend a
lot of time learning the underlying systems. Maybe analysts on a data team,
for example.

~~~
whalesalad
Seriously? RabbitMQ is a huge waste of complexity for like 95% of modern
webapp use cases.

I use tools like Dramatiq specifically because I want to abstract away the
complexity of exchanges, bindings, queues, dead lettering, etc...

If you write out all the nouns associated with RMQ you start to realize there
is a LOT going on. It’s a very powerful system but convention over
configuration doesn’t work here.

~~~
moogie
No offense, but this sounds a lot like "I can't be bothered to learn about the
tech but I want to put it on my CV anyway".

The "95%" of use cases probably don't need a _message broker at all_. Far too
many "modern webapp" developers are shoving message brokers into websites so
they can pretend to be more important and sophisticated than they are. If
you're not even willing to learn how to connect to RabbitMQ, or any broker,
without a big giant messy crutch, I'd suggest you shouldn't use them at all.

------
coleifer
Huey: [https://huey.readthedocs.io](https://huey.readthedocs.io) \-- even
simpler :) redis or sqlite, mit license, greenlet/thread/process worker model
support.

~~~
whalesalad
I love your work (Peewee is a tool I use very often). I would have liked to
use Huey in a recent project but it doesn’t have modular backend support for
Rabbit.

~~~
coleifer
Thanks so much. You're right, I think some people have written storage engines
for rabbit (as well as mongo) but you might have to search for them. The
storage api is simple enough you could write your own in an hour or two.

~~~
whalesalad
I might need to do this! And thanks again for your work on Peewee... I think
people underestimate it’s power due to the funny name.

------
whalesalad
I’ve submitted code to this project and just want to state here for the record
that Bogdan is very responsive and accepting of contributions. It was a very
pleasing experience compared to other open source projects!

I reached out to the Python community recently with the question: “Django is
to Flask as Celery is to ______?” Dramatiq was one of the suggested responses
and the mission/purpose behind it resonated with me immediately.

I’m a big fan of convention over configuration. Most of the time, if I’m
writing software in Python, I don’t want to worry about a system that is not
Python. IE, I enjoy an ORM that abstracts PostgreSQL. I enjoy this because it
abstracts Rabbit. Obviously you need to understand the underlying system and
an ORM is no replacement for knowing the ins and outs of Postgres, just as
this is no replacement for understanding Rabbit. But... I love that it does
everything I want it to do with minimal or zero configuration.

------
SEJeff
Nice touch on having prometheus metric support builtin, but in general, if I
want a simpler celery, I always use rq[1]. How does this compare to rq? Having
this one the website somewhere obvious and why you wrote it would be a really
nice touch.

[http://python-rq.org/](http://python-rq.org/)

~~~
Bogdanp
> Nice touch on having prometheus metric support builtin

Thanks!

> How does this compare to rq? Having this one the website somewhere obvious
> and why you wrote it would be a really nice touch.

This exists although I suppose I should highlight it better:
[https://dramatiq.io/motivation.html](https://dramatiq.io/motivation.html)

I would say the biggest differences to RQ are:

* Dramatiq supports RabbitMQ in addition to Redis,

* Dramatiq uses a mixed multi-process/multi-thread model whereas RQ forks for every task. In my benchmarks, Dramatiq has much, much better throughput than RQ,

* Dramatiq has support for task-level priorities, rate limiting and delayed tasks,

* Dramatiq comes with an in-memory broker you can use for testing.

------
jtharpla
I figured I'd add a mention of the task queue we use at close.io:
[https://github.com/closeio/tasktiger](https://github.com/closeio/tasktiger)

We run several hundred tasktiger workers in production (deployed using
Kubernetes) to process the various backend task queues for our service.
Project is MIT-licensed if anyone is interested in another celery alternative.

------
rcarmo
I might have missed it, but is this asyncio-ready?

~~~
Bogdanp
It currently isn't. I'm holding out until there is some demand for it as I
currently don't need it for my own stuff.

------
vosper
Does Dramatiq have first-class support for SQS? That's my biggest gripe with
Celery. I mean, it does work with SQS, but it doesn't support it as a result
backend, and it doesn't do (eg) batch PUTs automatically, so it's pretty slow.

edit: I search the docs for SQS and got no results, so I'm guessing it's not
supported.

~~~
Bogdanp
It doesn't, but you're the second person to request it. Feel free to open an
issue and I'll take a stab at writing an SQS broker. Having looked at it
before, it didn't seem like it would take much work at all.

