> I’ve done a ton of open source work over the course of my career. Companies have used that work to generate income for themselves. Inevitably, I’ve ended up supporting that software on those companies’ behalf for free and that is not sustainable long term.
This is totally 100% true and understandable. Unfortunately, a lot of devs are going to go "hmm, I could battle to convince someone that can approve purchases that it's worth forking out $2k/year for this, or I could just use Celery".
I hope it works out for the author. If not, maybe there are other revenue generation options that could be paired with a less restrictive license.
It will be OK if the commercial option only applied to large organizations. But with the current setup, it just makes Dramatiq unreachable for the small players.
And we already have pricing models for small/medium sized businesses vs large ones. OSS could easily have similar models.
The way I think about it is what if the only database solution was Oracle. I'm sure many would have wrote their own limited buggy DBMS or used flat files instead. It will certainly be more expensive to write your own Oracle RDBMS from scratch instead of paying for the licence. But you don't need most of the features and for the ones you need, the price point does not make sense for all projects. For some projects, flat files or JSON files saved somewhere on the disk wouldn't sound so bad.
That said, I do give out commercial licenses for free for companies that are just starting out. My goal with this is not to get rich, it's just to get people to contribute something back in whatever way they can.
Due to the viral nature of AGPL, they have hard time trying your code to decide if it is worth using. Due to copyright assignment and patent clauses, they have very hard time contributing anything back.
What more or less works is GPL + commercial license, the freemium model. You need to get enough free users on your bandwagon to show corporate users that the code is worth trying, and maybe paying for. You need to get corporations hire you to provide support and develop special features they need in the paid tier.
Not that this problem is completely solved, but likely you heard about e.g. MySQL or Nginx who use the freemium model.
As a separate point, it's odd that anyone would use a license that they "don't fully understand," when (presumably) they would hold themselves to a higher standard of understanding the behavior of libraries they uses in their software itself. Is it irrational to consider this, at the very least, an indication of lack of attention to detail, and a tremendous red flag?
I started that way with Celery and Django, maybe 6 years ago. At one point we found a bug with Celery not resolving 'chord' callbacks when all the parallel tasks had completed. It was difficult to debug, going through Celery's layers of code that try to make various backends present the same interface. We weaned ourselves off Celery and started using just the RabbitMQ and Redis libraries directly. It was definitely a shift worth making, allowing us to make performance/reliability tradeoffs that were better suited to our systems, and opened our eyes to possibilities with RabbitMQ and Redis streams that we hadn't been able to see when looking through a Celery lens.
If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example.
Celery's task workflow "features" have bitten me in the past as well and I agree that it is a complicated piece of software (having had to go through its source code many, many times). That doesn't mean all software in this space has to be like that. That is why one of my goals with Dramatiq is for it to have a very simple and easy to understand core.
> If there's a place for DSLs like this, it may be when you have very novice programmers needing to write quick throwaway jobs without wanting to spend a lot of time learning the underlying systems. Maybe analysts on a data team, for example.
I think you underestimate the amount of value that tools like these bring to the table when it comes to actually shipping a product and getting things done. Of course, you should make an informed decision on the tradeoffs you're making, but calling anyone a "novice" for using an off-the-shelf solution instead of spending valuable time building his/her own is a bit much. :)
We ditched celery after many years of use in favor of just connecting to RabbitMQ ourselves. Over the last few years, it's saved a massive amount of time and effort and has made debugging and tracing issues so much easier it's rediculous.
I don't think many long term celery users really appreciate how much time is getting wasted. I understand how it may look like being able to ignore the underlying technology is a time saver, but that time easily gets blown away the second something doesn't behave as it should (and having gone through many of these problems, they're not exactly uncommon).
For anyone curious how much effort it took to write our own consumer/publisher, both are less that 200 lines of code and can be arguably cut down further without losing much (the author liked whitespace etc). Both were written in less than 2 days, including learning time, and are reusable enough to use in other projects.
I use tools like Dramatiq specifically because I want to abstract away the complexity of exchanges, bindings, queues, dead lettering, etc...
If you write out all the nouns associated with RMQ you start to realize there is a LOT going on. It’s a very powerful system but convention over configuration doesn’t work here.
The "95%" of use cases probably don't need a message broker at all. Far too many "modern webapp" developers are shoving message brokers into websites so they can pretend to be more important and sophisticated than they are. If you're not even willing to learn how to connect to RabbitMQ, or any broker, without a big giant messy crutch, I'd suggest you shouldn't use them at all.
I reached out to the Python community recently with the question: “Django is to Flask as Celery is to ______?” Dramatiq was one of the suggested responses and the mission/purpose behind it resonated with me immediately.
I’m a big fan of convention over configuration. Most of the time, if I’m writing software in Python, I don’t want to worry about a system that is not Python. IE, I enjoy an ORM that abstracts PostgreSQL. I enjoy this because it abstracts Rabbit. Obviously you need to understand the underlying system and an ORM is no replacement for knowing the ins and outs of Postgres, just as this is no replacement for understanding Rabbit. But... I love that it does everything I want it to do with minimal or zero configuration.
> How does this compare to rq? Having this one the website somewhere obvious and why you wrote it would be a really nice touch.
This exists although I suppose I should highlight it better: https://dramatiq.io/motivation.html
I would say the biggest differences to RQ are:
* Dramatiq supports RabbitMQ in addition to Redis,
* Dramatiq uses a mixed multi-process/multi-thread model whereas RQ forks for every task. In my benchmarks, Dramatiq has much, much better throughput than RQ,
* Dramatiq has support for task-level priorities, rate limiting and delayed tasks,
* Dramatiq comes with an in-memory broker you can use for testing.
We run several hundred tasktiger workers in production (deployed using Kubernetes) to process the various backend task queues for our service. Project is MIT-licensed if anyone is interested in another celery alternative.
edit: I search the docs for SQS and got no results, so I'm guessing it's not supported.