A combination of LISTEN/NOTIFY for instantaneous reactivity, letting you get away with just periodic polling, and FOR UPDATE...SKIP LOCKED making it efficient and safe for parallel workers to grab tasks without co-ordination. It's actually covered in the article near the bottom there.
Celery + RabbitMQ is hard to beat in the Python ecosystem for scaling. But the vast, vast majority of projects don't need anywhere that kind of scale and instead just want basic features out of the box - unique tasks, rate limiting, asyncio, future scheduling that doesn't cause massive problems (they're scheduled in-memory on workers), etc. These things are incredibly annoying to implement over top of Celery.
We don't hate Celery at all. It's just a bit harder to get it to do certain things and requires a bit more coding and understanding of celery than what we want to invest time and effort in.
Again, no hate towards Celery. It's not bad. We just want to see if there are better options out there.
But if you are too small for celery, it seems a hard sell to buy a premium message queue?
My top problem with my celery setup has always been visibility. AI and I spent and afternoon setting up a Prometheus / grafana server, and wiring celery into it. Has been a game changer. When things go crazy in prod, I can usually single it down to a specific task for a specific user. Has made my troubleshooting go from days to minutes. The actual queue and execute part has always been easy / worked well.
The Oban folks have done amazing, well-engineered work for years now - it's really the only option for Elixir. That said, I'm very confused at locking the process pool behind a pro subscription - this is basic functionality given CPython's architecture, not a nice-to-have.
For $135/month on Oban Pro, they advertise:
All Open Source Features
Multi-Process Execution
Workflows
Global and Rate Limiting
Unique Jobs
Bulk Operations
Encrypted Source (30/90-day refresh)
1 Application
Dedicated Support
I'm going to toot my own horn here, because it's what I know, but take my 100% free Chancy for example - https://github.com/tktech/chancy. Out of the box the same workers can mix-and-match asyncio, processes, threads, and sub-interpreters. It supports workflows, rate limiting, unique jobs, bulk operations, transactional enqueuing, etc. Why not move these things to the OSS version to be competitive with existing options, and focus on dedicated support and more traditional "enterprise" features, which absolutely are worth $135/month (the Oban devs provide world-class support for issues). There are many more options available in the Python ecosystem than Elixir, so you're competing against Temporal, Trigger, Prefect, Dagster, Airflow, etc etc.
> It supports workflows, rate limiting, unique jobs, bulk operations, transactional enqueuing, etc. Why not move these things to the OSS version to be competitive with existing options, and focus on dedicated support and more traditional "enterprise" features, which absolutely are worth $135/month (the Oban devs provide world-class support for issues).
We may well move some of those things to the OSS version, depending on interest, usage, etc. It's much easier to make things free than the other way around. Some Pro only features in Elixir have moved to OSS previously, and as a result of this project some additional functionality will also be moved.
Support only options aren't going to cut it in our experience; but maybe that'll be different with Python.
> There are many more options available in the Python ecosystem than Elixir, so you're competing against Temporal, Trigger, Prefect, Dagster, Airflow, etc etc.
There's a lot more of everything available in the Python ecosystem =)
> Support only options aren't going to cut it in our experience; but maybe that'll be different with Python.
That's totally fair, and I can only speak from the sidelines. I haven't had a chance to review the architecture - would it possibly make sense to swap from async as a free feature to the process pool, and make async a pro feature? This would help with adoption from other OSS projects, if that's a goal, as the transition from Celery would then be moving from a process pool to a process pool (for most users). The vast, vast majority of Python libraries are not async-friendly and most still rely on the GIL. On the other hand, Celery has absolutely no asyncio support at all, which sets the pro feature apart.
On the other hand, already released and as you said it's much harder to take a free feature and make it paid.
Thanks again for Oban - I used it for a project in Elixir and it was painless. Missing Oban was why I made Chancy in the first place.
> The vast, vast majority of Python libraries are not async-friendly and most still rely on the GIL. On the other hand, Celery has absolutely no asyncio support at all, which sets the pro feature apart.
That's great advice. Wish we'd been in contact before =)
Thanks. My two cents would be to not lock technical features behind a paywall. Lock "enterprise" features like encryption, FIPS, compliance reports, etc which make sense for a big corp. This would be far more palatable for someone small like my one-two person teams to adopt it and pay if we ever become big enough to care about enterprisey features.
I'm sure you are aware but sharing anyway. Django 6.0 shipped an API called Django Tasks for background jobs so all Django code can implement portable, backend agnostic background jobs and swap backends on the fly but there are zero actual backends out there right now one can use in production. If you could add inbuilt Django Tasks support to Chancy or create a `django-chancy` package that bridged it, I think you'd see a lot of early adoption by Django projects.
Sure, here's a Rust/WASM procedural skybox generator I threw together the other day, and is much, much faster at 16k renders then Javascript. https://tkte.ch/night-sky/
I might have missed it, but no mention of _where_ data is stored in the FAQ and seems critically reliant on Cloudflare.
In a changing world, what's the selling point for those outside of the USA? Why would our company pick this over self-hosting when our country is threatened with American annexation almost weekly? If I go with Zulip, mattermost, rocket.chat, matrix, etc I introduce maintenance overhead but I don't have to worry about unstable politics or a disliked tweet getting us sanctioned and banished from American-hosted services. The chat platform we use internally is critical business infrastructure and so we're required to ask these kinds of questions for business continuity.
I recommend you take a look at bunny.net, an Cloudflare alternative which is European and can support deno workers and so you are much more likely to be able to use it
Regarding Cloudflare r2, there might be many but I like the idea of Upcloud and you can get yourself say a 3.50 euros machine there and (they got unlimited egress! Wish I was sponsored by them lmaoo) and their block storage is around 20 Euros per month for unlimited egress. They provide 1-24 TB per month free unlimited but after that its capped at 100mbps for unlimited amount of time and I don't think that it could be an issue in this case? And you can always get more 3.50 euros servers to get more 100mbps unlimited so the possibilities are endless
OVH provides unlimited egress as well and OVH is another good bet. I love both for unlimited egress if what you are doing is very bandwidth intensive in the first place
Both are European. I have heard good things about scaleway too
Also searched bunny and looks like it provides some storage service too and unlimited egress to bunny cdn from that
https://bunny.net/pricing/storage/
If Dock runs on AWS or Cloudflare infra it is, by definition, a no go for me and many others here. Would like to get an answer on that.
FWIW: I use scaleway far a medium sized project and find it a good experience. Of course there are bugs and some things could be better. But support is good and the responses to bugs in their terraform provider are quick.
Also using bunny.net, happy so far.
Only thing I am missing is to create mailboxes (not transactional mail) on those two providers. I needed a third one for that.
Scaleway doesn't provide unlimited egress the last time I provided beyond 75 gigs if I remember correctly and scaleway feels great in some situations but I wouldn't try to force it in here if the project is bandwidth intensive.
I don't know but Upcloud gained my loyalty when I created my account and I don't have credit card but I could still use their service and go talk to their customer service for the 7 days my trial account is active and their support was so phenomenal and nice and just, man, They aren't kidding when they say that their support can have 1 minute times in things)
I do feel like Scaleway and Upcloud too might charge more than OVH and hetzner but they both have better support and long term its really worth it.
For what its worth, I really like scaleway's stardust servers as I had analyzed some servers in Lowendtalk and other websites etc. which had benchmarks and Scaleway's stardust development servers are one of the cheapest in the market for the development boxes but they are limited to 1/2 per account but I feel like scaleway does this to get people try scaleway in the first place and I do find this really fascinating idea and not many other do it.
EU cloud is definitely under-rated especially for egress related stuff (like this) because AWS,gcloud,Azure from what I hear costs like a bank of money for egress.
I do think that more people should evaluate the right option of cloud for their job using the right "tool" for the right job.
Glad that you liked scaleway tho. I haven't had experience with scaleway tho but I have heard that they have a good slack (ironic isn't it?) server where their devs are active.
Perhaps they will go to dock which might get one day hosted on scaleway, could be really cool :)
This rule of three structure too: "We are a Western European company, so GDPR and data sovereignty are at the heart of our architecture, not an afterthought."
In terms of data sovereignty and security, the location of your servers is irrelevant if you're a U.S.-based company, thanks to the CLOUD act[1] (emphasis mine).
> The CLOUD Act primarily amends the Stored Communications Act (SCA) of 1986 to allow federal law enforcement to compel U.S.-based technology companies via warrant or subpoena to provide requested data stored on servers regardless of whether the data are stored in the U.S. or on foreign soil.
Hard to understate just how expensive. Here in Montreal where ice storms kill and cause billions in damage, we still don't bury the main transmission lines. We been burying almost everything _in_ the city where having to repair millions of individual connections (again) would be impractical, but it's relatively simple to repair the limited major lines into the city.
From CBC:
> Current estimates are that it would cost five to 10 times more to distribute electricity to a big city via underground cables, and that not all of nature's problems would be alleviated even if that were done.
It's a tiny pittance of what would have been owed had they been taxed appropriately, and seems conveniently timed to bury the news of Dell (via Dell Federal Systems) funding ICE (it's completely erased it from the first page search results for "dell funds ice" in less than 24 hours).
This isn't badass, it's a disgrace. They've hoarded an incredible amount of wealth generated by others and returned a sliver of it. You've been so conditioned to accept this system that you're even grateful for the scraps.
If they gave $100 billion, they'd still have over $51 billion in the bank. That's roughly $7 million a day for every day he's been alive, or enough to feed every child in America for over a decade. Imagine the regional economic stimulus if instead of being hoarded for nearly half a century that money was paid out in salaries to those actually earning that money. _And_ they'd still have billions.
Can you explain what part of that is "communist nonsense"? Appropriate taxation (and leaving them with billions still even after that)? Or was it feeding hungry children you're against?
America had a tax rate of 91% on the obscenely wealthy for decades and around 70% until the 80s. Reducing this to historical lows has universally, by both bipartisan and nonpartisan[2] parties, been found to have been the primary driver of inequality[1].
At even an absurd 99% tax rate, applied equally instead of tiered, Michael Dell would have $1.5 billion dollars.
One of the major complaints with Celery is observability. Databased-backed options like Procastinate and Chancy will never reach the potential peak throughput of Celery+RabbitMQ, but they're still sufficient to run millions upon millions of tasks per day even on a $14/month VPS. The tradeoff to this is excellent insight into what's going on - all state lives in the database, you can just query it. Both Procastinate and Chancy come with Django integrations, so you can even query it with the ORM.
Chancy also comes with a "good enough" metrics plugin and a dashboard. Not suitable for an incredibly busy instance with tens of thousands of distinct types of jobs, but good enough for most projects. You can see the new UI and some example screenshots in the upcoming 0.26 release - https://github.com/TkTech/chancy/pull/58 (and that dashboard is for a production app running ~600k jobs a day on what's practically a toaster). The dashboard can be run standalone locally and pointing to any database as-needed, run inside a worker process, or embedded inside any existing asgi app.
If you like Procastinate, you might like my Chancy, which is also built on postgres but with a goal of the most common bells and whistles being included.
Rate limiting, global uniqueness, timeouts, memory limits, mix asyncio/processes/threads/sub-interpreters in the same worker, workflows, cron jobs, dashboard, metrics, django integrations, repriotization, triggers, pruning, Windows support, queue tagging (ex: run this queue on all machines running windows with a GPU, run this one on workers with py3.14 and this one on workers with py3.11) etc etc...
The pending v0.26 includes stabilizing of the HTTP API, dashboard improvements, workflow performance improvements for workflows with thousands of steps and django-tasks integration.
