Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

My gripe with using a relational database as a priority queue is tends not to scale well. (Similarly as if you tried to use a relational database as a queue).

Once you get to the 10s of millions of messages per day (for example ~8M at priority X and 2M at priority Y), performance tends to go down the drain (due to contention), disk space bloats (due to vacumming/deletes).

I've yet to come across a good task scheduling system that will take into the account resources available and the priority of the tasks, while still usable on a largish dataset that isn't a pain to maintain.



Parent is only talking about keeping the job state in a database, not the queue messages.


How does that work? If the the queue is ultimately delivering the messages, doesn't that mean you can't have priority based queuing?




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: