
Ask HN: Worker queue with multi-dimensional rate limiting - mgliwka
I want to distribute tasks to worker nodes using a queue. The work depends on external resources.<p>The access to those needs to be rate limited across multiple dimensions (i.e. customer id, city, phone provider, there may be only 1 per customer id &#x2F; second, 1000&#x2F;city second and 20 &#x2F; phone provider&#x2F;second). The rate limit needs to apply to all worker nodes. My worker nodes share the same network.<p>How can I achieve that while maintaining a reasonable throughput? Any experience?<p>My naive attempt would be to use one leaky bucket per dimension implemented in Redis and check those prior accessing the resource. If there&#x27;s no token in all the necessary buckets available, I would put the request back at the end of the queue.<p>Are there better ways to implement this?
======
hguhghuff
I’d do it in Postgres and use sql columns for all your qualifiers.

Refer SKIP LOCKED

------
brudgers
What _business_ problem are you trying to solve?

