

How to be below API rate limits from multiple servers - ad93611
https://callhub.io/blog/2014/02/03/distributed-rate-limiting-with-redis-and-celery/

======
akerl_
Reading the title, I was expecting to see a somewhat shady article on using
multiple servers to dodge API rate limits.

Instead, they've got a pretty snazzy writeup of how to effectively keep your
distributed processes working together properly. I'm curious why they didn't
pass a token rather than the actual Lua over the wire via Redis, but it
certainly seems to work for them.

------
rryan
I may be missing something but I think this could be done more cleanly with an
atomic counter or semaphore [1] in Redis.

    
    
      1. Have a task that releases 1 resource to the semaphore    
         every (1.0 / cps) seconds.
    
      2. Workers wait to acquire a resource from the semaphore 
         before making a call.
    

Due to the rate of release being fixed at 1/cps no worker can ever exceed the
calls-per-second limit. Much simpler than sending Lua over the wire.

[1] [https://github.com/dv/redis-semaphore](https://github.com/dv/redis-
semaphore)

~~~
erikcw
The only downside I see to this approach in the context of this post is that
then you'd essentially need to dedicated daemon sending releases to the
semaphore for #1.

I suppose you could use a long running celery task for that purpose. You
wouldn't want to fire of (1.0 / cps) celery tasks just for releasing
semaphores unless the value of cps was very small -- as it would congest your
queues with a bunch of small cleanup tasks. Which aren't guaranteed to run
precisely that often.

------
hox
Interestingly enough, Alan Shreve gave a talk at RedisConf a couple of years
back detailing something very similar used on the twilio stack to rate limit
draining a queue:
[https://inconshreveable.com/talks.html](https://inconshreveable.com/talks.html)

------
wheaties
So they send code to Redis to make sure that they won't increment too high? I
know Redis already has a ton of functions prepackaged but this seems like it
would be a good one: INCRIF.

