One thing about how Instagram's load balancing that I don't like is that they rate-limit their proxies on image requests. In my recent testing, its roughly 5-6 requests every 3 seconds or so. Any requests more frequent than that return 503 status codes. I don't entirely understand why they do this, since their load balancer simply does 302 redirects to the S3-hosted image resource.
I can guess at some of the reasons, such as they didn't foresee a user loading more than a few images at once. Perhaps they perceive rate limiting as a protective measure.
However, I've done testing on Twitpic, imgur, and yfrog and haven't run into the same issues. Twitpic, for example, generates a lot more traffic than Instagram and they don't have the same rate-limiting.
> I don't entirely understand why they do this, since their load balancer simply does 302 redirects to the S3-hosted image resource.
S3 accesses cost money, so it makes sense that they'd rate limit access to them. A botnet hitting an S3 URL could incur large fees for the owner of the file very rapidly.
I can guess at some of the reasons, such as they didn't foresee a user loading more than a few images at once. Perhaps they perceive rate limiting as a protective measure.
However, I've done testing on Twitpic, imgur, and yfrog and haven't run into the same issues. Twitpic, for example, generates a lot more traffic than Instagram and they don't have the same rate-limiting.