

Ask HN: I keep wondering how embed.ly handles rate limits - bigpak

Ask HN: I keep wondering how embed.ly handles rate limiting considering the number oof request they make to sites like youtube how come they are not blocked?
======
screeley
It's becoming more of an issue now that we are growing.

We have had conversations with the YouTube guys about this and they have asked
us to use API's instead of directly scrapping pages.

Quora is currently rate limiting us a ton as well. We again have to directly
reach out to them.

Most oEmbed endpoints aren't rate limited because they are generally not
resource intensive. They are also meant to be used client side and therefore
expect more traffic.

Embedly also does a fair amount of caching that reduces the number of calls
that we have to make. This limits us, because we aren't a straight pass
through, but helps us handle spikes of traffic.

------
iSloth
Using their service doesn't affect the traffic flow, it's still going direct
from content provider to end user.

