Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

Agreed. From a quick look at code, it seems it just fires off every fetch request immediately after another compeletes.

Hopefully it gets patched to have a built-in rate limit (X requests per minute/hour).



If it's already serial, and only works with one backend (rather than being an arbitrary mirroring tool like wget), then the Wayback server can easily "express its preferences" for rate-limiting by adding artificial delay to request-responses that pass the rate-limiting threshold. Backpressure shouldn't be the client's responsibility.

(It only is traditionally, because so many sites do nothing to protect themselves from "being too nice", so arbitrary-backend mirroring-client devs allow their users the option to ask for less than they want. This isn't a sensible protocol design, on either side; it doesn't optimize for, well, anything.)




Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: