
Beer and Concurrent HTTP Pipelines - kiyanwang
https://damienradtke.com/post/beer-and-http-pipelines/
======
hrbrmstr
Rather than emoticon-ing away your responsibility in a potential site DoS,
perhaps read up on responsible crawling -
[https://webarchive.jira.com/wiki/display/Heritrix/Responsibl...](https://webarchive.jira.com/wiki/display/Heritrix/Responsible+Crawling)
\+ [http://blog.mischel.com/2011/12/20/writing-a-web-crawler-
pol...](http://blog.mischel.com/2011/12/20/writing-a-web-crawler-politeness/)
. You had no need for either unlimited or even 50 parallel crawl tasks. And,
if your intent is anything but personal use, you shld prbly (re)read
[https://www.brewtoad.com/legal](https://www.brewtoad.com/legal) since you
just got their attention in a pretty big way.

