What User-Agent do you use?
Do you crawl non-textual resources?
Do you save all headers from the crawled responses?
Do you perform any processing on the returned content (like de-chunking or de-compressing) or can it be retrieved verbatim?
If two customers request the same URL/site be crawled, are their requests merged so the site is only crawled once?
Do you save the exact time of the request (not trusting the returned 'Date' header)?
What User-Agent do you use?
Do you crawl non-textual resources?
Do you save all headers from the crawled responses?
Do you perform any processing on the returned content (like de-chunking or de-compressing) or can it be retrieved verbatim?
If two customers request the same URL/site be crawled, are their requests merged so the site is only crawled once?
Do you save the exact time of the request (not trusting the returned 'Date' header)?