derpy's comments

On: Robots.txt is a suicide note

[derpy]

ArchiveTeam mostly flashmob archives sites that are going to die an rm -rf / death, or atleast kill their users content.

Like Geocities, Yahoo! Video, Google Video, Friendster

On: Robots.txt is a suicide note

[derpy]

If it's public, it's public. Don't put private stuff up in public.

[smosher → derpy]

It's not about that. Explicitly so. Did you read the thread you're responding to?

On: Robots.txt is a suicide note

[derpy]

You do try to contact the "targets" sometimes though?

[sbierwagen → derpy]

Sure. We ask them to give us database dumps, when they refuse, for some crazy reason, we just download their website.

[dspillett → sbierwagen]

I can't say I approve of the "if the crazy fools won't give us what we want, we'll just take it some other way" attitude.

As the "some crazy reason" might be that the database contains data that is not publicly addressable for a reason and sanitising the data-set could be non-zero effort operation, letting you crawl the public pages and do some work at your side would be the sensible route from most points of view. Heck, even if there is definitely not anything not already displayed to the public in the DB, letting you go ahead and get the information the hard way is likely to be preferable to lifting a finger to send you a DB dump.

Sort of like a cat's attitude to playing fetch: You want it? You go get it. I've got more important things to do.

On: Robots.txt is a suicide note

[derpy]

Fuck you guy, that's pretty pricky. Is bandwidth that expensive for you?

[migrantgeek → derpy]

That's uncalled for. If he owns the data and pays for the connectivity, he should be free to do as he pleases just like I can exclude you from my home or business without justification.

[autoreverse → derpy]

I like humans browsing my sites but I certainly don't want scrapers copying my work wholesale, claiming it as their own and ranking ahead of me in SERPs.

I'm not alone :-) see http://incredibill.blogspot.com/