
Ask HN: Craigslist and crawlers - kinkdr
I wrote a small crawler for Craigslist making absolutely sure that it obeys the robots.txt rules.<p>Sure enough, after a few hindered of requests my crawler got banned.<p>Doing a simple search at Google, I can see from the results I get that CL is happily allowing Google bots.<p>I strongly believe that this discrimination against small players is highly unethical, but my question is, is it breaking any laws or maybe anti-monopolistic rules?
======
jacquesm
Craigslist is a privately owned website, they can ban you at will.

That said, you've already given the solution to your problem yourself. Instead
of crawling craigslist, crawl google instead.

~~~
IanDrake
In the epic 3taps vs. CL battle, CL claimed copyright on all posts, which
wasn't supported by their TOS. So they updated their TOS to give them
worldwide exclusive rights to ads.

CL finally backed down when the EFF told them to, which says a lot because the
EFF is practically owned by CL.

~~~
jacquesm
> because the EFF is practically owned by CL.

Sorry?

I'm totally floored by that, did I miss something?

I know that CL donated some sum of money to the EFF but that was a settlement.

~~~
IanDrake
The EFF is an outspoken critic of the CFAA and has admonished multiple
entities for threatening criminal prosecution. When Craigslist threatened me
the first thing I did was contact the EFF. They didn't ignore me. They told me
there is nothing they were willing to to at the time, which in my eyes was
worse than just ignoring me.

Only until it became a high profile issue with 3taps did the EFF speak out
against CL.

That was about a year after I contacted them and they only did after 3taps
started getting press.

To me it was pretty clear that they didn't want to bite the hand that fed
them, until they had to or else lose all credibility.

------
IanDrake
Stay away from doing this. I built a small business doing real-time alerts on
CL and eventually got a C&D letter from their lawyers. I gave up.

I got around IP restrictions by essential having SETI like software running on
my client's computers.

They will threaten you saying you're committing a felony by violating the
CFAA. They are insane, but have deep pockets.

After I shut down, others stood up against CL's ridiculous claims and lost.
Search for 3taps.com and padmapper.com to learn more.

~~~
kinkdr
Wow! Thanks for the info. So sad that they lost. How is this not illegally
thwarting of competition?

I wonder how can they claim that what you did was illegal or against their
TOS, but what search engines are doing is not.

I guess I am lucky I didn't spend too much time on this. Thanks again!

