

Craigslist Casualty: The Story of ApartmentsNearMetro.com - wlindner
http://williamlindner.com/projects/ApartmentsNearMetro.html

======
homedog
It's the inevitable fate for anything that uses CL for data. CL is too afraid
of competition, yet refuses to innovate. It's a shame.

~~~
a5seo
In defense of CL (which I have never before offered), I noticed some UI
improvements recently: pic view, grid view, and map view.

These are the first noticeable innovations on CL in many years. Might they be
the result of the on-going parade of sites improving upon CL w/ their own
data, not to mention the criticisms on HN and elsewhere?

Whether they 'hate' competition is debatable. Clearly they view their data as
THEIR property. They occasionally let things like PadMapper go while putting
the ban hammer down on other things, very arbitrarily.

The Wired story on them about 5-6 years back suggested they are just very
skeptical of third parties and human nature in general. It left me with the
impression that Craig is just deeply selfish and authoritarian.

~~~
wlindner
I've seen the UI improvements, but haven't needed to use them so I don't know
how good they are. If those work: Awesome! In a way, that's a victory. If CL
incorporates the best things about 3rd party sites into their own site, I
think that's a big win for consumers. If CL refuses to allow 3rd parties to
exist because they are skeptical of human nature. Talk about the ultimate
irony: a site that lives off of people's willingness to trust strangers being
unwilling to trust strangers themselves.

~~~
a5seo
Good point. Here's the quote from the end of the article:

"These all signal Newmark and Buckmaster's wariness about what humans,
including themselves, might do if given the chance. There may be a peace sign
on every page, but the implicit political philosophy of craigslist has a
deeply conservative, even a tragic cast. Every day the choristers of the
social web chirp their advice about openness and trust; craigslist follows
none of it, and every day it grows."

HN discussion: <http://news.ycombinator.com/item?id=784663>

------
pm90
If the application is blocked, can the author please openSource the script? If
everyone starts using the script, I believe it will be quite impossible for CL
to block it.

~~~
wlindner
Yeah, that's a good idea. Craigslist basically blocked any connections from
Google App Engine to their domain. But, if you ran the site from your own
computer, they wouldn't really have any way to block it.

------
typicalbender
Similar story as <https://www.padmapper.com/>. It's sad that craigslist isn't
willing to either update their application or allow someone to make it more
useful for their customers.

------
jdavis703
I had that exact same pain point in 2011. It's too bad the Craigslist refuses
to innovate or let others innovate on their platform.

~~~
wlindner
Yeah! It almost seems like a source of pride for them... What were you working
on that got squashed by Craigslist?

~~~
jdavis703
I was simply trying to locate an apartment near Metro. I toyed with the idea
of scraping their site but remembered this Wired story from 2009 [0] so I just
gave up on using them for my search.

0:
[http://www.wired.com/entertainment/theweb/magazine/17-09/ff_...](http://www.wired.com/entertainment/theweb/magazine/17-09/ff_craigslist?currentPage=all)

------
vampirechicken
If you want to compete with craigslist, build a better product and convince
people that there is value for them to enter their data. That's what
craigslist did.

------
TimMontague
FYI: this is unreadable on my WP8 device. The screen width is set smaller than
the text width so I have so scroll left and right for every line.

~~~
wlindner
Thanks for letting me know, I implemented a quick fix for mobile devices.
Unfortunately I don't have a WP8 to test it on, but it shouldn't require
scrolling left and right anymore to read the text.

~~~
TimMontague
Thank for fixing!

------
ScottWhigham
I don't get why people say things like, "It's too bad the Craigslist won't ...
let other people innovate on their platform." It's just childish in my view.
Here's another child's toys - and, granted, he has a LOT of toys[1] - and he's
chosen to share them to every child in the world regardless of their
background, yet you're upset because he's not sharing them in such a way that
would only allow people to play with the toys on another playground. "He"
doesn't want that - he wants to play, play, play with everyone. So he decides
to say "No!" to anyone who wants to take his toys off of his playground.

If you don't like the way "he" plays with his toys, then tough cookies - "he"
doesn't care and neither does anyone else except those who want to also
complain about him not sharing in the way they want him to. You'll see this
damn near every time someone posts a "Help! My API provider just changed the
rules!" type of post. There are two types of responses: "I hate that - they
should be more open", or "It's ridiculous to even think that you can build a
business based off of leaching from someone else and that that other entity
would be okay with that forever."

[1]Unlike children (who typically are given their toys), these toys are the
result of "his" work.

~~~
largesse
_Unlike children (who typically are given their toys), these toys are the
result of "his" work._

Er, no. They are the work of everyone who posts on CL.

------
oniTony
So there's Walk Score. Apartments near (Washington) metro example —
<http://walk.sc/YPUq1a>

This doesn't really solve the problem of Craigslist hoarding all their data to
themselves (as pointed out numerous times already), but there are enough other
sources to make it usable.

------
derwiki
`<http://trycourier.com/?utm_source=anm-blog`> linked in the post seems to
403.

~~~
wlindner
Sorry about that, it's just <http://trycourier.com>, I fixed the link. Thanks!

------
danso
> _Had around 1,500 unique visitors with an average time spent on the site of
> about 7 and a half minutes._

With no disrespect to the OP, that's not a _ton_ of users, even for a single
day...so I'm kind of surprised Craigslist noticed enough to block them. There
must be tons of unapproved scraper scripts checking CL on a daily basis.

~~~
wlindner
That's a very good point. 1,500 unique visitors is tiny. Hell, I will probably
get more visitors to this single blog post. Forgoing everything about CL
blocking me and such, I never did enough to promote the site. I started the
site in 2011 and had little experience with promoting content. As a newb
developer, I was leery of promoting something I created. What if it broke?
What if people didn't like it? When in reality, I learned over time, that most
people frankly don't care. And that's not a bad thing.

The past few years I have been learning over and over: ship it, get feedback,
improve, repeat.

So, getting back to _how_ they blocked me. If I understand correctly,
craigslist blocked any HTTP requests from the Google App Engine User Agent.
HTTP Requests from Google App Engine always append 'AppEngine-Google' to the
end of the User Agent
([http://code.google.com/p/googleappengine/issues/detail?id=34...](http://code.google.com/p/googleappengine/issues/detail?id=342#c12)).

You can recreate this block by doing this in python (you need the amazing
python requests library, so install that): import requests

r = requests.get('<http://craigslist.com>) print r.status_code

This should print 200 and you're good to go.

Then, try it with the 'AppEngine-Google' in the user agent: r =
requests.get('<http://craigslist.com>, headers={'User-Agent': 'AppEngine-
Google'})

and r.status_code will return 404

bingo bango, you've been blocked. I could have just moved off of Google App
Engine, etc. but I would probably be faced with a similar fate as Padmapper if
I ever got any kind of sizable user base.

 __I created this gist:<https://gist.github.com/wlindner/4752122> since the
python code formatting is wonky __

~~~
jzawodn
craigslist is hiring engineers. Interested?

