
Bye Bye Craigslist - ericd
http://blog.padmapper.com/2012/06/22/bye-bye-craigslist/
======
jamiequint
This is just offensive.

It would be one thing if Craigslist was doing their best to satisfy their
customers and they wanted to shut down others who were scraping their site and
leeching off their business. However, Craigslist puts absolutely no effort
into making their product usable. The continued existence of Craigslist is
just a testimony to the enormous strength of network effect lock in.

~~~
binaryorganic
Disagree. Craig Newmark has said that he's scared to make drastic changes to
the UX specifically because he doesn't know what magic recipe keeps people
coming back. Do I like the CL layout? Nope. Does that mean other people should
steal the content and profit from it? Probably not.

~~~
jamiequint
The magic recipe that keeps people coming back is that Craigslist has a large
network.

I would be willing to bet a large amount of money that adding a map to search
results and letting me put an upper limit on the number of bedrooms I'm
looking for would not make people less likely to use Craigslist.

~~~
mtrimpe
Their magic recipe is that an amazing amount of non-technical people have
figured out how to work with the site.

The Dutch version of Craigslist (Marktplaats, owned by eBay) is in a similar
predicament.

They have a very sucky UI and have experimented with new and improved
versions, but every improvement was quickly reverted after large drops in
engagement.

It's almost surreal to look at their site (<http://marktplaats.nl>) and
realize that there's tons of highly talented people behind it.

~~~
bermanoid
Yup, there's a catch-22 here: when your site has shitty usability but is the
only source of information, people hit more pages and spend more time there.
When you fix your UI mistakes, they're in and out quicker because they
actually get what they need efficiently. Because you're optimizing for
engagement by measuring time on site or something like that, you end up making
sure never to make your site easy to use.

This is not a recipe for success. It's a symptom of measuring the wrong
metric, especially when you have a near monopoly on a market due to network
effects. If this is really why Craigslist is not improving their UI, they need
to hire some analysts that actually know WTF they're doing.

~~~
user49598
Haha, I see what you did there. You confused UI with Usability. Craigslist is
incredibly usable. Have you taken a look at Ebay lately? Does it look
familiar? That's cos it hasn't significantly changed in 10 years. Is that
because they don't care? Nope, it's because gradients and drop shadows make
your product look better, but don't make it more usable.

Craigslist is a utility, not a website. They care about keeping it useful not
only for users today, but for users in 10 years. That's not something startups
care about at all. Startups are looking to amass users and then sell.
Craigslist cares not about such goals.

Go ahead, get mad at their inflexibility. But then take a step back and think
about what craigslist represents. And then wonder if you yourself would have
the cajones to stick to your guns in todays startup market. I for one commend
their ability to keep their product strong in the face of all the bells and
whistles of web 2.0 or whatever it's called these days.

~~~
sprokolopolis
Just because the site is simple in design, doesn't mean that it is built to be
used efficiently. Craigslist, as is, is a usability nightmare. When I use the
site, I am using it with various Greasemonkey scripts, Firefox extensions and
3rd party web apps to make their content easier to
consume/see/understand/filter. These tools are the only reason that I can bare
to use Craigslist.

If I am searching for a bike, a simple script to show image thumbnails by
every listing will let me see which listings interest me. This saves me time,
because now I only have to open a few interesting links. I don't have to scour
through many of text links, just to see if they interest me.

The rentals section is constantly bombarded by spam and has very few filtering
options. There is no way to exclude keywords. Boolean operators would greatly
improve the usability of the site.

~~~
prophetjohn
You can exclude keywords and there is a boolean OR operator: |

The problem is that you have to enter some search query before you can use the
negation operator.

Also, protip to get past the spam in rentals section (Austin is holy crap bad
about this): enter 'google' as your search term. This will only return ads
that have an address specified and hence have the 'google maps' links. Plus,
since you've entered a search term, you can filter out the locator who keeps
spamming and screwing up your search by putting his phone number in the
address section.

Here's a query I might have used during my last apartment search.

    
    
        google downtown | central -"512-555-5555" -"CALL NOW" -"austin apartments NOW"
    

etc.

------
readme
I am totally upset. I wanted to use padmapper to search for an apartment for
my next move. I had been looking with it for months and wow it was super
helpful.

Please, make a client-side solution so they can't shut you off! A browser
based plugin, for example that allows us to plot the listings on our end.

I wrote this email to craigslist:

Hi Craig and Jim,

I am very upset that you cut off padmapper. Using padmapper cuts out a lot of
the spam that ends up in the apartment listings due to the fact that it
doesn't have a lat,long pair most of the time. It also makes it much easier to
search for apartments based on your preference. With padmapper, I could see if
the apartment I'm looking at is in a safe area, how far it is from a potential
employer, and a myriad of other great things.

Finding an apartment with craigslist, in spite of the fact that you have the
data, is much harder than searching on padmapper.

I really wish there was a way you could work with them!

You guys are a solid blue chip, no doubt. But if you take this oligarchical
perspective to your business, someone is going to beat you eventually.

Thank you for your continued service

~~~
washedup
Great email... I will follow suit. They cut off the listings right in the
middle of a search and it took me a little while to figure out what happened.

------
ericd
Hi everyone, Eric from PadMapper here, thanks for all the comments!

To everyone saying that this is lifting their content, I disagree - it makes a
summary of the content and then points back to the original. If I wanted to
lift their stuff, I would have made separate pages that laid things out better
than the original. Literally none of the text except the title and location
string make it into the summary.

It's not a copyright issue, their main legal beef is that it's against their
TOU for anything but general search engines to index their content - no
vertical ones allowed. Who knows what their actual motivation is.

~~~
MichaelApproved
Do you think they're shutting you down in preparation of creating their own
version of pad paper an releasing that without your competition?

~~~
ericd
It's possible, they've been looking for a dev with maps experience for a
while.

------
kylelibra
For those wishing to take action:

jim@craigslist.org – Jim Buckmaster, CEO

craig@craigslist.org – Craig Newmark, Founder

@jimbuckmaster on twitter - <http://twitter.com/jimbuckmaster>

@craignewmark on twitter - <http://twitter.com/craignewmark>

Simple email to copy and paste:

Jim and Craig,

I'm a longtime user of Craigslist, but I'm really disappointed in your
decision regarding PadMapper. It would be great if you could weigh in on the
debate happening on Hacker News or elsewhere.

Thanks,

~~~
phaus
Good luck, but I doubt it will work.

My first experience with craigslist was trying to figure out why they had the
capability to block my wife's ad for a free dog that contained a mispelled
word while the site's creator was busy trying to tell the government that
there was nothing he could do about the many child prostitution ads that
appeared on craigslist.

~~~
jsdalton
Craigslist does not allow you to sell or give away animals -- it likely had
nothing to do with a misspelling.

~~~
phaus
The email that was sent to me said that it was because of a misspelled word.

------
bcrescimanno
I hope this doesn't get down-voted for the confusing title (I know it's the
title of the Blog post itself; but out of context it's a bit misleading).

When I moved from Atlanta to Boston, PadMapper was invaluable in finding a
place to live. Given that 1) we didn't know the area, and 2) most rentals in
Boston and Cambridge are rented by individuals rather than part of larger
complexes, it was great to be able to have the listings from Craigslist on a
map.

Ditto when I moved from Boston to San Jose. My wife and I were only interested
in a house for rent and the vast majority of those could be found on
Craigslist. Again, we didn't know the area so having a map was beyond
valuable.

I sincerely hope Craigslist does a 180 on this one; PadMapper is a great tool
that deserves access to that data.

~~~
sliverstorm
_PadMapper is a great tool that deserves access to that data._

Why do they deserve access?

~~~
bcrescimanno
"deserve" is probably the wrong word (forgive me for hastily typing out my
comment before running off to a meeting).

My point was really about PadMapper acting in good faith with Craigslist; they
do send traffic to Craigslist for the full data on the properties and such.
They're a good company doing good things--and it's sad that Craigslist's
refusal to innovate has now bled over into "no one else can innovate for us"

~~~
sliverstorm
Ok, no problem. I've just made a habit of calling people out when they use
"deserve" a little too frivolously.

------
akoumjian
First it was Carsabi and now Padmapper. While I'm not surprised, as the
Craigslist ToS explicitly prohibits crawling their site, it is incredibly
frustrating that the website with all the data is unwilling to innovate and
unwilling to help others innovate.

Craigslist will continue to be a mediocre, "good enough" solution. Since
everyone associates online classifieds with craigslist, none of the other
classifieds sites seem to have a chance. It's not even as if services like
carsabi or padmapper are competing in any way with them.

~~~
zach
Craig Newmark has often said that they innovate in response to their
community. That would seem to have three effects:

\- Incrementalism: faster horses instead of cars

\- Suburbanism: those in the community want it to stay familiar

\- Majoritarianism: outsiders, potential users, occasional users, mavericks
and dreamers have little impact

------
robmaceachern
Wow this is really surprising, especially after reading Craig Newmark's answer
on Quora just the other day to the question "Why hasn't anyone built any
products on top of Craigslist data?". His response:

"Actually, we take issue with only services which consume a lot of bandwidth,
it's that simple."

[http://www.quora.com/Why-hasnt-anyone-built-any-products-
on-...](http://www.quora.com/Why-hasnt-anyone-built-any-products-on-top-of-
Craigslist-data)

~~~
jaredsohn
To clarify for everyone, while the parent read the answer a few days ago,
Craig wrote this response in 2010.

~~~
jongraehl
Good point. What didn't seem a threat then, must now. You should expect some
changes in strategy over >1yr.

------
makmanalp
I've used padmapper twice to get an apartment so far, and It's great! I've
loved the fact that it's constantly improving and has _exactly_ the features I
need: Walkscore, bookmarking, map, proper filters. It's strictly better. To
the author: I would have paid for this.

It sucks that this waas mainly an interface to craigslist, and now they're
gone. I hope this gets enough critical mass that eventually so that the
unusable but unexplicably popular horror that is craigslist apartments just
dies an ugly death. Maybe you can even sell it to them, who knows?

All I know is while CL's response is perfectly rational, I can't help but be
pissed off that I'm locked into a shitty product that is only surviving
because of head start, critical mass, network effects, etc.

Edit: An alternate approach (that would admittedly not help Eric) is to open
source padmapper and let people do local installs for themselves. Would be
pretty hard to ban that. Although I'm not sure on the legality of scraping.

~~~
iamwil
Just a clarification, padmapper will continue to exist with listings from a
lot of other places. It will just no longer have Craigslist listings, unless
you help send an email to Craig and Jim.

------
kenrikm
Craigslist is in major need of disruption.

Actually apartment finding in general sucks I can't think of a single goto
place to look for apartments other then Craigslist and that just proves point
#1!

~~~
ghshephard
I guess a lot depends on where you live - Craigslist in the Bay area is like
magic for finding apartments, or a room to rent.

The Bay Area has a very advanced craigslist option, where you can sort not
only by geographic area (Peninsula), you can even zoom down to a particular
city. Add filtering by price and find a place to live in the Bay area has
always (Since 2003) been painless for me - particularly as I don't drive, and
so getting a place that is _precisely_ in the right place is important.
Housingmaps provides a free map interface for those who find such a thing
useful.

I don't actually know how anyone would _improve_ craigslist (for me) - I am
not sure what the disruption would be - as the free service provides me 100%
of what I would want from such a system.

~~~
juiceandjuice
Yeah but that _still_ sucks, because people rarely know which neighborhood
they are in. With padmapper, I could see exactly what was where and not have
to worry about somebody thinking excelsior is somehow the mission, and the
price per room/cats and dogs/walk score/etc... extras are amazing and
integrated, especially if you don't know an area.

I guess you haven't really used padmapper if you don't understand how anyone
could improve craigslist for you.

~~~
eshvk
>> Yeah but that still sucks, because people rarely know which neighborhood
they are in.

"rarely" is a tough word. I have found the city to be pretty well demarcated
unless you are talking about people confused about minor issues such as what
counts as Lower Haight and what counts as Upper Haight.

>> I guess you haven't really used padmapper if you don't understand how
anyone could improve craigslist for you.

I used Padmapper as a newbie to the city but quickly found that the latency
between postings on CL and the corresponding crawled updates on Padmapper were
so high that the postings would have already been hit by a few dozen responses
by the time I got it and in a hyper-competitive housing market like SF, this
made all the difference.

~~~
rdl
If you're looking on the Peninsula, where school districts and crime levels
vary by street within the same time, Padmapper is vastly better than
craigslist.

I mean, look at all the misleading fucking tools who post "Palo Alto apartment
for $1k/mo" which is actually in EAST PALO ALTO. With padmapper, it is clear.
(In Menlo Park, where some areas are unincorporated, some are Menlo Park, some
are Menlo Park but a warzone (Belle Haven, aka EPA North), etc., it's even
more essential).

------
GigabyteCoin
I expected padmapper to be shut down a long time ago. Mainly because they are
an incredible threat to the websites they are scraping all of their data from.

That's probably why padmapper has no way of making money at the moment.

They were hoping they could slink by, stealing other's content, until they
were eventually big enough for people to warrant visiting them to list their
apartments for rent. And then sell ads.

Don't get me wrong, I love padmapper, but scraping content is scraping
content. And it's generally not allowed.

~~~
ericd
You're right, I was hoping I could go on, but I wasn't stealing anyone's
content, I was posting links pasted geographically located with stat
summaries. Look at a listing on PadMapper - there's no listing text, and when
you click through, it goes to the original listing.

~~~
GigabyteCoin
My apologies. I may have used some improper wording there.

But if I may... you must admit that you have taken "something" from
Craigslist.

Whatever you want to call it... be it geographical location points, links-to-
listings, content, data, images, etc... it is inevitably some form of
information that was originally curated by Craigslist through their 10+ years
of existence online and is thus available to be used (and not used) at their
discretion.

Canada Post for example charges $5,500/year
([http://www.canadapost.ca/cpo/mc/business/productsservices/ma...](http://www.canadapost.ca/cpo/mc/business/productsservices/mailing/pcdp.jsf))
just to have access to a postal code verification system. It's all publicly
available information... if you walked to every city, suburb, nook and cranny
in Canada and asked the local people what their postal code is, they would
tell you without hesitating.

But if Canada Post gets wind that you may have copied any of that publicly
available information from their databases instead of doing the hard work
yourself, you can expect a lawsuit at your door fairly quickly:
[http://www.cbc.ca/news/business/story/2012/04/13/technology-...](http://www.cbc.ca/news/business/story/2012/04/13/technology-
canada-post-postal-code-copyright.html)

I don't think Canada Post had a case in the above example because the company
in question didn't actually copy anything from their database, but you did
when scraping Craigslist (for however insignificant you found the information
to be), see what I'm saying?

Just because you are sending them traffic, doesn't give you the right to take
(insert whatever might have been taken here) from them without their
permission.

That being said, I love padmapper and have used it numerous times already for
myself and friends. I wish you all the best going into the future!

~~~
ericd
Thanks for the explanation! I've talked with lawyers about this in the past
week, and facts, totally rewritten summaries of fact, and collections of facts
aren't copyrightable in the US, so it's not really an issue of their content,
as far as I understand. They do have a right not to be crawled by me, though,
especially if it causes a burden on them. I wrote that crawler specifically to
be very light on their servers including making tradeoffs to usability (it
never recrawls listings), but that's still their decision to make. If there
are ways to get the data that don't involve touching their servers, then the
TOU no longer applies, if I understand the law, but it would obviously make
them angry, and so I'm most likely not going to do that.

~~~
nabraham
I agree that this isn't a copyright issue. If you're operating in California,
the 9th Circuit is probably your best bet for holding that your scraping of
their data doesn't damage or impair their computer system. But given your
other comments, it is unlikely you want to raise the funds or spend the time
to battle it out in court.

But there is an another way - let us (all of us) get the Craigslist data for
you. You could create a browser plugin that would save craigslist posts a user
views, and then upload only those posts to a central database that feeds
padmapper. With just 1,000 users you would get most but not all the data you
currently are scraping.

This is the same approach used by Recap (<https://www.recapthelaw.org/>) as a
response to the federal government's paywall to access public court filings.

~~~
ericd
Thanks for the advice! But yeah, court sounds like a pretty big clusterfuck
with a potentially bigger clusterfuck at the end of the rainbow. I have a
feeling that the plugin would get me sued as well, though I suppose that would
be a more winnable battle.

------
AmericanOP
Craigslist should buy you, not block you.

I'm basically a Luddite and yours is the one service I always recommend.

~~~
ceworthington
Maybe they should block you then buy you. Probably would help them in the
negotiation...

But in all seriousness, PadMapper was a great service and seemed to do it the
"right" way (providing deep links back into Craigslist's site/content), if
there is a "right way" to unofficially gather data without use of a public
API.

------
lindablus
PadMapper is like the super helpful feature that craigslist did not figure out
how to build correctly, or never bothered to build. I'm disappointed that
Craigslist chose to respond by making a great social utility disappear from
the market and anger the users. What about charging a license fee? What about
acquiring PadMapper? What exactly does Craigslist gain by pulling its index
off from PadMapper other than damaging its own public image?

------
jacoblyles
Craigslist is an evil monopolist that literally wastes millions of hours of
its users time every year through the enforcement of shitty UI (if the amount
of money that I save using padmapper is typical). They get a good reputation
for some bizarre reason because their CEO is an anti-capitalist, but they are
still evil for holding back innovation.

A human lifetime is about 700,000 hours. Craigslist is the moral equivalent of
a serial killer.

~~~
DustinCalim
For those who use Chrome web browser, there's some fixes by way of extension
that really help save time on CL... My favorite:
<http://www.craigslistextension.com/>

------
Mc_Big_G
This is the whole reason I started foreverlist.com. Besides better/more
photos, geolocation, comments, etc..., I always wanted to open up the api and
let anyone develop on top of it. Alas, competing with craigslist is folly. I
regret cloning the design of CL, but the logic was that it would be simple for
people to use.

I absolutely LOVE pad mapper and hate craigslist even more now. Many good
sites have been killed off by craigslist.

[edit spelling]

------
juiceandjuice
So should they ban Google? Because I use google to search for things on
craigslist with much more success than craigslist's system, in the same way I
used Padmapper to find a room/house (3 times in the past year... I move a lot)

This is stupid.

------
makeramen
Carsabi (YC W12) was also affected by this recently. Especially annoying
midway through my car search. It is really disappointing that Craiglist won't
license their data.

~~~
romansanchez
I would say this pretty much killed Carsabi.

~~~
pg
Actually not. Enough of their content is direct from dealers that it looks
like they will be able to bounce back.

~~~
makeramen
Yeah, they still have quite a lot of content, and to be honest craigslist
listings were some of the worst on there. Still, it was handy to just have
_one_ go to place for car shopping.

------
fleitz
Kudos to CL for standing up for user privacy. When I use CL it's with
consideration of their TOS which bans these practices.

When padmapper copies the data they violate my privacy, my copyright, and the
agreement between myself and craigslist as outlined in the TOS. What padmapper
is doing is a violation of many data privacy laws like PIPEDA, EUDPD, and
various copyright acts.

If padmapper wants the data why don't they just obtain consent from the CL
posters, instead of copying it without consent.

Would the padmapper team be ok with me deciding to stick their logo where ever
I deem it necessary? I bet they'd probably sue me for copyright and trademark
violation.

~~~
robrenaud
This isn't making a permanent record of casual encounters.

It's putting apartment listings on a map. People listing apartments tend to
want it to be widely known.

~~~
dr42
People listing apartments tend to want it to be widely "known."

those same people are already getting their listing widely known, by posting
it on CL.

------
AceJohnny2
Here's the email I sent to Craigslist. I tried to use the following template:
intro (what are we talking about), how padmapper is good for craigslist, why I
care about padmapper, course of action.

Feel free to reuse it.

\----

Hi Jim and Craig,

I learned today that Craigslist has sent a Cease and Desist to padmapper.com.
Padmapper provides a service to visualize on a map rental listings from
sources like Craigslist.

Real estate is all about location, and let's be frank, Craigslist's location-
based search sucks. Padmapper helps the searcher easily see where real estate
is located. It turns a grueling search into a breeze. It adds great value to
Craigslist's listings without competing with Craigslist's business.

Padmapper, along with Craigslist, were essential in helping me find my current
home in the San Francisco Bay Area. Without Padmapper, I wouldn't have used
Craigslist in my search.

I'm sure you can find a mutually beneficial deal with Padmapper, instead of
shutting them out entirely.

------
ndubya
Just out of curiosity, why would they not want you to use their classifieds?
Unless I am missing something, they don't collect revenue for listings. I
thought Craigslist was all about spreading good will. If Pad Mapper provides a
more efficient way of spreading that good will, then why not?

As a user of PadMapper, I am very disappointed to see this happen. Craigslist
is almost unusable if you don't know the area you are looking to move to.

~~~
dredmorbius
As best I can tell, it's a matter of corporate philosophy. Craigslist is
predicated on local, person-to-person interactions. It has long taken pains to
curtail uses of the system which allow aggregation of multiple CL local sites
(there is, for example, no convenient way to search all of California for
listings), or to include CL listings in third-party services.

As a long-time user of Craigslist, and a recent fan of padmapper, I'm
disappointed, to say the least.

I do suspect it's time for some disruption in CL's space.

------
dap
I'm surprised how many people here are defending Craigslist. They've got a
history of both providing a terrible experience for apartment seekers and
shutting down any attempts to use their data to provide a better experience.
That would be okay in a competitive market, but they've got a strong monopoly
in a market with a high barrier to entry. They're playing the textbook role of
an evil monopolist.

~~~
chrisguitarguy
Craigslist is a company whose terms of service were violated by PadMapper.
Whether or not PadMapper provides a better experience is irrelevant:
Craigslist has a right to defend its content and lists by whatever means it
sees fit. Any businessperson on this site would expect to have the same
rights.

Craigslist is not in the business of helping anyone but Craigslist be
successful. It's not monopoly; it's business.

~~~
dap
Companies are not free to do whatever they want. The government compels
monopolists to avoid anticompetitive behavior, and even breaks companies up,
for examples. The market in this case is a little hard to define, since
apartment seekers aren't purchasing the product, so I'm sure it's not actually
illegal, but it sure seems to violate the spirit of these laws, which are
designed to protect consumers from exactly this kind of behavior.

------
martingordon
This is horrible. I used Padmapper extensively when looking for an apartment
here in New York two months ago. I guess I'm never moving again...

------
wave
Padmapper has been a wonderful service. It is confusing why Craigslist is
blocking it unless they (Craigslist) are thinking about coming up with similar
user interface soon. Craigslist has been hiring UI designers.

------
rdl
Why not buy a license and go hard on mobile? Then use that to get more direct
listings, ideally with ACTUAL STRUCTURED DATA unlike the shit that is
craigslist (i.e. closer to MLS, or even better, closer to the proprietary
systems used inside large real estate investment companies), and then re-
release a web version using only those listings.

Maybe focus on SFBA only, since the real estate situation is acutely broken
here AND smart+rich people still rent here (unlike Texas or the midwest, with
NYC being its own weird market).

~~~
ericd
Maybe, if you want to chat more about this, email me at eric@padmapper.com

------
blackysky
At the end of the day it is Craigslist's data. I know padmapper was useful for
a lot of people so the next smart move is to build your own data. A
"craiglist" focus on rental only with the same features. The sad reality is
you need your own platform to scale any type of services like that.Is it going
to be easy and fast to build. The answer is no. However the reward and
opportunity are huge....

Of course it would be great if craigslist could have an API or licence his
data but it won't happen in a bear future ...

~~~
icebraining
_At the end of the day it is Craigslist's data._

Nope. It's the users' data. Craigslist only has a license to use it.

~~~
jbigelow76
In that case Craigslist is doing right by its users, the licencor of the data,
since the licensor has not granted a license to padmapper to use said data.

------
angryasian
I think its pretty well documented that craigslist is known to shut down
services that use craigslist data. I really don't think should come as a
surprise to the creator.

------
japiccolo
Unfortunately, scraping another site's information is generally not a long-
term viable business plan (speaking from personal experience). I suppose
CraigsList, much like insurance companies, doesn't like its valuable
information being displayed on other websites without endorsement or consent.

That said, I love PadMapper, having used it a couple of times for finding
apartments in the Bay Area. So it's a bummer that they won't be able to show
CL listings anymore.

------
sh1mmer
Good luck, I think it's a sign of your success that this has happened.
Providing more value with their data than they do is a threat to their
business.

~~~
juan_juarez
I can't see Craigslist ever stepping up their game and providing meaningful
visualization. I can't really see why CL cares about where people are reading
the listings - their business model is based on selling the listings, not
monetizing the viewing of them. I suspect it's out-of-control lawyers at work.

~~~
jessriedel
Craigslist is closer to a non-profit than a business. You can't model them as
a self-interested agent.

~~~
potatolicious
And like many other non-profits, the quality of the service shows - the
service is merely "good enough" (in many categories, far less than good
enough), with no competitive pressure to improve.

We have already seen the meteoric rise of many niche, category-specific
Craigslist competitors eclipse CL in scale and influence, particularly in real
estate. I suspect due to Craiglist's general crappy experience this
fragmentation will continue.

And I for one think it's a good thing.

~~~
jessriedel
Well, the lack of competitive pressure stems more from network effects than
being non-profit per se.

Because of this, I don't see the fragmentation as a good thing. One of two
things will happen in each craigslist market (rentals, used goods,
ridesharing, etc.): either a single for-profit company will ride networks
effects and establish a monopoly, or there will be severe fragmentation.

If the former, then you can expect some features to improve (in order to steal
the market from craigslist in the first place) but many other things to
languish due to a lack of competitive pressure. (Also, if it's a for-profit
company, you'll pay money.) This is ebay.

If the latter, then it will no longer be possible to go so one site to search
in that market, greatly decreasing consumer utility. This is how classified
ads worked before craigslist.

Much better is a well-run, benevolent non-profit. Wikipedia, though not
flawless, is pretty much the epitome of this idea. It's scary to think what it
would be like if Wikipedia's data was spread over dozens of websites of
varying quality and motivation. (Then again, in the scenario no one would
bothered to write most of the articles anyway.)

Craigslist doesn't fit this exactly because they impose a philosophy
("localism"?) which most people don't share and which decreases usability.

~~~
potatolicious
There are, for example, real estate websites that are eating the CL for-rent
category's lunch right now. Craigslist isn't doing much in response to these
competitors, despite the fact that in some markets (particularly SF and NYC
where I have experience) Craigslist is now very firmly in second place (or
worse).

I've been apartment hunting in NYC lately, and it's mind-boggling what
proportion of Craigslist posts are scams compared to other sites like
StreetEasy or NYBits.

IMO if Craigslist was run more like a scrappy startup looking for their big
exit, they'd have more than responded to these competitors by now. The network
effect exists, but IMO is an insufficient explanation of their lack of action.

I agree that the fragmentation is, all things being equal, a bad thing for
users - but considering how utterly useless Craigslist's moderation systems
are (many categories are full of duplicate posts, spammers, and scammers), and
how poor the UX is otherwise (search? hah!), I welcome it. Fragmented sites
like PadMapper, StreetEasy, etc, have done _way_ more to ease the life of
renters than Craigslist in the last decade.

------
mullingitover
This is a damn shame. I used padmapper to find my last two places. CL
should've been doing this feature on their own, navigating rental listings on
their site is a nightmare. The map feature was a godsend.

------
cwilson
Curious if Live Lovely (<http://livelovely.com/>) will get a similar cease and
desist. They do essentially the exact same thing, but with a nicer design.

~~~
JaggedJax
Except Padmapper works everywhere craigslist does. I went to livelovely just
now and they don't have my city. It's not exactly a small town. Also their
filtering is nothing compared to padmapper.

On a more related note, I would imagine any sites like this that come up on
Craigslist's radar will get a similar notice.

------
cwp
Disappointing. I'm currently living in an apartment I found via Padmapper and
Craigslist. Padmapper was absolutely crucial, without it I wouldn't have found
this place.

I understand why they're complying with the C+D, but I wish they'd fight it.
Craigslist is better than printed classifieds, but that's about the best thing
you can say about it.

~~~
iamwil
I think it's harder for small guys like Padmapper to fight them, because 1)
There's no money for lawyering 2) Craigslist says that they listen more to
users for UI.

So if you'd like to see CL give Padmapper license to use its data, send them
an email. Your voice as a CL user weight more than Padmapper's would.

------
cldrope
Craigslist does not allow scraping. Beg for help and for support for breaking
their rules elsewhere. This is Hacker News, not "help me retain a business
model after I broke rules" central. You could alternatively hire people to
spam them with nice requests to reconsider.

------
jluxenberg
Just sent a mail to {craig,jim}@craigslist.org. Not sure if it will help but
if they hear enough noise about it, maybe they'll change their stance.

~~~
coreygoodie
It's most likely an indication that they'll be developing their own UI.

------
JumpCrisscross
Could Padmapper reform itself as an extremely intrusive browser extension?

Is there any evidence CL was pressured by any privacy groups or regulators?

------
mutant
The pro CL comments here are ridiculous and so anti-innovation. "Don't crawl
my data bro!". PM wasn't modifying the data in any way, just redis playing it
on a damn map. Now CL is useless, yet again, for housing.

------
kthakar3
I wonder why CL doesn't create an API and charge for it's usage. The data is
clearly valuable and they own it, but that doesn't mean that it can't use
multiple views.

~~~
runako
This is only useful if the teams scraping CL have budget to pay a market rate
for the data. Considering that CL seems to have a lock on this data for many
markets, I'm guessing that's worth a lot of money (which startups will not be
able to afford).

~~~
product50
If they cannot afford, then they should not be there. Scraping illegally is
not a long term solution. Craigslist should conduct some market research from
their end to charge a "reasonable" fee which fills in their coffers as well as
allow development an ecosystem around their data. Even DuckDuckGo pays for
Yahoo! BOSS Search API to show their results. Not sure why CL hasn't thought
about it.

------
smalter
I'm pretty sure that Craigslist disruption is coming, and it's going to be
AirBnB.

I'm not too worried. What craigslist is doing (ie, sucking) isn't sustainable.

~~~
pbreit
Hah, people have been saying that for a decade. What people fail to realize is
that Craigslist is ultra successful _because_ of what it does, not _despite_.

------
coreygoodie
Why the hate on Craigslist? It's their information, and are free to do what
they want with it - most likely, develop their own platform.

------
mikemoka
Quite a misleading title..I think many clicked thinking to read about an
unexpected shutdown of Craigslist actually

~~~
ericd
Sorry, I didn't mean it that way - if you know what PadMapper is and see the
URL, I think it makes sense.

~~~
iamdave
With the OP here, I saw the thread title and silently went "thank god" to
myself immediately knowing what this meant, having used PadMapper _very_
extensively in the last year after moving to Austin. I knew it meant a good
portion of ads being gone, but I was going to get much more _qualitative_
results in finding a condo.

I really hope PadMapper can make up the volume but continue to have good
results for people searching. It's a wonderful resource.

~~~
ericd
Haha you could always filter out Craigslist with the sources filter (though
it's a bit hidden)

------
pbreit
I propose that Craigslist has been ultra successful _because_ of its actions,
not _despite_. I think it's success is due both directly to how it has built
and operated the site as well as due to the overall mentality that fends off
all the lame ideas from the armchair product designers.

------
Alex3917
"They allow mobile apps to display their listings if you buy a license from
them, but not websites."

Really? They used to allow any website to use their data for free as long as
they didn't run ads against it or monetize it in any way. Kexter.com has been
doing exactly what you're doing for almost ten years now.

------
bryanjclark
Padmapper's a brilliant solution. I'd like to think that Craigslist is
building their own great solution, but knowing how slowly CL moves on
everything, I sincerely doubt they'd build a worthwhile replacement. Bummer!

Padmapper's helped me and a few friends find our apartments in SF.

------
yalogin
Could it not mean that they are coming up with some padmapper like
functionality? Padmapper of course is the only usable rental search site after
craigslist. The rest of them like rent.com and apartments.com are absolute
crap. I hope they work something out.

------
aninteger
How is this different from when Bing was using Google search results? If
people want to use Padmapper great. if people want to use Craigslist then
great.. Am I missing something here? I don't see the problem. What if it were
the other way around?

------
smattiso
This sucks. Craigslist is not setup to handle apartment searches at all. I
literally just finished finding an apartment and I solely used padmapper. I
can't imagine having to go through the process just using craiglist search.
Bummer.

------
omarish
Cached version:
[http://webcache.googleusercontent.com/search?q=cache:http://...](http://webcache.googleusercontent.com/search?q=cache:http://blog.padmapper.com/2012/06/22/bye-
bye-craigslist/)

------
eli
I personally would have tried to find a way to work with them. But it's not my
site.

Craig isn't doing anything wrong or unethical here. Quite the opposite;
Craigslist continues to deliver _exactly_ what its users signed up for.

------
glenntzke
I'm sad to hear this. I used padmapper to find my place in NYC among an
incredibly scam-filled and otherwise difficult to navigate sea of lies and
frauds. PM helped me utilize CL in a way their filters could not.

------
BadassFractal
I used PadMapper with great success numerous times, I hope it can survive
this.

------
phear
Once read about how its a bad idea to base the core of your business around
someone else's platform. Once they(and their lawyers) decide to re-evaluate
your relationship with them it can nearly destroy what you've built. Testament
to this are the number of apps based on twitter that died when twitter decided
to up their offering and cover what they were doing. How to do it well is
Zynga's relationship with Facebook

The people at PadMapper are doing a good thing looking for other sources but
like the author said Craiglist was/is an important source of pad listings.

All the best to them

------
citricsquid
I decided to consider moving to the US on a whim about 6 months back (in the
end I didn't) and decided to try and find an apartment and guage the cost of
living. People recommended craigslist as apparently the US doesn't have a
rightmove (<http://www.rightmove.co.uk/>) type site. Craigslist was probably
the worst experience I've had trying to find an apartment... I can't believe
that people actually use it, why wouldn't they build a padmapper esque system
(or acquire padmapper)?

~~~
dclowd9901
Craigslist is, unfortunately, the "best" example of the network problem. Due
to its popularity, it is entrenched.

That said, there are methods of disruption. AirBnB bootstrapped off Craigslist
to get their initial dataset, and have since moved to their own content.
That's probably the strongest approach if you're lucky enough to be able to
move quickly when you need to.

~~~
scarmig
The thing is, I think Craigslist's network effect is exaggerated. Its network
is very clustered because nearly all of it (particularly housing) is focused
on people in a particular area. Compare to Facebook, which has a large degree
of inter-connectedness between all clusters.

If there were a site that had, say, all the rentals available in the SF Bay
Area, it would be highly useful to you, as it would have gobbled up most of
the benefits of the network effect available (of course, outsiders moving in
wouldn't automatically know its name like they do for Craigslist, but that's
relatively minor).

Once you've reduced your problem to "develop a really strong network in a very
localized geographic area," it seems to me like the solution would become in
reach, with a lot of hard work. You could even scrape pretty heavily early on,
or enter it by hand if somehow that got around the ToS.

Maybe start at Berkeley or similar college town, maybe reducing yourself to
using physical paper and highly targeted online advertising, and rely on
students heavily to get the word out. Make it ridiculously easy for someone to
post an ad on your site and have it automatically propagate to CL. Gradually
expand to different parts of the Bay Area, and then pick out other localities
to target. NYC, maybe, especially if you can somehow make its real estate
market less dysfunctional.

------
MitziMoto
Craigslist needs to be stopped. This is the third incredibly useful service
I've seen shutdown in as many weeks. I was heartbroken when Carsabi had to
stop indexing Craigslist a couple weeks ago.

------
khangtoh
Could the C&D be PadMapper violating the use of the trade mark?

<http://www.padlister.com/listings/new>

"Better looking, effortless Craigslist Ads."

------
staunch
Was there some attempt made at monetizing Padmapper? IIRC Craigslist isn't
super hard core about not-for-profit projects using their data, but takes a
hard line on commercial use.

------
maked00
Let me guess your young and your pathetic education taught you very little,
and your sheltered life made you think everything would be handed you on a
silver platter.

Instead of whining, you either need to offer large sums of cash to CL, or some
deal. CL does not owe you a living. Your crappy little app is just skimming
the cream off the top of CL's infrastructure and you expect CL to give you a
free ride? Not going to happen. You are simply lucky you got away with it as
long as you did.

------
jbarham
FWIW Craigslist is a ghost town in Australia and virtually useless for buying
& selling the type of stuff I used it for frequently when I lived in
California. For example here's the furniture for sale section for Melbourne:
<http://melbourne.craigslist.com.au/fua/>. Seems that in Australia people use
Gumtree.com.au or even ebay.com.au for online classifieds.

~~~
beedogs
gumtree is where it's at for apartment rentals and other big hard-to-ship odds
and ends. then there's carsales.com.au for automobiles, and ebay for
everything else. seems to work out okay.

------
sayemm
Really sorry to hear this. I've successfully found 3 different apartments in
perfect locations through PadMapper (and one that I'm moving into in 2 weeks
actually). And I've had several friends use it too.

Great job with all the time you've put into the site, Eric - hope the service
pulls through this somehow, but also looking fwd to seeing what you work on
next if you decide to move on.

------
philmcc
Solution/Question:

Could the padmapper guy just make an application, which he sells for $10-$15,
with the same functionality, except all the "scraping" happens from the user's
computer or browser?

And if this violates the TOS, assuming I can sell a browser which only visits
Craigslist -- then where is the line where a browser does too much
"formatting" and becomes a TOS violation?

------
jblock
Sometimes people seem to forget that the most frequent users of Craigslist,
and the ones that keep it popular, are not complaining about it having bad UX.
We are.

I want PadMapper to succeed (it helped me find a place super-quick for grad
school in the fall), but Craigslist has every right to legislate their
content. We'll see what they do/don't do about it.

------
product50
If I were at Craigslist, I would strongly look at commercializing their
listings. By that I mean charging a fixed amount (say $5) for every 1000
results sent to the requester (such as Padmapper). That ways Craigslist gets
to keep the revenue while other companies can innovate on top of the massive
data which Craigslist has.

------
raldi
This post could use a new title, unless you want to scare HN'ers into thinking
Craigslist is shutting down.

------
enobrev
CL is obviously in their full rights to do so, but I really wish they could
have waited a couple weeks before they did it. I'm moving to another state
this Tuesday and PM is especially helpful when you don't know the
neighborhoods.

------
kylek
as someone typing this in the apartment they found via a Craigslist ad on
PadMapper (and as someone who has recommended it to several people because of
the feature). time for a mobile PadMapper with a proper license for the data?
:/

------
nickfromseattle
Craigslist is hiring Senior UI / Usability / Front End Engineers - maybe they
plan on releasing a similar feature?

<http://sfbay.craigslist.org/sfc/sof/2841395914.html>

------
Avitas
I would make the same argument that many are making with Cragislist about IRC,
the use of multi-digit telephone numbers and a huge variety of other
entrenched technologies that work reliably for enormous numbers of people.

The entrenched technologies are clearly excellent in the context of what they
do. Can additions be made which allow one to comprehend data faster, summarize
and/or visualize groups of data, filter out garbage and/or present things in a
simpler way? Of course.

The agreement-violating leeching activity of some can be easily be forgiven
because is not happening to your data, your servers, your love and your life's
mission.

In my mind, remoras like this are exposing themselves to additional risk by
doing something that is in violation of TOU and potentially a violation of
various laws.

There are so many fitting cliches here that it's making my head spin.

~~~
MichaelGG
Or maybe those that are "forgiving" find that this "leeching" is creating a
positive benefit for society as a whole, and that ultimately, IP laws are
there to benefit the public.

------
mhartl
Craigslist is one of the greatest blown opportunities in the history of
business.

------
marquis
Could Craiglist shut down every single IP who scrapes? Could you build
Padmapper to be less 'realtime' and more 'tell me the city and I'll set up a
client-side scraper, come back in 15 minutes?'

------
jarnix
I found this : [http://m.techcrunch.com/2012/06/22/padmapper-craigslist-
ceas...](http://m.techcrunch.com/2012/06/22/padmapper-craigslist-cease-and-
desist/)

So no it was not bye bye.

------
nickgeiger
>> They allow mobile apps to display their listings if you buy a license from
them, but not websites.

PadMapper already has an app so couldn't they continue to use the listings in
the app?

------
CoffeeDregs
Bummer. I'm completely mystified that CraigsList doesn't have an API.
Fortunately, others do: <http://3taps.com/> [no affiliation]

------
felixchan
Can you open source padmapper so we can run it ourselves? That way, no matter
what Craigslist does, people would still be able to use the great service.

~~~
ericd
Hm, that's kind of an interesting idea. I'd be pretty embarrassed at all the
hacks I've added over the years, but I'll definitely keep that in mind.

~~~
ryanmerket
Yes. Please open source it...

------
mehulkar
I don't think I would have ever found a place to stay in SF after moving here
4 months ago if it wasn't for CL's data and PM's ease of use.

------
philwelch
This is an existential threat to Padmapper. Which is very sad to me, since
it's easily one of my favorite sites to search for anything.

------
whichdan
This really sucks. Finding an apartment in Boston on Craigslist is nearly
impossible without PadMapper.

------
joshwa
Here's an idea-- why doesn't Craigslist just buy padmapper? They certainly
have enough cash to do so.

------
folktheory
Nooooo!

I was going to start looking at places in Padmapper next month, for my
upcoming move!

Is there any other similar service?

------
eibrahim
How about all the ipad apps that access craigslist data? how do they get away
with that???

------
swayfm
Counting down for a new wave of PadMapper clones / CL scrapers to start
popping up.

------
limanoit
oh sad to hear this. padmapper service with craigslist really helped me!

------
maked00
Just read the entire thread, got to be one of the most sock-puppet, straw-
manned, astro-turfed, meat-puppeted mess of self serving wanna be entreprenuer
whining ever.

CL is successful, get over it.

~~~
jps359
You're absolutely correct.

------
malandrew
Can you please open source all the code specific to craigslist data extraction
so that others may incorporate your work into open-source projects?

------
mwhite
Take them to court. I will donate to your legal fund.

~~~
icebraining
On what grounds, exactly? They have every right to deny you access to theirs
servers.

~~~
mwhite
They have every right to implement technical barriers to scraping. Whether
they can invoke the law to establish a legal barrier to re-publishing their
proprietary-but-public facts -- not even the apartment summary text, just
extracted metadata -- is not at all clear. When an auction aggregator was sued
by eBay for doing something similar, the judgment in eBay's favor rested on
the fact that the aggregator was posing a real undue technical burden, not the
inherent fact that they were scraping.[1] Since I assume Padmapper does
intelligent scraping, they might have a fairly good case.

[1]: <http://en.wikipedia.org/wiki/EBay_v._Bidder%27s_Edge>

~~~
DannyBee
I think you may be confusing two issues.

It's almost certainly true that they could republish the _data_ without
violating any law. That has nothing to do with whether they have the right to
scrape the data from the website.

IE do not confuse their right to republish the data with the right to get it
from a particular source in a particular way.

The line of cases you are citing is also about trespass to chattels, which is
a pretty bad claim to make about website scraping (for the reasons the
california supreme court set out in Intel vs Hamidi). It was mainly used back
in the days when courts barely knew what websites were. Trespass to chattels
may still get used when you can show actual harm caused by the scraping, but
most claims nowadays would be around terms of use violations, which are a much
more grey area (some courts have gone for it, some not).

------
wilfra
Hi Craig and Jim,

I was disappointed to learn you dont want PadMapper to use your listings.
Their site and Carsabi are two sites I recently used to find my apartment and
car, respectively, only after I had given up on dealing with your firehose and
sparse search options. In using both sites I knew full well I was simply using
them as a better, more elegant, more feature rich version of CL. Both
ultimately referred me right back to your site for specific listings, which
was exactly what I wanted them to do.

I dig and respect that you guys are staying true to your mission, running a
lean company and keeping it simple. I've read all I can find about both of you
and how you've built and how you run CL. You are two of my heroes. But you are
really betraying your mission and your credibility with this move. You should
let your data be completely open. Let people do whatever they want with it. CL
is an institution. You are a backbone of the internet. Just like Facebook and
how they share their graph with anybody who wants to use it, as it spreads the
FB brand everywhere - you should embrace people wanting to use your data. It
will only serve to spread the CL brand further and even more important, it
will make the lives of people all over the World better. I thought that was
what you stood for. I hope you don't prove me wrong.

Regards,

-Bill

------
lhnn
I get that CL has a fast UI that is very easy to understand for non-technical
users. But why can't they somehow work with others who want to improve the
experience?

There have been several sites that I have personally used to aggregate
searches, eliminate duplicates, filter on different attributes, etc. that all
were better for me than CL. (Craiglook, padmapper, etc.)

Every, time, CL flips them off and threatens suit.

FFS, guys, just strike a deal with some of these devs and make a 'tech-
friendly' portal for people who want actual search capability from your site.

------
PythonDeveloper
Total BS Craig... You should be REWARDING innovation, not joining the cadre of
douchebag trolls that try to kill it.

------
gcb
You already got the traction. Offer something complete now. It's already
better.

