
Craigslist blocks Yahoo Pipes - slay2k
http://romy.posterous.com/dont-be-evil-craigslist
======
hyperbovine
Hold on here, let's call a spade a spade. Your project has its own doofy .com
name, calls itself "alpha", and was rolled out on a web site whose primary
function is to connect people who are trying to launch online business
ventures. So to wave the bloody shirt after Craig stated that RSS feeds for
noncommercial purposes are golden, is a little disingenuous.

As others have pointed out, Craig's whole m.o. appears to be preventing people
from building for-profit businesses on top of his site. Craig is anti-
business. That's his prerogative. You obviously disagree with his philosophy,
but a difference of opinion is not cause to go around calling somebody evil.
He's done more good through his site and the $$ he generates off his site
(which gets plowed into his foundation) than you, me and the next 100 people
who read this combined.

~~~
slay2k
It would only be disingenuous if there was any kind of commercial aspect to
the site as it stands. There was none, and "rolling it out" on HN is merely
asking for feedback, not trying to sign up paid users.

In the email exchange I've had with Craig, not all of which I've posted, I've
asked him to clarify on this point:

"Oh, and about being noncommercial. A bunch of the RSS mashup sites I've seen,
like Craiglook, throw up advertising presumably to cover server costs or make
a few bucks. If I didn't want to go that route, would asking for donations be
acceptable ?"

In his response, he said he can ask. I said please do, and that was the end of
it. So for CL to break things when there is absolutely no sign of commercial
purpose is a little unnecessary in my eyes.

~~~
cdibona
I hate to break it to you, but it doesn't matter if there is no 'sign' of
commercial purpose. Are you a non-profit? If you are, let them know. In my
experience they'll hear you out.

------
jeremymims
For the life of me, I can't understand why craigslist has had this mentality.
I basically have to chalk it up to Craig being stubborn. He was burned by ebay
grabbing 20%, he hasn't changed the design ever, and may constantly live in
fear of being co-opted by the man. But like it or not, Craigslist is the man.
And when your users go out of their way to create pieces of software that
actually improve the experience for your users (like letting me sort through
the pile of duplicate, spam, and misleading listings) don't shut them down.
Open up and help them. Craigslist lets people post their information for free
in many cases, but for some strange reason won't let people take information
out.

I had hoped that the culture might become more developer friendly after Jeremy
Zawodny (formerly at Yahoo) joined up. Zawodny even wrote an article about how
all this terrific data was available via RSS, but there was no good way to
piece it together: <http://jeremy.zawodny.com/blog/archives/008513.html>

Craig is a programmer and he comes across as a normal, humble sort of guy. We
like him and we like his service. I don't understand his obsession with
keeping craigslist a) locked in the dark ages for functionality and b) not
letting anyone build something that adds useful features. It really starts to
make you think. Is he just strange or (if you don't give him the benefit of
the doubt like I do) evil?

~~~
jimboyoungblood
Craigslist nets tens of millions in profits each year, most of which I would
assume ends up eventually in Craig Newmark and Jim Buckmaster's personal bank
accounts.

I would guess they are operating under the principle of "If it ain't broke,
don't fix it"

~~~
shawndrost
It's very clear that their primary motivation is not money. It would be
trivial for them to squeeze much more money from their service. Craig could
also sell his company for more money than he could ever spend.

~~~
qeorge
That's true, but they couldn't continue to run with the skeleton staff they
currently do (30 people I believe). It would fundamentally change the nature
of the business.

I agree with you that money is not the primary motivation, but I think the
choice is more complex than simply opting in or out of potential revenue.

------
luckydude
You guys. Get a grip. It's his company, he built it, you didn't. He makes the
rules, you don't.

If you don't like that, then go build your own site that has the same level of
success and define your own rules. And deal with all the people who tell you
you are doing it wrong, that's a lot of fun.

For the record, I have no connection to craigslist other than admiring it. I'm
flaming because I've got my own company and I deal with this crap as well and
it's annoying. If you don't like the way someone else is doing it then either
go do it better or shut the fuck up.

~~~
jacoblyles
They're a monopolist with a bad product, like Microsoft, but they have a
crunchy image so bay area hackers love them, despite the fact that they are
freezing a whole sector of business in the dark ages (again, like Microsoft).

Everything you wrote could have been used as a defense of Windows 98, or IE 6.
You may be defending a shitty user experience with a monopolist position, but
we love you Craig because at least you hate capitalism.

~~~
luckydude
As a long time user of craigslist I disagree with the "bad product"
characterization. I find their website extremely pleasant to use, much like
google. When will people learn that content is king and all that
flash/image/whatever crud may look great but rarely does anything but distract
from the actual content.

But rather than asserting that it is a fine product (which it is, though I
agree it could be made better), how about spelling out what it is that you
think would make it better? I wouldn't be surprised if Jim and/or Craig reads
this thread at some point.

~~~
jacoblyles
I was on the fence about Craigslist's minimalism. Then I tried Flippity's
search of Craigslist's data (the service that Craigslist just killed for no
reason) and realized that an equivalent Craigslist search would have took me
about ten times as long, manually searching through all the ads.

This isn't simplicity. This isn't minimalism. This is Ludditism, backed by a
monopoly position. How many millions of hours are wasted _every day_ because
of Newmark's anti-market fetish?

And Jim and Craig have made it abundantly clear in interviews that they don't
care about user feedback, so I don't expect anything to change.

------
callmeed
People here on HN often make comments about the dangers of building your
business/app/service on top someone else's business/app/service (be it
Twitter, FB, etc.).

If your app depends on Yahoo! Pipes AND Craigslist, well ... that's probably
extra dangerous.

------
luckydude
I know Craig personally, slightly, we've been on panels together. I also know
Jim and Susan, been to their place in the city, hung out.

I'm old, 47, been in the valley for 20+ years. Seen a lot of shit, seen a lot
of people get screwed over in stock options, seen the good and seen the bad.

What I'm trying to do is establish credibility (prolly didn't do a good job).
Whatever. You can listen or not.

Craig is good people. Really really good people. He cares, he does the right
thing as much as he can do it. Craigslist could be at least 2 orders of
magnitude more rich if they wanted to fuck you over. They haven't done that,
they repeatedly try and take care of their users.

If you are unhappy with craig I pity you. Real life is going to fuck you.

~~~
tezza
Thanks for your comment.

It's hard to establish your credentials (account created 7 hours ago, just
like your post).

Normally I would have looked through your previous comments/blog and tried to
establish veracity of your claims. Freshness of account means thats not
possible.

But in my mind there's two options

1) You're claims are not legit, you're a stooge

Shame on you.

2) You are legit, and you are mentioning your personal opinion

Thank you! It's nice to hear someone vouching for someone.

The HN startup community do it all the time. They just happen to be 20
somethings looking (rightly) to fit into the YC community.

I don't think you are appealing to authority as your comment will be judged on
its own merits. Fuck those argument technicality Nazis anyway... you're not
claiming to be a doctor, or the head of some company. Nor are you claiming to
have special information that other people don't have... I'm sure lots of
people have met this Craig (not me I'm in London) and could deny your view
hiding behind screen_names.

I'll wait to see whether you continue to post to see if its 1) or 2). In case
you _are 2)_ please don't be discouraged by ignorant haters.

~~~
luckydude
Hi Tezza,

good point on the recent account creation, does make me look like a stooge.

It was/is my personal opinion and I'll try and stick around and establish a
track record. I mostly don't do that sort of thing any more (used to be very
active) because I got tired of all the armchair quarterbacks telling me I was
doing it wrong. There are millions of them and only one of me :)

One slight quibble, I am the head of a (small) bay area software company. I'd
kinda like to not be, I'm tired, but for now it is what it is. To all you
folks who want to run your own thing my only advice is have fun along the way,
spend time with your family, friends, whatever you like, but have fun. The
time passes faster than you can imagine.

Anyway, thanks for the pleasant reply, I'll try and emulate your civil tone.

------
jacoblyles
It hurts me when people chose to keep things broken (Craigslist) for
irrational reasons, especially when it wastes hours of time for millions of
people.

~~~
jimboyoungblood
their decision seems perfectly rational. if services like this one become
popular, they are a huge threat to their business.

they're not morally obligated to provide unfettered access to their data to
anyone who wants it- i fail to see how it is "broken"

~~~
jeremymims
It's broken because the people who go out of their way to create software for
craigslist aren't largely duplicating functionality.

Want a map view so you can see all the relevant listings in one view?

Want craigslist to deliberately filter the same text or picture used for
different listed apartments or jobs so you don't waste your time or get
scammed?

Want to search by squarefoot or by actual bedrooms or by picture?

Apple would say "There's an app for that." Craigslist says "screw you".

~~~
jimboyoungblood
It just means there's a huge opportunity to build a great craigslist
competitor by building a service that has all these features.

~~~
Kaizyn
Have you ever heard of the network effect? Craigslist has such a lock on its
market that you would have to have something extremely compelling to dislodge
them. A few flashy maps and a web 2.0ish appearance wouldn't be nearly enough
to do it.

~~~
elai
Classified sites gain a regional dominance. Craigslist in canada for example,
is only really dominant in vancouver, one of their biggest foreign markets.
Kijiji is dominant in other towns.

------
scelerat
Perhaps it's as simple as CL has no way of determining who is on the other
side of the pipe. In Craig's quoted statement about RSS feeds, it seems that
they are concerned about commercial use of CL data. If Yahoo Pipes essentially
masks the origin of the request, then it makes CL's job more difficult.

As for determining motive -- profit or otherwise -- I suspect it's more about
keeping others' greedy mitts off their community than cold cash. CL has been
about the community of people using it since it was just a mailing list, and
publicly Craig and Craigslist have been pretty clear about what they value.
So, they probably see pipes as a threat to that community.

------
nym
This is not the first time a site has been shut down when an enthusiastic
developer emailed Craig saying "Hey check out what I built!"

...nor will it be the last I suspect.

~~~
madair
References?

[Edit:] Thanks!

~~~
biotech
<http://www.codinghorror.com/craigslist/>

------
skolor
You know, I was playing with a couple scripts (mainly Markov chain
implementation), that used Craigslist as a source for data. I got a good ways
into it, and got a working model (I could generate text based on the handful
of inputs I had set up for it). It wasn't until that point (I know, stupid)
that I went back through all the ToS and realized I really couldn't do that.

Now, I know I might be able to get some where with them if I tried emailing
them, and asked for the data I wanted. Then again, it sounds like I probably
wouldn't. Either way, while the data was surprising well suited for what I
wanted to do, it isn't _really_ available, at least not in any way I could
actually use it.

It makes me kind of sad every time I see someone do something like this.
You've created something super cool, something with tons and tons of data. For
what I was using it for (localized text samples), I can't think of a better,
more complete place to find data from. But when you see something cool,
something you never thought of/never thought was important enough to
implement, and go out of your way to squash it, it makes the part of me that
loves data feel bad about it. Craiglist has, at any given time, literally gigs
of written text with all sorts of metadata on it. In the short bit I
processed, I took all of the data from just a handful of boards, and ended up
with over 150mb of data. Considering that most of that was <2 weeks old, They
likely process up to a terabyte of data every year, with fairly good metadata
on it. Seeing all of that only being used at face value seems like such a
waste.

(As a side note, anyone with a bunch of written text samples, with information
as to where each of them was written, I would really like to talk to you.
Actually, that goes for anyone with any sizeable amount of data. Just because
you don't see a use for it doesn't meant that there isn't someone out there
who would absolutely love to get their hands on it.)

------
petercooper
Back when I was running a service called Feed Digest (which I went on to sell
- mostly due to the issue I'm raising here - now known as Feed Informer) this
was a common problem. Not just with Craigslist, but with feeds from Google and
all over the place. We wouldn't hit a single feed any more than once every 30
minutes but getting whitelisted everywhere was an exhausting task and ensuring
you could always get the feeds for your users to reprocess was so difficult
that I was heavily turned off of getting into a "man in the middle" situation
like that again.

------
antidaily
Unfortunate, but predictable... as pointed out in your previous thread:
<http://news.ycombinator.com/item?id=963634>

~~~
slay2k
Craig Newmark: "as a rule of thumb, okay to use RSS feeds for noncommercial
purposes."

What's predictable here ?

~~~
rms
Predictable is that craigslist seems to have some vendetta against external
web apps, and you showed them a big data hole they had.

I'd ask Craig again on Twitter about the Pipes thing, he's better at
responding to tweets than emails sometimes. <http://twitter.com/craignewmark>

~~~
slay2k
Not today he isn't, asked him 3 hours ago, no response.

------
justinhj
Surely if you build something on a stack of services that belong to other
people, this is a hazard you have to face.

Craigslist have no obligation to let you use their data however you like.

------
leelin
Maybe this isn't as reactive as you think. I noticed just in the last few days
CL blocks any listing containing the string "bit.ly" (a net good for CL
users). Try creating a post and you will get an error on the screen where you
normally solve a captcha.

Maybe your correspondence coincided with a more general security tune-up they
decided to roll out on Thanksgiving?

------
mmagin
As a random datapoint: Back when I used bloglines, I had subscribed to some
searches with them, and after a few weeks, it stopped working. Craigslist
seems to block anyone who hits a given rss feed much.

------
blasdel
Your problem is that in their eyes Yahoo Pipes doesn't have any _legitimate_
uses against their feeds -- it's a naked proxy.

They'd never block Google Reader -- they'd sooner just disable RSS entirely.

------
fizx
<http://housingmaps.com> has been up for the last, what, three years? I guess
some mashups are ok.

------
sh1mmer
For what it's worth YQL still works (it uses a different UA, "Pipes 2.0") so
the YQL module in Pipes will probably work too.

That is until they turn that off too.

------
mcantelon
Suck the RSS to a VPS and read it via Pipes through that.

------
xtrmntr
so how is <http://housingmaps.com> still allowed?

~~~
ig1
they got special permission from craigslist

------
maurycy
Sorry to say but company that treats developers like dirt is doomed.

~~~
misuba
Craigslist has continued to thrive over at least ten years of neglect and
hostility towards developers as well as interaction design. They seem to
disprove your claim.

~~~
maurycy
It is rather blunt to assume so. Nearly every week we discuss whether Google
can be defeated. Yet, you think that Craigslist cannot?

My point is that their closeness will be major factor in their failure.

EDIT: I don't buy Craigslist hype. To me, their service is poor. Outside the
US, especially in the East Europe, there's a lot of better alternatives. There
was a lot of discussions why they succeed. I point out how they're likely to
fail.

