

Craigslist posts shown on a map while viewing Craigslist housing - welder
https://chrome.google.com/webstore/detail/omonmigaleaafgpkgoammpclbcdepjpi

======
chimi
In the end, client side scraping of the data actually increases the traffic on
CL's servers, because multiple people have to scrape the same posts, rather
than a central server. If padmapper remained small, then they probably scrape
more posts than they have unique views. Of the 100 they scrape, maybe only 10
are seen. As padmapper et al grow, it may reduce the bandwidth on cl.

In 10 years, if all clients looking for apartments use a plug-in however, now
you have cl paying for a lot more bandwidth than outsourcing this
functionality to a centralized service, because for every 100 posts they
scrape, they may only really look at one and they want the ability to show
thousands on a single map.

I really don't understand why cl doesn't just do it themselves. It's not hard.
I've built these maps countless times on top of all the popular mapping
services and using different geocoders.

The hardest part is figuring out what the poster means in their location:
where is "forth and main" [sic] or do you mean "East 4th St at Main St?", but
I can see a simple address validator that updates a map with their location
until they get something that will geocode correctly. That's the best solution
and everyone wins here, except padmapper and the like, but there are other
sources of data to map that aren't from craigslist too.

It's really trivial. The only other issue is cost, but those can be covered
easily with all the money CL leaves on the table.

~~~
olog-hai
> client side scraping of the data actually increases the traffic on CL's
> servers

Of course, but not in PadMapper's case.

"Padmapper gets its data from 3Taps, a site that collects and republishes
Craigslist's content. 3Taps, in turn, says it gets its data from cached copies
supplied by Google and Bing, rather than Craigslist directly."

Source: [http://arstechnica.com/tech-policy/2012/08/craigslist-
tighte...](http://arstechnica.com/tech-policy/2012/08/craigslist-tightens-
grip-demanding-exclusive-ownership-of-ads/)

~~~
cdibona
I'm sorry, but you should really ask yourself if this is true. Bing maybe, as
they can do what ddg does and pull a feed from Yahoo/Bing BOSS (maybe?) but to
scrape Google would be a very bad and unreliable (and counter to the Google
TOS) way of getting this information.

This whole situation is very sketchy, honestly.

~~~
cbr
Padmapper used to scrape Craigslist, but stopped after a cease and desist
letter. While buying the data from a service that scrapes search engine caches
may be unreliable, my understanding is that it's legally much more defensible.

------
welder
I've just pushed the full source code to github:
<https://github.com/alanhamlett/CLMapper>

If you find issues or have feature suggestions, please add them and I will
push fixes: <https://github.com/alanhamlett/CLMapper/issues>

~~~
efuquen
This is not working for New York for me, didn't show up at all, but it was
definitely working in Gainesville, FL (where I'm at now), popped up right
away. Are there certain cities it will not function in, for whatever reason?

~~~
welder
It works for: <http://newyork.craigslist.org/hhh/> but newyork.craigslist.org
has housing categories different from sfbay.craigslist.org and a Chrome
extension's client script only runs on the pages which match the
manifest.json:
[https://github.com/alanhamlett/CLMapper/blob/master/src/mani...](https://github.com/alanhamlett/CLMapper/blob/master/src/manifest.json)

I'll add the newyork urls to the manifest.

------
untog
Cool idea. Will be interesting to see what companies like Craigslist can/will
do about this stuff. It's not like Craigslist is ideologically opposed to
maps, they just didn't want them to be hosted on a competing site.

So I imagine they might not mind this at all..

~~~
hyperbovine
I use a Chrome extension that automatically displays the embedded photos as
thumbnails next to each listing. Per request, it probably increases the
resource burden on CL's servers by one or two orders of magnitude. I can't
imagine they are happy about that and keep waiting for the day when they
implement some sort of rate limiting.

Or maybe it really is as they say, and they just don't want 3rd parties doing
this sort of thing, even though it's far more efficient. Personally I'm
skeptical.

~~~
degenerate
What extension? I was just looking for something like this.

~~~
hyperbovine
[https://chrome.google.com/webstore/detail/jmcbgcbedienblgnfe...](https://chrome.google.com/webstore/detail/jmcbgcbedienblgnfeecolmmcgocefnf)

------
electronous
I like this. It's interesting that a viable solution to the Craigslist third
party problem is to just make everybody scrape Craigslist. From a purely
technical standpoint, that seems rather unfortunate, but I'd rather have a
local scraper + Padmapper UI than have no Padmapper at all.

------
chintan
<http://housingmaps.com> \- the mothership - how do they still exist?

~~~
ghshephard
They aren't trying to use Craigslist to jumpstart their own business - they
are simply offering a service to users, so CL is fine with them.

CL isn't opposed to these services existing on top of CL, it's opposed to them
using CL to jumpstart their own business that is supposed to _replace_ CL.

Padmapper's eventual goal is to destroy CL - which, of course, CL is not going
to go out of its way to enable.

~~~
MartinCron
_Padmapper's eventual goal is to destroy CL_

What? I don't think that's ever been Padmapper's goal.

~~~
flatline3
It is the unavoidable end-game if padmapper is successful.

~~~
ericd
I think I have some say in the end-game for PadMapper, no?

~~~
flatline3
<http://www.padlister.com/> \- "PadLister is a full suite of tools for
renting"

Are you _seriously_ telling me that you're not trying to compete with
Craigslist in the Apartment Listings space?

 _All_ incentives, evidence, and your actions point in this direction. The
only dissonance is with what you're saying, not with what you're doing.

Even if you mean it now (in context, this seems unlikely), incentives have a
way of wearing down your resistance. If PadMapper actually succeeded in
supplanting Craigslist as the primary destination for apartment ads, you'd be
outright foolish to not start accepting ads directly yourself ... ___ _which
you already do, through PadLister._ ___

So how are you not aiming to surplant Craigslist, exactly? There's nothing
wrong with competing with them, but it's insanely disingenuous to claim that
you aren't.

~~~
ericd
Yeah, that might be the issue, and I'd strongly consider shutting it down if
that would make working with CL possible. I disagree that competition is
inevitable, though.

~~~
flatline3
> _Yeah, that might be the issue, and I'd strongly consider shutting it down
> if that would make working with CL possible._

Carrot and stick. That's not a highly principled stand.

> _I disagree that competition is inevitable, though._

You're doing this out of the pure altruistic goodness of your heart?

All the incentives point in a single direction. There's nothing _wrong_ with
competing with Craigslist. You'd be foolish to not take a piece of the pie --
if you can. Likewise, Craigslist would be idiotic to give away the valuable
data that you need to take a piece of their pie.

~~~
ericd
I'd need to be able to talk with them to see if it's actually the thing they
have an issue with before I would nuke that project. And yes, I started and
continue this project because I wanted to solve a problem, and I find it
fulfilling. Money isn't a strong motivator for me.

~~~
ghshephard
" Money isn't a strong motivator for me."

Please don't say that - it is disingenuous. Money is a strong motivator for
99.99999% of the world. Money lets you do things you otherwise could not do
without that money. It's reasonable to say things like, "Money isn't the only
motivator for me."

~~~
ericd
Sorry, that wasn't specific enough, money past my current point isn't a strong
motivator for me, since I have relatively cheap tastes and low expenses. Even
below market salary for an engineer, being childless, I have such low expenses
that I can afford to go on trips and do the other things I want without
needing to make more. Obviously, if I couldn't afford good food, I would care
a lot more about money and less about doing stuff I enjoy doing. And of
course, that might change over time if I ever wanted to buy a house or car or
had kids.

------
cadwag
Local client-side scraping of CG to bring about similar results to Padmapper -
I like it.

Noting that CG has recently gone on the offensive, I wonder where they would
draw the line. At the moment, everything is done locally and it doesn't look
like the extension is communicating with any central source. What if there was
a central server that aggregated the results of all the distributed scraping
to cache results and a) display them more quickly to users, and b) reduce the
number of hits to CG?

Would CG rebuke the extension b/c its communicating with a central server and
sharing CG's data in a manner not controlled by CG?

~~~
marquis
I had the idea that browser clients could scrape content as they browse then
send that anonymously to a central system that could be used by all, but with
the copyright issue now in play it seems that it would give CL due process to
shut down such a service.

~~~
IanDrake
That's essentially what NotifyWire.com did and got a C&D.

~~~
marquis
Oh well, good to know the idea was put into practice at least.

------
dredmorbius
It'll be interesting to see how CL views this in light of section 5 of their
TOU:

    
    
        Any copying, aggregation, display, distribution, performance or 
        derivative use of craigslist or any content posted on craigslist
        whether done directly or through intermediaries (including but not
        limited to by means of spiders, robots, crawlers, scrapers, framing,
        iframes or RSS feeds) is prohibited. 
    

<http://www.craigslist.org/about/terms.of.use>

Would CL really want to sue its users though?

~~~
netfire
Isn't a user's browser technically an intermediary? The browser reads html
from an initial request and makes automated additional requests for images,
javascript and css files, etc and then renders the page using its own
rendering engine. The only difference here is that the browser is making
multiple requests and using a different rendering engine (the extension) to
display it using a map.

There's an obvious market for people who want the amount of data that
craigslist has, but wanted it displayed in a smarter way that allows better
filtering and searching. They could make a killing licensing their data to 3rd
party sites (or launching a new site with an improved interface). Instead they
are giving users more and more reasons to switch to other services.

------
ericd
Neat. I have heard of CL C&Ding some extensions as well, but it may have just
been a subset of extensions that were doing something they didn't like.

~~~
dclowd9901
You mean like overloading their servers with needless scraping?

~~~
hnriot
There's no reason to assume this will overload CL servers or CDN.

------
ebun
FYI, <http://www.housingmaps.com/> is still a decent CL housing + Map mashup

~~~
deltaqueue
As are <http://mapliv.com> and <https://www.padmapper.com/>

------
adrianwaj
Would be great if it worked for cities outside of the US.. the ones I tested
(even in English) did not. Map stayed centered on California.

~~~
welder
Sorry, I made this in 24 hrs. That's one of the things that will be fixed
soon.

~~~
adrianwaj
No need for apologies. Craigslist should've put out a plugin like this, that
way they can keep their site simple, but also give it some boost for people
that want it.

------
reustle
I was very close to making a bookmarklet / extension like this, but I'd rather
see padmapper succeed.

