
How I built a Slack bot to help me find an apartment in San Francisco - rmason
https://www.dataquest.io/blog/apartment-finding-slackbot/
======
ThePhysicist
When I was searching for a flat in Berlin I wrote a bot that would
continuously crawl EBay small ads and look for suitable apartments (based on
surface, neighborhood, price, ...). When finding a compatible one, it would
write to the poster and if a phone number was given send it to me via Slack. I
would then always be the first person to call or write, and this helped me to
find a really nice flat within just three weeks. The hardest part was
pretending to have read the ad when people called me back after my bot wrote
them, or explain how I could already call them when their ad was online only
for a few seconds :D

Here's the code btw:

[https://gist.github.com/adewes/c9b2a71457c6c6f01f2f](https://gist.github.com/adewes/c9b2a71457c6c6f01f2f)

~~~
TillE
That's quite an unusual selection of districts! From Charlottenburg to Tegel
to Friedrichshain and Neukölln? I can't even guess what they have in common.

I didn't know that many people used eBay Kleinanzeigen for apartments. Good to
know.

~~~
ThePhysicist
My main criterion was good access to public transportation. I lived in various
areas in Berlin already (each one has its individual flair), eventually I
ended up in West Berlin close to Kurfürstendamm, which is a nice, quiet and
central area.

The housing situation in Berlin is not nearly as bad as in other large cities
BTW (contrary to what most people say), and although it got more difficult to
find something nice for a reasonable price it's still possible.

------
lisper
I like this post not for its technical merit but for its pedagogy. Scraping
Craigslist and posting the results to Slack is not rocket science, but the
project is described in enough detail to make it accessible to -- and
potentially reproducible by -- someone who didn't already know how to do it.
IMHO that has a lot of value.

~~~
HiroshiSan
Absolutely, I've really been enjoying the influx of good pedagogical posts
lately like this one and the one on the lisp dinosaur.

As a beginner to programming, not knowing how to build anything, and seeing
someones thought process. It's very valuable and enjoyable to read.

------
brchr
Even with tools to automate the search through listings and get you ahead of
other buyers, one of the tricky parts of apartment hunting is its “optimal
stopping” structure: you have to decide when to commit without knowing whether
an even better apartment might be out there.

This is a famous math problem (the “secretary problem”) with the lovely
solution that you should define a time interval over which to search and not
commit to anything until you’ve spent 1/e of that time -- 37% --
noncommittally exploring your options.

[https://medium.com/galleys/optimal-
stopping-45c54da6d8d0#.lj...](https://medium.com/galleys/optimal-
stopping-45c54da6d8d0#.ljw4cby46)

------
toomanybeersies
I went hunting for a flat a while ago here in New Zealand, I gave up pretty
quickly and just ended up staying with relatives. I've now moved cities and I
live in a backpackers because I can't be bothered finding a flat (plus
interesting people here!).

It's just so draining to have to contact all these people, and view their
places. I guess it's like job hunting, except you're paying money. It doesn't
help that I'm not very good with strangers.

~~~
osxrand
Sounds like you're enjoying living where you are precisely because of
strangers, some of which I'm assuming you meet ;)

~~~
praptak
Different kind of stranger relation. Renting with its information asymmetry
has a huge potential for creating distrust. Did the landlord hide how nasty
the neighbors are? Will the tenant vandalize the apartment?

Backpacking/couchsurfing is much more lightweight.

------
subspaceman
To find our current apartment in NYC, I just used IFTTT, plugging in the URL
of a custom search (neighborhood, price, bedrooms, etc) on craigslist. Worked
a treat

~~~
lighttower
How did you filter out duplicates or old ads?

~~~
jhalstead
There's a "Posted Today" filter on Craigslist that I used. That helps with old
ads, but not continuously reposted ads though, which was still a problem.

------
gcr
Those real-time notifications definitely make a difference.

When I was searching for apartments in the NYCish area, I asked Zillow to ping
my phone when a good listing came in.

One day, I got a ping during class and called the broker as fast as I could.
"Yeah," he said, "I posted that listing 20 minutes ago but you're like the 6th
caller. You almost certainly won't get it, sorry."

~~~
Nilzor
The concept of selling to the highest bidder isn't popular in NYC?

~~~
rtpg
Well as the landlord, would you rather spend 2 months going over the
appartment with 12 people or just take the first offer at 10% over list price?

Especially if you could go through 12 people and end up with nothing (since
that gives time for people to find alternatives)

~~~
icebraining
_> Well as the landlord, would you rather spend 2 months going over the
appartment with 12 people or just take the first offer at 10% over list
price?_

Neither? We schedule an open house for the next weekend, people make their
offers that day, then we accept one the following day.

~~~
rtpg
Shows what I know about selling houses. Definitely sounds like a better
solution

~~~
yitchelle
I would have thought that selling vs renting are two different situations.
Different enought to warrant a different approach for each of them.

~~~
icebraining
Depends on the type of rental, I guess. If you're looking for long-term
rentals (years), the process is probably much similar than if you're looking
for seasonal or shorter-term rentals.

------
mcpherrinm
While this is technically fine, the process strikes me as unnecessary. I
suppose that's not too surprising for somebody new to the area and mostly
experiencing it from scary anecdotes.

The list of "San Francisco" neighborhoods being searched are almost all more
than 1.2 miles from BART. I'm sure it's hard to find an apartment if you have
a robot excluding nearly all listings.

FWIW, there's been an apartment vacant in my building for 2+ weeks, with very
few open house attendees. Other nearby buildings are much the same. If you're
willing to look slightly outside the most desirable neighborhoods, it isn't
that hard to find an apartment. Affordable? Well that's another story; this is
SF after all.

~~~
yarou
> If you're willing to look slightly outside the most desirable neighborhoods,
> it isn't that hard to find an apartment.

Not trying to be pedantic but "willing to look slightly outside" is pretty
ambiguous for a city like SF. The Tenderloin is _slightly_ outside of Nob
Hill, a very upscale and posh/safe neighborhood, whereas the Tenderloin greets
you with used heroin needles on the street, homeless crackheads running
around, and the noise of constant sirens.

I agree with you on affordability, though. Number one reason why I'm glad I'm
back in the South Bay. Also, I'd rather enjoy the laid back nightlife of Los
Gatos or Campbell over the Marina or Polk Street (20 year old sorority girls
puking all over the place) any day.

~~~
mcpherrinm
Sure, don't live in the Tenderloin.

But that doesn't mean you can't live in Forest Hill, or the Excelsior, or
Balboa Park, or West Portal. I find lots of people totally discount what I
think are some of the nicer parts of SF to live in because they're not walking
distance from the mission.

~~~
kbar13
how's the public transit in those areas?

~~~
uiri
Balboa Park has a BART station.

The K, L and M lines of the Muni Metro serve Forest Hill, West Portal. The L
line goes to the SF Zoo but the K and M (along with the J) lines terminate at
Balboa Park station. Personally, I prefer the Metro over BART due to its
cleanliness and frequency of service.

I'm not sure about transit in Excelsior but I don't think it is that great.

~~~
mcpherrinm
Much of the Excelsior is a short walk to Balboa Park or Glen Park BART/Muni

------
4ad
Well I also built a program[1] to help me find an apartment in Vienna, in
2011. We actually needed two apartments, and we wanted them close to one
another. We quickly found out how bad real estate websites are with moderatly
complex queries even when wanting only a single apartment. With the additional
constraint of needed two apartments close to each other... yeah, they were
mostly useless.

So I built some shell scripts that scrapped all the websites, and generated
some tables of pairs of houses sorted by the distance between them that can be
easily read by awk. Then I could run queries on them.

All the scrapping was done with regular expressions, no fancy HTML parsing
here.

For the distance calculation, I got the geographic coordinates by piping the
address of the residence to Google Maps. I then calculated the geodesic
distace between them in my scripts. Initially I wanted to let Google Maps
calculate the more useful walking distance between residences, but that made
the algorithm O(n²), and I ran into Google API free quota issues even with
O(n). Geodesic distace was a good enough proxy though.

Being able to use awk, I could use any kind of arbitrary query I could think
of. I filtered all residences that were not direct sales (used an agency),
they were unfurnished, that were in a place where I didn't want to live (few
such places in Vienna though), that were outside my price range, etc. basic
stuff.

However, I could create arbitrary utility functions. For example it was really
important for me that apartments were close together. So I was willing to
sacrifice location, or the total area, or the numbers of rooms if they were
really close, but if they were further away, I required more rooms or better
location. No real estate agent or website will be able to do this for you.

In the end, it was too much trouble to rent two apartments, so we only rented
one. However, the software was still very useful as it presensed all data in a
much useful format, multiplexing data from all websites, and the data coming
already filtered.

Plus being all text-based, and this being Unix and all that, I could easily
manually input the 100 or so metro stations as "houses", so we could sort
apartments based on the distance to the nearest metro station. And again I
could create arbitrary utility functions. For example the U3 line is much more
important for us than the U6 line, and we really don't care about U2 at all.

[1] [https://code.google.com/archive/p/operation-
housefinder/](https://code.google.com/archive/p/operation-housefinder/)

------
visakanv
I have a feeling, from the title alone, that some people are going to find
some way to mock the OP. "You built some thing to do some task? I do it
differently. Loser!"

~~~
rootlocus
To be honest, I was hoping for something more impressive, like making a slack
bot which did some heavy lifting (scraping conversations about real estate?).
The moment I saw "scraping craigslist" the magic was gone.

The title sounds like "How I used a cherry to make a birthday cake".
Intriguing, yet disappointing when you learn he only placed the cherry on the
top of the finished cake for pretty decoration.

~~~
kelnos
Technically impressive or not, I think what he built is super useful, and it's
a testament to his skill, foresight, and charity to build it in a way that is
configurable and reusable by others. My plan is to live in my current
apartment until I decide to buy something, but if I end up looking for another
rental, I'd likely dust this off and use it.

~~~
rootlocus
You are right and I forgot to mention it. I was disappointed by the
'misleading' title, but I still appreciate that he built a useful and easy to
use tool. If I had made something similar for myself, I wouldn't have polished
it and it wouldn't have been usable by anyone else.

------
viach
Is scraping CraigsList legal?

~~~
lowglow
I've been hit with a Cease and Desist from Craigslist. Felt bad man. :)

~~~
jxm262
Me too :(

I wonder how many people have. A quick search on GitHub shows all kinds of cl
scrapers.

~~~
lowglow
I was providing a public service and offering free trend analysis graphs on
technology stacks listed in job postings over a couple of years for different
area. It was nice, but they wanted to say I owed them something like 2k per
piece of information, fore every day I stored it.

------
LeonM
I'm currently trying the same approach to get an apartment in the Hague area,
Netherlands. I find it very unreliable, but that's probably due to the poor
state of house listings here.

The whole experience of searching a suitable house is just aweful, all listing
sites are basically full of spam and scams:

\- most houses are already taken, but they leave them on the site because free
exposure/advertising for the realtor/landlord.

\- pictures that are not taken in the actual apartment (they just reuse
pictures of a different apartment that kinda looks like it).

\- ads that look like normal list items (dark ui).

\- super good looking houses for low prices, that turn out to be fake, just to
promote the realtor/landlord name.

\- loads of hidden costs (service, administration, VAT, parking etc).

\- all sorts of requirements (based on sex, income, ethnicity, type of work
and what not).

~~~
Veen
I had similar problems finding a place in Budapest. Craigslist proved
particularly unreliable: the usual process was to find a great looking
apartment, do a reverse Google search on the image, and find the same place
advertised in Paris, Berlin, and numerous other places. And those we did
contact were scams: "I'm in Portugal at the moment but I'm happy to travel to
Budapest to show you the place if you send me the deposit in advance."

~~~
fosco
try
[http://www.hireahungarian.com/tablet/index.html](http://www.hireahungarian.com/tablet/index.html)

Disclaimer: I know the owner (friends with my wife who is from the area, we
are living in Eastern US)

They are flexible and probably willing to be a proxy on your behalf.

~~~
Veen
Thanks! I found somewhere decent and reasonably priced in District VI after a
lot of hunting.

The best place to look seems to be Facebook groups. People quickly call out
scammers and the real estate folks post new listings regularly.

------
bayonetz
Recently moved to SF and wanted to be able pick prospective apartments from
Craigslist based on how close they were to BART/Muni stops...so I hacked
together a little bookmarklet to help with this. Check it out if you are in
the same boat:

[https://github.com/christopher-skeels/add-walking-times-
to-t...](https://github.com/christopher-skeels/add-walking-times-to-transport-
on-craigslist-sf)

------
danielmorozoff
Cool project. A chat bot plugin to handle intro convos and scheduling with the
brokers would be great :)

------
nxzero
Honestly suprised that Craigslist doesn't embrace people interfacing with them
via an official API; yes, I know lots about CL, but the after bring on (forgot
his name) they changed their back an a lot and a bit on the front too.

------
wcummings
I feel like all these newfangled slack/whatever bots fall into two categories:
stuff that used to be or would otherwise be a (toy) IRC bot, and what
basically amount to annoying phone support menus, in text form. Not impressed.

~~~
dev1n
Hammers are that impressive when you already have a rock that can do the job
just as well.

Yet people still prefer hammers over rocks..

~~~
wcummings
All this proves is that people still prefer to reduce support costs at the
expense of consumers.

------
punnerud
Improvement: Calculate real walking distanse using Google Maps or
OpenStreetmap.

------
ambicapter
Anyone else have beautifulsoup crashing at line 160 of __init__.py?

160 p_text = row.find('span', {'class': 'p'}).text

'Nonetype' object has no attribute 'text'

~~~
ambicapter
It appears the pip install is out of date and this issue has been fixed in the
github repo.

------
fudged71
By documenting the process and making this open source it seems like it will
be even harder to find an apartment in San Francisco.

Might be worth trying this code for other cities though :)

------
nommm-nommm
Has anyone tried to post in the housing wanted section?

------
davidhariri
I built something like this a couple of years ago for my own apartment hunt,
but my IP got blacklisted. Has Craigslist changed their policy on scraping?

------
savrajsingh
Craigslist has a built-in version of this -- email alerts on specific
searches. That's the layman's solution. :)

~~~
lh7777
In my experience, Craigslist's email alerts are terrible, for a few reasons:

1\. Craigslist combines listings that come in at once into a single email.
That's fine, but they truncate the results (... SNIP...) if a lot of listings
come in. Why? Are they worried about exceeding your incoming email size limit
with an all-text email?

2\. Sometimes they send emails with zero results. Again, why?

3\. Maybe not an issue in NY or SF where good apartments rent minutes after
listing, but even in Seattle's hot housing market duplicate listings are a big
problem. I've seen the same listing posted multiple times per day for days on
end. All of these listings appear in Craigslist's email alerts, so you have to
wade through them all. While the author didn't address this in his bot's code,
I see filtering out duplicate or "already seen, not interested" listings as
the biggest benefit to a home-made solution like this.

~~~
nommm-nommm
>I've seen the same listing posted multiple times per day for days on end

My landlord's property manager actually does this with the place I'm renting
now. Every 3 hours the previous ad would be deleted and a new one with the
exact same content would pop up in its place, obviously to stay on the front
page. I thought it was a scam at first because surely that's not what normal
landlords do? I did my due diligence that it wasn't a scam (which I would do
anyways) but I was super, super suspicious. Turns out real landlords do that.

------
samirillian
Aren't there existing Craigslist apps that do most of this already?

------
shortsightedsid
The headline is slightly click-bait. Most the work is outside slack - parsing
Craigslist, classifying etc.. The slack channel bit is one part of the puzzle.
The article itself says there are 4 steps and devotes 2 paras to slack.

I guess the author chose slack in the title because it will result in eyeballs
rather than saying how he used Python, craigslist because they aren't "sexy".

~~~
amazingman
You're really reaching for something to complain about. Why?

~~~
wineisfine
I guess the answer is in his user name :)

~~~
pop8row9
You guys can be real dicks to people who express slightly alternate opinions.

