I'm in the middle of an apartment search. I was reflecting the other day on how much searching for an apartment has changed over the last 10 years.
Conclusion: it hasn't changed much at all.
It's still necessary to search on a myriad of sites. Most of the time, by the time you inquire, a place is gone. Or the ad was misleading and that "$2000 2BR" is actually "$2000 for the cheapest Studio we have, but we also have 2BR for $3.5K".
Finding places that match my criteria still requires careful inspection of the description. Checking a filter box for "Parking" excludes places that only mention parking in the description.
With all of the modernization and streamlining that has happened in other parts of our lives, apartment hunting remains unstructured, messy, frustrating, stressful.
Aggregators like Padmapper helped for awhile, but these days, it seems like most listings are posted by scammy middlemen just trying to make a commission on the listing, and now the site actively makes the process even harder.
Is hunting for an apartment doomed to be this messy forever? It seems like an area that is ripe for disruption or a better option, but what would it take to actually solve this?
---
With all of that said, this post is exactly what I needed right now. I've been toying with a very similar idea to improve the efficiency of my search process and this is good inspiration.
I used to work at a startup that aggregated apartment listings. Long gone now, we couldn't compete with Zillow or Apartments.com. But what we were doing is just aggregating all of the rental websites we could possibly scrape into one interface.
It was hard. There are so many scams out there. If you are not hand curating listings, you have to rely on somewhat novel approaches to filter out all the bad data. For example, any mention of 'Jesus' or 'God' automatically blocked the listing. Sure there might have been legitamate listings, but anytime I would go in and check 99% of the time it was a scam. You also have scams of people listing units they don't own or have any relation too. They ask for a security deposit up front and just pocket it, leaving the new tenant to have an awkward conversation with the real resident.
Data is often unformatted too. Scraping out bedrooms and bathrooms can even be a challenge on websites like craigslist where the listing is just one big paragraph. (Not anymore, craigslist has come a long way, but that's how it was 10 years ago). Often times you had to just search for the closest number around the word "bed" or "bath". Don't even think about getting features like "driveway" or "laundry" out of them.
In the end, we ended up utilizing some pretty intense ETL pipelines to collate historical data, census information, property assessment data, and other things to try to get a more accurate picture on our listings.
But that didn't win out. What won out are the sites like Apartments.com or Zillow, where legitimate property owners can post their listings in a formatted searchable way. We could scrape them and post the same listings on our site, but at that point we were just pushing our customers to another platform that honestly worked better than our own.
We couldn't have the most up to date data, that was determined by how fast we could go back to scrape a listing. And often times we were knee deep in a battle to avoid being blocked by these companies. Often times, after we had exhausted our proxies, the only thing left to use was Tor.
> I used to work at a startup that aggregated apartment listings. Long gone now, we couldn't compete with Zillow or Apartments.com. But what we were doing is just aggregating all of the rental websites we could possibly scrape into one interface.
[...]
> We couldn't have the most up to date data, that was determined by how fast we could go back to scrape a listing. And often times we were knee deep in a battle to avoid being blocked by these companies. Often times, after we had exhausted our proxies, the only thing left to use was Tor.
The flip side is if you are looking at listing from small property management companies, they are probably low on resources. The websites aren't very well optimized to serve thousands of concurrent users. Every request to search inventory hits SQL Server which is probably one small box with no automatic failover. I don't like it but unless we somehow help these people better optimize their website (how?),they will continue using heavy handed tactics like blocking scrapers.
They say they're out of the country on a mission trip and are OK renting the place out below market to a responsible person (faith preferred but not required) who will take good care of it. Then they ask for a "viewing fee" before they send you a code for the non-existent lockbox with the keys.
Source: got to the "send me a viewing fee in itunes gift cards" stage of one of these scams, once. Googled the initial email and it's a common scan template.
As someone who just jumped through all the hoops to find a new apartment, the pain is real. I think the main issue is that the incentives are misaligned from the perspective of the seeker. Where I'm currently at (the Netherlands), the market is dominated by agents. Using an agent makes sense from the perspective of the apartment owner — they will take care of managing the ads, initial screening, arranging viewings etc. Given how many people contact a single viewing, you definitely don't want them hitting your personal inbox or phone.
From the agent's perspective, it's good to drive people to their company's website rather than one of the big aggregate ones. Having more people visit it means they're more likely to do business with that company specifically. Getting people to visit your company's site is pretty straightforward: they put up ads there earlier, often so much earlier that by the time you see an ad on an aggregate site, it's already fully booked for viewings.
And that's how we end up with dozens of different company sites, each crappy in their own way, and in the middle of it all, people trying to find a home pulling their hair out, refreshing a bunch of different websites every few hours.
Housing website https://funda.nl is pretty dominant in the Netherlands. Why would you need to search many agent's websites when nearly all inventory shows up on funda.nl ?
I find it much preferable to North American sites like https://realtor.ca that think that number of baths is more important than floorspace (which to my utter amazement is often not even available).
You're right in that most listings eventually end up on sites such as Funda, Pararius etc. However, almost all offers go online on an agent website first, oftentimes considerably so. Your milage will vary of course, personally I started using agent sites after calling an ad that had been up on Pararius for 5 minutes yielded the response that the viewing list is already full. Turned out it had already been up on the agent page for a day and well-priced offers go fast in desirable areas. It is worth noting that I live in a very contested area, I would hope life is better in smaller places.
I’ve noticed this as well. That is why I’ve started scraping those agency websites and send an email whenever a new listing matching my criteria came online. I even made a tool out of it for others to use [1] (it currently crawls around 50 sites).
Same in Belgium. Agencies first email new offers to their registered users, then publish on agency websites, and only afterwards publish on immoweb.be.
There are sometimes rare jewels published directly by individuals on immoweb.be but the bulk of the offers is trash.
I can't escape the feeling that if an apartment ends up being bought by a person who learns about it hours or even a few days ahead of others, then it was priced too low and the seller left money on the table.
+1 to https://funda.nl. In fact as a new immigrant I bought house completely through funda. They have nice tie-ups with all the regulatory/certificate providers such as construction report etc., Fantastic experience overall.
Selling though is a completely different story it's almost impossible without an agent. And they do provide a valuable service; like photo session, publishing on funda and others, taking care of house-viewing etc.,
>Given how many people contact a single viewing, you definitely don't want them hitting your personal inbox or phone.
This is something I noticed last time I was apartment hunting in Montréal. You'd see a sign or listing, call, and the number would be out of service. As if landlords were using virtual numbers/call forwarding services and just shutting it off once the unit was rented.
I might test this one day just to verify it did start with zillow, but after an interesting set of experiences* calling phone numbers for rentals on Zillow I started getting some very odd text messages about bank accounts I don't own and suspicious activity.
I don't call phone numbers on listing sites anymore, even if there's a face and an agency next to it. Better off looking up the agency, going to their website an calling that number (which in all three cases of the above, were different than the one on Zillow).
---
* by "interesting set of experiences" I mean calling the number on a listing took me to an automated menu and a robotic voice where I had to enter the street number, and then first five letters of a street name so it could "look up" the property I was inquiring about, and connect me to the right seller.
Hanging up on that nonsense immediately and calling the leasing agency's number via their website got me a human being in two rings, and the leasing agent for the unit I was interested in after about 45 seconds of hold music.
> From the agent's perspective, it's good to drive people to their company's website rather than one of the big aggregate ones
This is strange, wasn't my experience at all. Just about every house I viewed through funda was advertised by agents. And when I put up my house for sale the first thing the agent did was to publish on funda. All the agents care about is selling a house they get a hefty sum (0.3% if I remember correctly) and are totally fine advertising on funda.
As a house seeker I dealt with a bunch of agents and the experience wasn't all that bad to be honest. Mind you all this was in early 2019; from what I hear late 2020 onwards the market has gone bananas in the Netherlands and Amsterdam in particular.
In my experience in NL makelaars (brokers) serve particular niches, and I expect they can outcompete larger firms that don’t understand the neighborhood or municipal market as well, the market segment (roughly price), the kind of buyer/renter, the kind of seller/leaser, not to mention just their access to “deal flow” in their networks unrelated to their expertise.
Think it’s all related apartments and houses not really being fungible.
ESIT: Also, more directly to GP’s experience, when I was in the market in NL I found the biggest aggregator to be very useful, which is apparently different from their experience: funda.nl
Because the duration is too long (a year) and the cost is significant for all parties to let a middleman take a huge cut of this pie. For shorter duration stays, we already have the consolidation around airbnb.
Because it's a mostly unregulated market. A new competitor has minimal government regulations to satisfy in order to start and everyone and their dog can start doing this job (and believe me, I've met some interesting agents when looking for the best deals both looking for rent and renting out my property). The big centralised agencies don't really have a way to stop smaller agents offering cheaper services and serving different markets (even if they're just different neighbourhoods) is hard.
This is how capitalism works when there is no government helping monopolies to form in the name of security / justice / anti-money laundering / insert-politician-buzzword-of-the-day.
The natural end condition of an unregulated market is a monopoly. That is why we need regulation.
So, something is operating to provide implicit regulation. In this case it seems like chaos serves that role. Monopoly relies on a self-ordering property common to most market conditions.
The explanation GP provided is extremely low cost of entry. An agent doesn't even need an office. Here is my country many agents go free wheeling with a phone number and a bike. All they need is some connections with homeowners and they're off to business.
In San Francisco, when I was looking for an apartment,, I always ended up with an apartment with the worst pictures / missing things in the description on Craigslist, because they had less interest / were cheaper.
The market for rentals is never efficient and will never be! People who post to Zillow with a perfect description know they can extract more money for it.
I bought my house in the late-ish 90s before the web was really a big thing and certainly before things like Google StreetView. I spent a lot of time driving around to see places where I ended up not even stopping the car once I got there.
Today, there would be a lot more information available. However, as you suggest, I assume all that available information probably increases the efficiency of connecting a property with potential buyers/renters so you're less likely to luck into something that others just haven't found yet. (The funny thing is that the house I ended up buying turned out be being sold by someone I knew quite well at work.)
We almost didn’t stop into the house we now live in. It was #10 of 10 of open houses on a rainy Saturday and we were just looking to get a sense of things we both liked/didn’t like. This listing had terrible photos and the building had almost no updates since 1993. We walked in and in 2 minutes knew we’d want to buy the place, in part because it showed so poorly and the sellers were motivated to close. Zillow was a thing, but nowhere near as big a force as it is now.
When I saw the house I eventually bought, I had seen one property I was interested in but was on the fence about. When I saw this one I was pretty much sold. Great property. The house was old and very little had obviously been done with it for decades. And there were some things (like on the small side and one bathroom) that would probably have been showstoppers for a lot of people but weren't an issue for me.
Various things cost me a fair bit of money, effort, and angst that I'm glad I didn't know about at the time. But, at the end of the day, I got a ridiculously good deal--especially by today's standards--for a semi-rural place only about an hour out of Boston.
The worse the advertisement the better. I lived in an absolutely lovely townhouse which had no pictures posted and a description which failed to mention any of the amenities. My current house was shown with no staging and blackout curtains. They only got 3 offers in 2 weeks, in a market where most listings get t 30 offers in a week. It pays to look seriously when doing so takes more effort.
When I moved to Sunnyvale, I found a place on Craigslist where the entire description was typed in capital letters. The landlord was indeed quite eccentric, but we got along well. 2 bed 2 bath, vaulted ceilings. There was a pool that I maybe used once. It was good for 3 years, until his sister took over the family business and evicted him! Poor guy had a lot of mental health problems, but she was the insane one...
Because you can use your position to promote expensive listings with more profit. A truly functional aggregator would lower prices by creating efficiencies. It's more profitable to let the market utilize price discrimination as long as the favor is to the supply side. If there is an oversupply then the aggregator would work to raise prices and would be more profitable. Right now the most profitable renter favoring model would be an aggregator that charges renters a fee. The problem is the same people don't rent frequently enough so customer acquisition costs would likely be too high.
walk around the neighborhood you're thinking of renting in, and look around for signs advertising rentals with phone numbers on them.
call them with the bdrm count you're looking for, and if they know exactly what building you're talking about and units they have available, sometimes they'll give you the code to walk right in and check the place out.
you can use this convo to push the rate down a bit or ask for a rent concession, and since you're likely talking to the landlord/agency it'll be no-fee.
If you are in a hard market (SF) I would highly recommend stalking. Get a list of buildings you truly like. In SF I had like 10 I would kill to get into. When your about 1-2 months from your lease expiring, start stalking hard: call the managers, talk to people walking into the building, talk to the mail man - anything. Nice SF places rarely have openings. "Oh Mrs. Baskin passed away - we have will have a new listing in about a month". On it, I have my cashiers check, no site needed, my credit report printed, 4 references in hand, I even help the apartment manager carry in her groceries - and I don't mind hearing her stories. There are also a few families that own 20 or so buildings - once you get on their "good list" - you are in.
The places you get are amazing. Clear views of the SF harbor, tucked away in PacHeights, full parking, cheaper rent etc. After doing this for 15 years I knew all the managers of those 10 buildings by name. My friends that "wing it" get sooo screwed. They go through listings, trying to piece if it is a nice place, always getting their initial deposit stolen, application fees swiped, going to open apartments 20 people deep, dealing with large corporate landlords, on and on. Make friends!
> Is hunting for an apartment doomed to be this messy forever? It seems like an area that is ripe for disruption or a better option, but what would it take to actually solve this?
I hesitate to say it, but there is already a solution to this, and it has been the one that New York has had for years: you hire an agent. A good agent will sort through all of that for you, and streamline the process to the point where you can go from search to signing in under a week (if not quicker). Of course, there's the problem of bad agents, but that's true of anything. As you note, online services are often bad, as well.
In other words, I don't see this as a problem without a solution; I see it as a problem without a software automated solution, which is true for a lot of problems in life, particularly ones that involve huge numbers of messy, opinionated people with different motivations and interests.
In other countries, not in the USA, I've never once had a good experience with renting via an agent.
It seems it's in their job description to try to convince me to pay more for less. They listen carefully to my requirements and then take me to places that don't match at all.
I guess they figure since I've taken my entire afternoon with them, and since they've been friendly, I might give up on what I'm looking for and rent whatever it is they decided to shill that day.
Of all the countries I've rented in and had occasion to interact with agents, ones in Turkey behaved the most like cartoon caricatures: they use every sitcom used car salesman trick in the book. It would be funny to watch, except that they're perpetually wasting their time and mine.
This doesn't work in all US markets unfortunately, as I discovered recently when considering that very option you suggest. In some slower towns a few agents are willing to do this on the side as a source of income, but then in places like SF the home buying market is so hot, that it's comparatively bad ROI for agents to spend any time on most rentals. I'm curious how NYC manages to pull that off.
This was my experience as well in Seattle. I did a corporate relocation and they also provided a paid-for agent to find rentals.
She was pretty much useless. Most of the places she sent us didn't match our criteria very well, and quite a few were above where we wanted to spend. As each day went by and we nixxed place after place she clearly de-prioritized us in her work (I'm assuming she was working with multiple other people who were looking for places) and became less responsive.
At the end of the day I had to do my own hunt and fly up a second time to look at places over a weekend to find something.
And that's a paid agent from a large moving corp. I do wonder if a (more expensive) boutique agent would have been more effective but honestly having worked in real estate I consider them all at the same level as used car salesmen anyway.
I just spent two months apartment searching in NYC. I talked to many brokers but not one was interested in actively searching on my behalf. The market is incredibly disadvantageous to renters right now. Maybe parent is sharing experience from a different time or has broker connections I did not.
As an aside I also tried to automate my apartment hunt. The main thing that matters in NYC is time to respond. Unfortunately Zillow, StreetEasy, etc are not very easy to automate on the messaging side due to bot countermeasures. It was an incredibly time consuming, manual process. Happily found a great place though.
A lot of NYC agents also do this to supplement income. Real estate is feast-or-famine, and especially during the slower sales markets you can find that agents are highly incentivized to do a good job. But yeah, you're going to take a back-seat to the purchase transaction in a hot market, unless you make it worth their while on a dollar-per-hour basis. That's the downside of using a "regular" real estate agent.
In NYC specifically, there are agencies that make most of their business from renter-side representation. You can find the good ones easily on the usual review sites.
There are a lot of licensed real estate agents in the SF Bay Area. Very few of them get the lucrative sales listings. There are plenty of extra agents with time to work for renters if they wanted to. But most renters don't seem to want that service, and most lessors aren't accustomed to working with agents.
> that it's comparatively bad ROI for agents to spend any time on most rentals.
In Melbourne Australia, what I have seen is that rental agents are different and the real estate agents are different although they work for the same company.
This isn't strictly true. Historically many do, but increasingly many don't. It's been changing. Whether you pay for your own representation is also an entirely separate question.
As someone else in this thread noted, the landlord often hires an agent because they don't want to deal with the tidal wave of crap that comes from dealing with applicants. That person gets paid, obviously, and the landlord isn't the one paying them. That's the rate you're talking about here. These people are not incentivized to provide good service to the renter, and typically don't. They suck.
Higher-end buildings often have in-house agents (again: you're paying for this, whether you realize it or not), and these are called "no fee" buildings. But an increasing number of places don't have either, and just provide access to professional renter's agents instead of listing publicly. Or they do list publicly, and barely respond to the tidal wave of yahoos, knowing that motivated renters hire an agent. YMMV.
That solution is bad. You have no idea what you’re missing out on by using an agent. You might think you had a good agent, but unless you’re already really familiar with the market, you could have missed out on significantly better matches and you wouldn’t even know.
You know how we know this is a solvable problem? Listings for places to buy don’t have nearly as many problems.
Like I said, there are bad agents. There are also bad plumbers. There are bad doctors. There are bad housekeepers. This is not a problem unique to real estate, and tools exist to help you find reliable service professionals.
> You know how we know this is a solvable problem? Listings for places to buy don’t have nearly as many problems.
Just as OP said, there are tons of sites out there that claim to "solve" this, but don't. So no, I don't know that this is a computer-solvable problem.
Essentially. In NYC, many real estate agents will represent the renter for a service fee, just like they represent the buyer in a purchase transaction. It supplements the income that comes from the higher-value (but much less regular) money that they get from sales.
There are agencies that make most of their business in this area, and generally have better rate structures (typically a month or two of rent, flat rate). Others want to charge you on a percentage basis, which you can negotiate, and obviously changes the incentive structure. All of these are "expensive" (relative to free), but worth it depending on how you value your time.
Other people are (rightly) noting that there is risk of getting a scammy real estate agent, but that's true of any service. You have to look at reviews and trust your gut. Good real estate agents are motivated to do the right thing for you, because they see it as a marketing channel into larger sales in the future. Bad real estate agents are in it only for the immediate cash. These people behave in such obviously different ways that they're pretty easy to discriminate, in practice.
If the agent is in any way getting compensated when you sign a lease, or as a percentage of the rent, or anything like that, they are not working for you.
The only way they might be working for you is if you are paying them an hourly rate or per showing or something like that.
If you're saying that the historical "broker's fee" when renting in a place like NYC doesn't benefit you, then I agree. That broker represents the landlord, not you, and you're just being charged to make up the cost.
If you're saying that someone doesn't "work for you" unless you pay them a flat or hourly rate, then I disagree completely. All incentive structures have problems. Someone who works on contingency is incentivized to get you to sign. If they take a percentage cut, of course, they want you to sign the most expensive thing they can. Someone who works on a flat-rate basis is incentivized to do as little custom work as possible. Someone who works on an hourly basis is incentivized to drag out the proceedings. There is no perfect system.
Most of these agreements are structured where the fee is paid by you, after signing a lease, as a function of the monthly/annual rental price. The agent works for you, you just don't pay up front or a fixed cost or on an hourly basis. Does this change the incentive structure? Yes. Does it mean that they don't provide you a benefit? No. You just have to be aware of the incentives.
Just like when buying a house, an apartment broker works on behalf of the landlord or the tenant. They charge a fee which is a percentage of a year’s rent. It is negotiable.
Plugging my site, but only because it addresses some of your core complaints, and may be helpful.
Dwellsy.com has the largest inventory of any rental property site out there, is aiming for zero fraud (and has quite a few features implemented to prevent against it,) parses descriptions to more accurately represent the actual amenities (it's decent, but not perfect) and doesn't clump units in apartment buildings into one ranged price listing, as we prefer to have the units represented as individual listings.
Hopefully it'll help, but feel free to reach out directly if you run into issues.
You didn't even mention all the straight scam advertisements you have to wade through. Next time I move I am going to hire a PA to handle searching and making viewing appointments.
In really competitive markets it seems like you either get lucky, or you basically know someone in the area who can hook you by word-of-mouth.
In the ancient "before times" there were also actual rental agencies where a human would look for an appropriate apartment for you. I didn't know about them, but a local NorCal person turned me on and it was the only way to find things. They followed all of the different local listings daily and knew the area fairly well (e.g. how parking, transport, different commutes would be). It was $25/mo and ~$100/yr for them to go through the listings call the renter, if necessary. They usually also collected $100 fee, if they hooked you up (though I'm sure you could work around it) with one of their (usually several times a week email of 5-10) available recommendations and usually a $50+ fee for running a "credit check". The reason it seemed to work was that the renters trusted the rental agency to only send them real referrals, and the agency knew all about scammers and fake listings, and daily follow-up. It only worked due to the relatively small area and and high volume. Each agent was looking for ~50-100 people during a month and making several $k/mo themselves, filling them typically in a few months.
Like travel agents used to be an actual thing... but now you either do it yourself on-line, or you're going on some giant family cruise thing.
Asymmetric information works in favor of the landlord so I doubt it'll ever get better barring some legislation -- which we know will also never happen because most voters and politicians are also landlords
I think it depends on a few factors. The culture where you're searching. The laws. And the economy. And the available services. Where I'm from the law works pretty well, and there's really only one de-facto site that most people use for selling their pad, though Facebook has tried to take over some of the competition. I don't think you can get away from the stress of viewings, but IMHO it sounds like it's a lot more difficult to search where you're situated.
When I moved to San Francisco in 2001, there was a store front around Market and Church that was an agency with apartment listings printed out on the windows like a real estate office. They ran my credit check, made me a packet with copies of it for each landlord, and set me up with appointments to see three units, one of which I rented.
I think there was a fee in the $50 to $75 range, but it included the credit check and access to their listings.
> With all of the modernization and streamlining that has happened in other parts of our lives, apartment hunting remains unstructured, messy, frustrating, stressful.
If multiple candidates compete over the same apt, no searching method in the world will resolve the issue. Either you have high supply of apts, or a super intelligent city software that matches people to available apartments based on their socioeconomic and professional profiles.
i don't share this experience at all. i've used zillow and apartments.com recently (found a place through apartments.com about a year ago and just found one through zillow a few weeks ago) and had no issues. the filter options work how you'd expect, and i wasn't put in touch with any middle men - it was either the private property owner or the apartment management company who handled the leasing directly.
Had an excellent experience finding my last two places from Zillow - one in SD and one in Denver. Truly wonderfully luxury 1 beds at below market prices from landlords who had this as their only secondary property.
Tried the same thing with NYC and things went nowhere fast.
I had pretty good luck with Zillow and Craigslist all the times I've tried
There were a few outright lying apartment management companies, but I managed to look at some nice places being rented out directly by their owners and settled on a good one
You gotta get creative. Walking down the street I met this one guy who's like 70, eccentric, military vet, owns the house, and rents it out individual rooms for $550/month. He's paid off the mortgage, just needs to pay taxes, and he's clearly making a profit since he's retired and constantly has new projects working on the house. For context that's 1/3 asking price out here.
That's more of a banal aphorism then a piece of concrete actionable advice when trying to find a place to live. Expecting to get a rent below market rate by arbitrarily walking into somebody on the street is just...
It would also be nice if we would stop voting for politicians that promise policies that limit supply - either directly and explicitly such as zoning laws, or as a consequence like rent control.
To which court are you going to appeal for redress when those rights are not upheld? The UN declaration of human rights is nothing but an aspirational fiction.
[In 1917] the government spearheaded great changes in people’s lives, starting with housing. The Soviets nationalized private property and started the so-called “уплотнение” (housing compaction), giving rooms to working-class families in large central apartments that used to belong to [previous owners]. Note that renters were “compacted” too. This project officially laid the groundwork for the often bizarre kommunalka housing - a resilient symbol of the Soviet past.https://www.rbth.com/lifestyle/331621-lenin-putin-soviet-rus...
So you are having problems looking for a 2bd apartment for a family of 3? What about one room in an apartment with a bath shared by 4 other families?
In the past I've done something like this: pick the neighborhood you're interested in. Go there and walk or drive around looking for "For Rent" signs. Look at public bulletin boards in markets and laundromats. Start making calls.
That worked in the 1990s. Not sure it's viable today.
Even if the population doesn't change, if more purchase power moves in (e.g. struggling artists move out, software developers move in) the market will go up.
Also, Berlin was an odd place compared to other cities in Europe, when it comes to housing. Looking from the outside it seems like it was due for an upwards correction after 3 decades of pretty low rents since the fall of The Wall.
Rents are capped at the amount people can afford. If you earn €1000 a month as an artist, you can't pay €1500 a month in rent, no matter what supply and demand curves say.
Say there's a supply on 100 apartments and a demand from 120 artists. 100 will pay €1k per month, and 20 will have to leave the city. Landlords can't increase to €1050 a month because nobody can afford that.
If demand is higher than supply, rents will increase, but if the demand is entirely from people earning €1000 a month those rents won't increase past €1000 a month.
However you have a demographic on €5000 a month move in, they can afford higher rents. The same 100 apartments exist, and there's a demand from 120 people, but now those people are earning €5k a month. Rents thus shoot up to take all the money that can be spent on rent, because the alternative is moving out of the city and thus not being able to earn the €5k/month.
The people owning the property do well in this situation, and thus don't want supply to increase to the points that there is more supply than demand
I have friends who used to let their Berlin flat. They stopped doing it after the apartment-seizure law passed [1]. Better to lose the income than lose the property (or be caught fighting an illegal law).
What you're referencing is not a law it's a referendum that most likely will never be adopted as it's illegal. On top of that, how does that referendum affect your friends unless they run a giant business with thousands of apartments (to be exact 3000 or more)? And at that scale it's certainly cheaper to fight in court.
> is not a law it's a referendum that most likely will never be adopted as it's illegal
It reflects a political environment that makes renting risky. And illegal laws can still cause hardship while they're being stayed and struck down. The corporate landlords can afford to fight. Single-home landlords cannot. This time the referendum constrained itself to corporate landlords; next time it may not.
From the outside, this sounds scary. But if you look closer how germany works, forced selling to the state or city (at the current market price) is quite often happening in building infrastructure like streets, autobahn, railways or airports. If there is an overwhelming public interest, the rights of owners can be limited.
And btw: it's stated in the constitution, that ownership introduces also responsibility to the society.
> If there is an overwhelming public interest, the rights of owners can be limited
Eminent domain exists everywhere. Berlin’s politics are more extreme than the norm. Most homeowners would pay a low single-digit percent of their home value to insure against expropriation in that environment. Taking a property off the rental market is analogous; sacrificing a cash flow to safeguard the investment.
We can go back and forth on theory all day. I’m m reflecting what people I know are doing. Being flippant with (or even demonising) owners’ interests circles back into affordability crises worldwide, from Berlin to San Francisco.
Same in San Francisco. My old landlord never rented out my old place when I left (3 unit building). She was getting close to retirement and had gotten advice that the building would fetch 2-3x if fully unoccupied. So she was just waiting since giving up $40,000/yr in rent for a $500,000 higher pay off is an easy trade off.
People forget that you can always “price in” regulatory changes. Then you just see which path maximizes profit. What may seem ridiculous (leaving a unit empty) actually makes a lot of sense based on the rules.
The regulation gives absolutely insane incentives. In my case (old contract in hip mitte area), it was easier to keep it and for my gf to rent another apartment on the outskirt where her work is as she can deduct the full rent from tax. Effective price for renting two apartments is lower than one in between the center and suburb.
I don't see how that has anything to do with the rent control regulation? What would your expectation be without that regulation?
Doppelte Haushaltsführung is a tax relief for people needing a 2nd place close to work. Also the work of your gf needs to be at least 1 hour away from the main apartment [1], she has to pay at least 10% of the cost of the main apartment to be allowed to deduct it (and maybe more requirements that I'm forgetting right now). She also has to pay 15% additional Zweitwohnungssteuer on the rent of the 2nd apartment.
Rent control simply locks in a set of winners when it is implemented (which for the most part stays static and doesn’t apply to new demand), the government then has to promise that rent control won’t be implemented on new supply so developers, businesses, and private landlords just don’t give up on the rental market completely.
I don’t see rent control as very useful, but I doubt it is very relevant to the current housing shortages throughout the cities that have it either (given that the same problems apply to cites that don’t have it).
Rent control needs means testing. It suppresses mobility when the googler making $300k doesn’t move to a place they’d rather have because their current unit is so far below market rate, which hurts the minimum wage worker who could have lived there.
Its a bit tough to talk about nothing changing in 30 years and not talking about unification, which makes it a completely different market. After the war East and West overbuilt Berlin to make it a showcase where economically it didn't attract a lot of people.
And a ton more work to file a college application and a smaller group of applicants. Zillow apartment applications is just clicking an "I'm interested" button.
I guess I would have expected a few top Ivies to have significantly lower acceptance rates on the grounds that one could imagine a lot of average-ish students putting one of those schools down just in case they hit the lottery.
But actually, the top Ivies, MIT, West Point, top music schools, and a couple of schools I've never even heard of (!) are all in the 5 to 10% admissions range.
> I guess I would have expected a few top Ivies to have significantly lower acceptance rates on the grounds that one could imagine a lot of average-ish students putting one of those schools down just in case they hit the lottery.
There are many, many of these applicants who have roughly a 0% chance of admission.
I am not saying that elite schools are easy to get in (they are not), but someone who is a strong all-around applicant will most likely get into at least one elite school.
Most applicants are challenged by the fact that they only have one or two strong points (like grades and scores), and those “strong points” often aren’t even that strong. Athletics, community involvement, leadership, above-and-beyond academics, etc. go a long way.
These 1:1200 are exceptional anecdotes that happen if you have some idealistic landlord that doesn't max out the allowed rent but rents it out at some price point that they consider reasonable, which is perhaps the rent from 20 years ago. Of course they get flooded with applications.
If they'd max out, they'd still get multiple applications, but maybe 10-30 of those, half of them from people with bad credit rating.
The pandemic allowed remote work to flourish. Why are people still crowding the cities? And of all the places, specially Berlin. Not even that great of a city.
A lot of my friends fled the city.
Because in most of Europe (Austria for me) it didn't flourish and not every job can be done remotely as not everything IT related is web development and in many companies management did not give ownership and resources to maintain remote work IT infrastructure to anyone, so everything is chaotic and crumbling to bits now that things are opening up and most people are back in the offices.
Few companies here are keeping 100% remote once the vaccine mandate came through and things started opening up this spring, and now the new standard is various hybrid work variants where you get 1-3 days per week remote and the rest in the office, or one week remote, one in the office, without any logic or arguments behind these choices, just "because management said so" and "this is how we do it here".
>Why are people still crowding the cities?
Because that's where the amenities and the networking opportunities still are. Universities, clubs, bars, music halls, swimming pools, great restaurants, bouldering halls and gyms, music and jazz bars and basically anything to do with art.
Oh, and most importantly, dating. It's way more difficult to meet someone when you live alone in the sticks VS being surrounded by people your own age sharing similar interests.
Prior to the automobile era, we were all pedestrians. A small town was a town you could walk the length of. Europe is dotted with old villages denser than contemporary North American cities.
While true, I suspect most of the people who are looking for walkability want something more than a small town with a pub, a small market, and maybe a couple other stores/cafes/etc. I've stayed in many of those small towns and they can be very pleasant--but there's not a whole lot within easy walking distance.
My partner and I were looking for a walkable urban neighborhood but ultimately ended up in a small town with a pub, market, cafe, store... and a train to the big city.
Couldn't be happier, but I wouldn't recommend it to someone who is single.
Likewise, chose a cute little New England town when I realized that the benefits of living in a city are mostly... drunks, drugs, and concerts that are neat, but I'm unlikely to attend.
There's culture and museums and lots of different food in cities, but all I really need is a small smattering of decent establishments in exactly the vein you just mentioned. And of course a good environment for working from home.
Same. Given our requirements and price range we would not be living within walking distance of the city museums or cultural amenities anyway, and we would be about as far a walk from the closest supermarket or cafe as we are now.
I think there is something to be said for New England towns where the design of the town center predates the automobile and is therefore pleasantly dense and walkable. They just don't seem to make towns like that any more.
If you can be 100% remote and don't need the transit options to the big city then just a little further out you can find charming towns for even less.
Depending upon circumstances, there's certainly something to be said for spending money/effort to go into a city now and then rather than spending an ongoing premium for living there day to day which may even have certain negatives.
> The pandemic allowed remote work to flourish. Why are people still crowding the cities?
Because the top employers by number of people are all about schlepping shit from A (where population isn't) to B (where population is), and that can't be done remotely. Then you have the manufacturers, and that can't be done remotely either.
Most jobs cannot be done solely using a computer keyboard.
Portland, OR solved that problem by making landlords accept the first offer which matches the stated requirements. Which leads to some… interesting incentives :p
Seattle has this also. It leads to very stringent requirements, with little room for flexibility, you also have to the first qualified person to respond to an ad. It has also led to a lot of rental supply not being advertised, relying on personal connections to find tenants instead.
It's also difficult to nail landlords on, so it goes largely unenforced (at least in Seattle). Most renters aren't going to be able to afford to take a landlord to court over an apartment they didn't get, they're just going to keep hunting because they're usually on a strict timeframe to move.
Most landlords are sophisticated enough to follow the rules, but it does affect corporate landlords (who always advertise and whose employees can lose their jobs if they don't follow the rules) more than private ones.
As I am also looking for a flat in Berlin I can only say: At least you got a response. It's nearly impossible to find anything below 20€/m² cold rent and some people are trying to (illegally) sell on old, cheap contracts for 6000€ and more.
Can I ask what is cold rent here, does that does not include utilities or similar? Also how does the second illegal thing work, is that subletting basically?
> Also how does the second illegal thing work, is that subletting basically?
I'm not sure if this is what gp meant, but:
In some cases the current tenant (who is looking to move out) offers you the landlord's contact details if you agree to buy their kitchen for way too much money. This works because for a landlord looking for new tenants is a lot of hassle so they're likely to take you if you contact them.
Can't answer the second question. As for the first one: in Germany, you pay rent plus an advance for certain utilities (heating, cold and hot water, trash collection, general maintenance of common areas etc.). The rent without the advance is commonly called "cold" rent (Kaltmiete). The advance is variable based on actual usage. So the common way to compare rents is by excluding that.
I'm german and i can save roughly 15k€ per year (and that's a lot in german context). A freestanding house in this area (think Munich) costs a little bit north of 1.5Mio. Do the math.
You pretty much have to buy an apartment in Berlin, that's been the case for the past 10 years or so. Landlords took them off the market and put them up for sale.
You might still be able to find something decent for around 100k euro, I'm not sure.
Because a law called "Mietpreisbremse" makes it entirely unattractive to rent out. That's a strict regulation on how much rent you are allowed to take. And that's just the direct financial aspect. Once a tenant lives in your place, it's really hard to get rid of them. Even if they act like total scumbags, trash the place, disturb the neighbors, pay their rent late.
No reasonable investor finds it attractive to build new rentals in German cities. Result is a shortage. And because of the shortage, rents skyrocket. And because of that, you have "Mietpreisbremse". And because of that, there is even less rentals and even more shortage.
If only anybody would understand that you can't fight the market like that.
Berlin politics also seriously discuss seizing landlords' properties [1]. It's stupid. It's illegal. But it makes the potential downside of renting out a home the value of the property. (Or the cost of defending against an illegal law.)
It's not illegal. It's in the German constitution. Land is seized all the time, too. E. g. when the autobahn or mining operations are expanded. Nobody bats an eye there. It's only when corporations shift from being beneficiaries of disappropriation to being the target of such laws that the whole world all of a sudden is outraged.
First of all: the problem at the core is that Berlin is pretty much built out, there is not much space any more to expand to. Yes, skyscrapers could be a solution, but everyone seriously thinking about this gets invited to look at Frankfurt am Main and then forget about the idea because that city is simply hideous. Skyscrapers are very unwelcome in Germany because of that example, not to mention they need some seriously solid ground to make a proper foundation... and Berlin doesn't have that.
Second: that law was enacted because people were fed up with rents going up all the time - including absolutely ridiculous practices like "kalte Entmietung" (aka having construction crews in allll the time to drive out tenants) or charging ridiculous prices and raising rents while the property devolves ever more because the profits from the raised rents are clearly being siphoned off instead of being used to keep up maintenance.
Third: People were especially pissed off about the large landlords (Vonovia etc.) which bought their stock at fire-sale prices (literally - the 90s CDU government under Diepgen had left a veritable banking scandal and the Senate was forced to sell everything it had in the early 2000s for cheap) and now raised rent like there was no tomorrow. It was naked and utter greed that turned the public opinion so bad.
> Once a tenant lives in your place, it's really hard to get rid of them. Even if they act like total scumbags, trash the place, disturb the neighbors, pay their rent late.
Paying rent persistently late is grounds for termination, it's easy to grab an eviction order at a court. Disturbing neighbors... well let's keep it at "having illegal hotels aka AirBnBs as neighbors is way worse".
> No reasonable investor finds it attractive to build new rentals in German cities. Result is a shortage.
People build like mad wherever they still can and that is the problem. Land values in urban areas exploded - Munich for example had land values double in four years [1] - and the result is that because the land is so expensive, the resulting rental price are going to be exorbitantly expensive as well.
In the end the solution is not the market, as the market will over time only serve to evict the weakest of society - and as a result, as we have seen in Berlin and with the Yellow Vests in France, this has the potential to erode trust in democracy itself.
The solution is to revitalize the rural areas. Public transit and high-speed Internet access are the key points where the suburban and rural areas across the West have the worst issues - fix that and people will actively move towards these areas instead of all being forced to compete for urban quarters!
Please don't. That's not how a civilized debate works.
> First of all: the problem at the core is that Berlin is pretty much built out
Sorry, when comparing to other cities world-wide, that's hard to buy. Places like San Francisco, ok, maybe. Ocean in 3 of 4 directions. Vancouver, ok, maybe. Mountains on 2 sides, ocean on one side, border to US on one side. But Berlin? It's all flat, fields and tree plantations ("forest") all around. Even just adding a floor or two here and there would be fine. There are many areas that could get quite some more density with 5-6 floor houses. But then NIMBY hits. Easier to be angry at the "rich landlords".
> Second: that law was enacted because people were fed up with rents going up all the time
I'm trying to tell you that that's not solving the problem. High rents are just a symptom. The root cause is a mismatch of supply and demand. It's as if you are trying to legislate away gravity. That won't work in the long run. Either you need to increase supply or decrease demand. I can see that you argue that rural areas should be strengthened, but I fail to see how that would suffice. People want to live in Berlin.
> Third: People were especially pissed off about the large landlords (Vonovia etc.) which bought their stock at fire-sale prices
The people should be pissed off at the politicians that sold them the stock. It was rational to buy it. Selling out was not. And that wasn't just CDU. Even the SPD+Linke senate did similar things later on. Yet, everybody gets reelected all the time. (Not defending CDU here, they are just as guilty.)
Evicting tenants is not easy. Even with an order, you can't simply dump tenants out on the street if they have no other home to go to. And there are ways to claim extenuating circumstances. We've had our share of tenants that trashed their places badly while not paying. And we didn't get a single cent of what they owed us for rent or damages. This wasn't in Berlin, but I don't even know how much money we lost. Renting out apartments is a risky proposition in this country.
The prices mainly went up so fast because interest rates in Germany went to 0 and below. Get the central bank rate up to 3% and you’ll see property values crash.
But yes I agree, make living outside cities worthwhile and attractive.
Berlin pretty much built out? The city has a huge "hole" in the center - Tempelhofer Feld. 355 hectares of land that at some point it was decided to leave untouched. Complete waste of space that could be used to build thousands of units.
Anyone who lived in Berlin in the early 2000s can attest to the bizarre real estate situation back then.
The idea that the Tempelhofer Feld would have become anything but a mall with adjacent luxury apartments is bonkers.
The German constitutional court had just forced Berlin to sell of most of its housing infrastructure and the mayor and his confidants clearly were either unwilling or incapable of enforcing laws that would have ensured some form of public control and affordable housing options.
This was the time when people were offering tours putting all these vacant buildings on display that were left there waiting for the rent moratorium to end. When corporations freely redesignated large and central areas to the next biggest mall and luxury apartment project.
You can't have it all. No new areas for housing and no higher buildings and keep real estate prices and rents at a certain level and more people move into the city. That's impossible. Something must give.
Ideally it's a compromise, everything gives a little. A few more areas for housing, a little higher buildings, a bit higher prices, a bit more attractive rural areas.
Manhatten is one of the densest populated urban centers in the world. Nearly no other green areas, even street trees are a rare sight. Everything is concrete. Berlin on the other hand is very green. Parks, trees on the street, Tiergarten, Grunewald, and yes, Tempelhofer Feld. You could build housing for tens of thousands of people while still keeping the area green with lots of trees and parks for people to relax in.
Just keeping it a huge empty space and at the same time complain about high rents is ridiculous. This is not solved by complaining about money hungry investors. It's the supply-vs-demand ratio that's the problem. Not greed. (It may play a role, but would not have much of a chance without the underlying issue.) Of course you can keep it "for climate reasons". But that comes with a price and you need to live with those consequences. "Mietpreisbremse" is not going to solve any of those.
Dunno, it doesn't really make sense to me. Sale prices were around 20 years' of rent, so it seems like a great asset to hold. I assume they just wanted to cash out on rising prices.
A few years back, I was looking for a rental in a German city with a similarly hot market. One place I looked at was not very modern but had quite low rent, so there were 300 applicants and I didn't even try. Another place was put on the market 3 days prior, there were 5 people looking at it with me, 3 of them applied (including me), and I got it. It was more expensive though (not insane, mind you) and very modern, recently renovated.
So, of course there will be crazy places, but if you only shoot for the ones with top demand, then of course you can come with anecdotes that sound insane.
Interesting article: I did something similar but for Zürich (finding a low-price, but good apartment there is almost impossible).
I did follow a different approach though, and reverse engineered the APIs of the most common rental listing websites here.
I didn't publish the code (or better said, I didn't make it public), but it's similar to yours. Instead of SQLite I used a PostGis database where I stored the apartments with their point to point public transport distance from my office (pre-covid search). I did it in Rust.
Whilst the application is not available to the public (yet), all of the libraries I've created are now available.
I should have both a Rust and a Golang version for most / all of them.
On a side note, I did a similar project (scraper + visualizer / search) for finding a job for my girlfriend by scraping LinkedIn, Xing and a couple of other local job posting websites.
> I didn't publish the code (or better said, I didn't make it public), but it's similar to yours. Instead of SQLite I used a PostGis database where I stored the apartments with their point to point public transport distance from my office (pre-covid search). I did it in Rust.
How did you convert an address to long/lat for postgis?
I was lucky, the data also included lat/lng and the address for most (if not all) the entries.
If I would have been confronted with such a situation though, I would have probably used some Geocoding API like Google Maps's geocoding [0] or OSM's Nominatim [1].
If Zurich is anything like Lausanne, it has been that way for awhile. I had a heck of a time finding a place in 2005, the property management companies and landlords were all very picky even if rents were high as well.
Found a really cheap apartment, although a bit small, in a very convenient location.
The apartment was also new. I'd say a very good price/value ratio.
I now moved away from Zurich, but that apartment was very convenient. When I organized the viewing, ~100 people were interested and ~30 visited the apartment.
It frustrates me that there are so many "information programs" like these that are computationally easy to solve but rely on the free availability of information. Sadly, companies like athome.lu have no interest in sharing this data and so here we are in 2022, still having to manually search multiple sites.
In Internet olden times, we dreamed about automated agents that would run searches on your behalf, scour the web to compare prices and hunt down things you were interested in. But user-centric fantasies like this don't make fortunes, so walled gardens is what we ended up with.
athome.lu is specifically selling the 'search all listings' functionality, though. If they had to make this data available they wouldn't be interested in making the data in the first place, since it'd basically be charity work for whoever else is interested in throwing that data into elasticsearch and creating a frontend for it whilst undercutting athome.
All it would do is create a race to the bottom. Athome seems to be a data aggregator of many listing sites; I can see open data on the actual apartment lister’s end being useful (ie. Apartments.com) but athome is the ones doing the aggregation, so if they had to turn the data over they just wouldn’t create that data in the first place since there would be no revenue stream for doing so.
In highly competitive markets the good apartments don't even make it to the websites that aggregate listings before someone takes them.
The ones that do are either very expensive or will have 20+ people showing up the next day (sometimes the same day) for a visit. It's crazy and while this article is very neat I'm not sure a bot that scrapes listing in batch is the answer.
We only got ours because we were very nice to the agent after losing out on an apartment (thanked her, asked her to keep us in mind if anything else comes up etc).
A week later she phoned us before she listed a new one, and we went to see it 3 hours later and took it.
It goes to show the human aspect still matters, a lot. That one email thanking her was probably worth way way more than eg hours spent on a bot to automate applications.
Many apartments also don’t even make it to an agent because people have friends they promise it to and refer them to the landlord.
A long time ago I knew someone who's parents found them an excellent apartment at a great price in Boston. Great neighborhood, 2BR, nicely kept, off-street parking, you name it. Few years later they had to move since the owner was selling the place, and went on an apartment search by themselves (using an agency). Could not find anything anywhere near nice at that price point, and settled on something a lot more expensive.
I decided that their parents must have tipped/bribed the original agent to get at that listing (since none of these ever made it to any search site, not that there were many at that time).
I don’t know the story of course, but apartments are one of the largest profit growth centers for capital firms; most metros have experienced >50% rent growth in the past decade[0], for instance. Chances are a renter will never see a rent decrease unless they downgrade in a key area (specs, amenities, location) or are moving in very short intervals.
At least OP had the full address of those apartment on that website, so he could plot them on the map.
Here in Austria, most leeches, sorry, realtors, avoid putting the street in the ad to prevent you from trying to contact the property directly and bypassing their fees, so apartment hunting is even more of a headache.
When I searched for my apartment, I was able to reverse engineer the home's address sometimes when the general vicinity was specified, then looking at the pictures, and comparing them with Google street view. This was a manual process however and only worked if enough outside views were available. Despite the time effort, you still spent less time on it per house than making an appointment for a viewing, getting there, and looking at it. If Google allows you to download the street views for an entire city, one might build an ML solution to bring up candidates at least.
Similarly, in the city where I studied for university, I immediately knew where the house was located just from outside pictures of it in the ad, at least for the districts of the city I was familiar with.
I am looking for an apartment in eastern Europe. I was shocked to see listings with the windows covered in Microsoft Paint. I was able to find the location in some instances by finding the same property in other listings (the images were still slightly altered). This is the amateur move, what I see more often is image alteration so that the light coming through the windows makes it impossible to see what is outside. The same apartment is usually posted by 5-10 agencies, pictures and description almost identical, with different location. You get different responses depending who you call. I hope to end my suffering this year and buy something.
Oh it is definitely possible to avoid this. It's kind of an arms race. The overexposed window effect might be reversed with computational photography for example. Maybe one can just lower brightness in an image editor and discover that there is a residual of the outside left. You might use "looking around a corner" like tricks on white walls, etc etc.
I use a similar process to look at interesting job listings sent to me unsolicited by recruiters. You can anonymize the company but it only takes a few unique phrases to locate the posting with Google.
Oh, around here (Spain) usually they don't put the address in the ad so you can't cross check the information with the national registry and find that the apartment is 20% smaller than claimed.
I've searched for houses in multiple places around the world and the unwillingness to provide the full addresses is often due to agent competition. More precise whether the seller has one or more selling agents.
In places like The Netherlands, a seller generally works with one seller's agent only, and there's no incentive to hide full addresses. Funda.nl has a huge amount of detail, from high quality photos, to floor plans.
On the contrary, in Singapore, a seller often has multiple agents. The agents compete with each other and would rather not share any information at all publicly to avoid other agents using their information/photos, or buyer's bypassing them. It leads to propertyguru.com.sg with no addresses, bad photos, all photos watermarked with the agent pictures. As little information as possible, just enough to draw you into an in-person visit.
There are genuine privacy concerns at play here, for example there may be existing tenants/owners who don't want people randomly knocking on the door because someone saw an ad online. Or getting a bunch of junk mail because there is an extra data point about the property.
I guess the variety of concerns will vary from country to country depending on local laws / existing open data though.
Yeah but this is a "the customer isn't the user" situation. The tenant wants and deserves privacy, and it's their privacy that can be compromised by the listing.
The owner doesn't really have a reason to care, I don't think, and may even be incentivized in some situations to reveal information the tenants would rather they not.
What information? “This apartment exists at LOCATION and will be available for rent at DATE?” And you think the tenant is entitled to stop the world from knowing that because the tenant is residing at that location before DATE?
Goodness. I should give the First Amendment a hug.
I'm still not sure renters really have a reasonable expectation of privacy when the place they are living in is listed for rent or sale by someone else.
In most places renters are only granted 24 hours notice to have complete strangers come walk through their living spaces while the space is shown off. They have no real right to refuse to allow that.
If people start knocking on their door asking to come look at the place they are free to refuse, or ignore the doorbell or anything else. Of course they have a freedom against being harassed, so if people persist they can get police involved. That seems like a separate issue to me from having their address posted online in a rental listing.
>I'm still not sure renters really have a reasonable expectation of privacy when the place they are living in is listed for rent or sale by someone else
In some EU countries they do. The landlord need to ask permission to have visitors at your place and the tenant has the right to refuse.
The tenant has the right to refuse to have the home shown when they are moving out and the landlord is trying to find new tenants? How about maintenance workers fixing appliances or something? That seems pretty hard to believe.
Are you sure the tenant isn't just allowed to refuse "a time slot" but also must offer alternatives?
Outside of necessary repairs or maintenance, you can legally refuse unwanted visits from the landlord to the apartment you are paying rent for.
Story time: A couple of years ago, I was looking to buy an apartment and saw one spacious apartment centrally located that was way below market price but had no pictures in the ad. I called the agent and asked when can I visit as soon as possible since there's no pictures. Agent said the old lady currently renting it does not allow visitors inside, that's why there's no pictures. I asked how in the world can you buy an apartment without seeing it? Agent said that's why the owner is selling it below market price. And the charry on top, if you do buy it, you'll be forced by law to keep the current tenant who's paying frozen rent, not adjusted for the past 20 years, until she either decides to levees voluntarily or dies. That's why the owner had to sell below market.
sorry but every part of this is good? you can't go in someone's home if they don't want you to. you can't kick someone out of their home if they don't want to leave.
someone being this hostile about it is rare but it sometimes happening is a necessary consequence of these rights.
anyway though in most places our protections are not this extreme. you can refuse a specific visit but have to provide a time when they can come later.
About ten years ago I was attending San Jose State University and really needed to find a 1 bedroom apartment at a decent price (lol) in the downtown SJ area since I would no longer be living in the dorms.
I tried a few different ways of finding an apartment, but in the end it was one case where the old phrase "it's not what you know, it's who you know" worked in my favor.
During my freshman year I had found and been working a job on campus in one of the department offices (side note: if you're a student, you're best off trying to find a student assistant opening in a department office; at least at SJSU, most paid above minimum wage, were pretty lax and could be used for some study time, and you have so many opportunities to make connections with professionals).
One of my coworkers there had a family and they were living in one of these small ~8 unit apartment buildings downtown. It was a privately-owned building, not one of these newer, fancy conglomerate ones.
She connected me with the management/owner and long story short, I got a decently livable place with rent control for about 1/3 of market rate due to being a lower income student and taking a unit that wasn't recently refurbished (old carpet, old paint, etc.).
I did something similar 10 years ago during my move to SF. I needed to find an apartment within two weeks, so I built a crawler to do complex filtering and notify me immediately.
In my case, the tool was ultimately useless because I discovered the way SF housing worked for nice apartments was 1) Openings are posted on Friday 2) Saturday, open house collects applications 3) Deal is signed on Sunday and post taken down. Anything my crawler picked up was stuff that was overpriced or in bad condition. For all my sophistication, the dataset was just the bad stuff.
Apartment hunting is a great example of needing to know the underlying behaviors of your dataset before you use it to make decisions.
Ciao, I had the same problem moving to Italy, specifically Bologna.. Super student rich town so apartments are gone fast, I setup a crawler of the usual sites but one or two had live updates for locations that would be published through their site's websocket, after a few hours I had the rasPi subscribed and could get notifications to me as soon as they were available, bypassing the need to sit on the website.
It was a pain after for bureaucracy but that was after landing an apartment.
Interesting details brought up here about location specifics.
At least in Zurich most ads contain an address. I fed it to google maps api to get the time by bike to my workplace which gave us some cheaper options we wouldn't have considered before. We also ended up taking/getting the one with the worst pictures.
Imagine someone would scrape all platforms for your city of choice and aggregate those with some value added and maybe automatically apply with your personal documents to make sure your application gets in first even when you're busy, what would it be worth to you? Despite local differences it might still scale.
Ah... this brings back memories. A few years ago I wrote this article https://funnybretzel.com/datamining-a-flat-in-munich/ when I was searching a flat in Munich.
Some time has passed since then and meanwhile I've enhanced the process quite a lot, just like OP I'm currently searching for a place (a house) and I'm using quite the little bot farm to massively automate things.
I'll post the results if it converges to something useful. Meanwhile, all the luck to OP and excellent write-up! :)
This is a great project and reminds me of a small project I put together when searching for an off-grid property. Price, property size, and distance to home were key constraints and using something like follium would have made it much nicer than the tabular format I used. Thanks for sharing!
I wish all apartment listing included a floor plan. I could easily reject 90% of apartments without having to go personally look at it if they's show me a floor plan. It's a little surprising to me they aren't more common.
In Japan they're often missing other things like lots of pictures but at least the number 1 requirement to list an apartment is a floor plan.
I wish Craigslist would require one or somehow strongly suggest that the person offering the apartment would get better matches and have less of their own time wasted if they'd add a floor plan.
I also wish they'd add some examples of good and bad pictures. So many people take pictures with their phone and the field of view is so small and lighting so bad the pictures are basically useless. I'm not saying they need pro-equipment but my guess is there are at least some guidelines that would be helpful. How about no portrait mode pics as just one example that would likely lead to more useful images.
In Japan they're also often missing the "willing to rent to foreigners" flag. Using aggregate sites like Suumo to find places you like ends up being a waste of time when so many places won't even consider you.
Tangential. Information on the internet is often intentionally incomplete or difficult to filter. For instance, it is impossible to filter apps in the Google Play store by the presence of ads or in-app purchases within them. Why? Because Google wants it that way. In order to perform such a filtered search, one needs to create a custom crawler, then parse the data and then filter them.
If anything, with all the ads and SEO games taking place, searching for the relevant information has become _more_ difficult than in the early days of Google.
It is 2022 and we still do not have a universal search system which would allow everyone to quickly and efficiently perform more sophisticated searches. The most skilled and determined people are capable of doing so by creating various single-purpose data collection and data mining tools themselves. But even for them it takes a non-trivial time and effort to produce something usable. There is nothing out there (that I know of) which would be easily applicable to everyday needs.
Many of the public search boxes are backed by elasticsearch or other similar “recommendation” tools which are optimized for speed and point-someone-in-the-right-direction accuracy.
Since we no longer have exact text matching or many of the advanced operators (such as `not`), I’d argue that for sophisticated searches search boxes have actually gotten significantly worse.
The OP points the way, I think, to what will happen in practice. Three trends will meet: 1, more apps live in the browser, 2, search programs that combine data in the browser get formalized/paramterized, 3, user friendly methods arise to make writing ad hoc search programs very easy (e.g. voice driven, context aware AI).
I love blogs like this. Showing the dirty side of data acquisition with an end result of useable software, that hacks around provider limitations to give something way better. We’ll done.
Despite all the fancy new websites that crop up every couple of years in the apartment rental space, imo the best place to find an apartment remains craigslist. I've only ever leased spaces through there. For the mom and pop landlords who you really want as your landlord, they might get a new client every 10 years. 10 years ago, they were using craiglist, and it works exactly the same as today, so that's what they continue to use. If it gets them a tenant its good enough to not bother learning some other new website. For the large corporolandlords who have those bougie amenities that perhaps you crave, they will be posting on all the major websites including craigslist anyhow, so you might as well use craigslist for that use case too if only to use a less bloated website.
Actual title: Scraping apartment offers to show them on a map
(since the current title just conveys some general topic of apartment need, with no indication if it's about "my partner died and it's interesting to notice how large a whole house is" or something)
I appreciate the talent and ingenuity behind this apartment hunting project.
I just wonder if OP ever thought of contacting a broker? I would have done that before I wrote a single line of code.
Scraping all this data is really great, but a human broker is going to know the local area and the other human nuances of apartment searching in the local area.
Plus, brokers already have access to their own software that’s different than what’s on the public Internet, along with human connections. Why reinvent the wheel?
My last apartment came up as an option before it even hit any sort of public property advertisement platform, and that was thanks to our broker working directly with the landlord.
In my experience finding a broker who actually does broker work is hard enough for housing you’ll buy. Even harder for just renting. The money isn’t there for them to do much more than setup a few saved searches.
Half the houses we looked at fit the profile we wanted but we found ourselves.
I didn't actually, I'm trying to save as much money as possible at the moment and while a broker would be a good solution it would be a bit pricey for me. Maybe when I'm older and willing to spend money on relocating that would be a better solution for sure!
I'm not sure how Luxembourg works, but in the US the "seller" or landlord in this case pays a fee. Obviously this may be drastically different in Europe, but it's worth having a conversation with an agent/broker.
My wife is a realtor here in TX and when she is in-between home sales she helps people find rent houses and apartments. The seller pays a pre-agreed amount of 30-40% of the first months rent as a type of compensation. The buyer doesn't pay her anything at all. She is incentivized to find the right property for her client so she can get paid, and she also is better positioned to talk to the listing agent/landlord about you and the current competition among applicants. She ends up being able to save people money on application fees that would never get approved.
A lot of brokers work on commission that they get from the landlord. At least where I live, you don’t really pay for a broker as a buyer/renter, but I admittedly don’t know how that works in Luxembourg.
A lot of brokers specialize on ex-pats, too!
To expand on my point about knowing the local area, brokers know things like “what kind of person likes to live here?” Is it a young and trendy area or a quiet family neighborhood?
(Admittedly, that information can be based on biases rather than data.)
It’s something to think about. I know you might be looking to save money, but where you live is going to be a major purchase decision. Plus, spending time on your own apartment hunting platform isn’t technically free, your time is valuable.
In any event, I’m sure with the kind of skills and dedication you have you’ll find a place in no time.
> A lot of brokers work on commission that they get from the landlord.
It depends on the city. In NYC, the broker is pretty much always paid for by the person looking for an apartment, and it's quite a bit of money. I haven't looked for an apartment for 10 years, but I think it was 5-10% of the annual rent. (I imagine this gravy train is coming to its end, though. 10% was fine when your employer was paying for relocation, but in the All Remote future, this free money goes away.)
Kind of similar in Spain. Usually it's equivalent to a whole month rent.
A new law a couple years ago mandated that, if the landlord is not a person (a corp / company / org...) they must be the ones paying the fee.
But I've been "extorted" twice in the last year by companies moving the payment to my person. And I say "extorted" because they defer details or information about the rental until the last minute, when you are days away from having to leave your old place and you have no other option.
I think in NY specifically it's still structured this way. But for the US as a whole, it's much more common for it to come out of the seller's end.
Honestly here in Austin, unless you're walking into an apartment leasing office -- you're going to waste so much on rejected application fees by not utilizing an agent to rent a single family home.
This is the prime example of a phenomenal approach to aggregation of various data using various languages, in this case on about hunting for a decent apartment.
I am so lucky to have found a room in a descent apartment in a nice part of NYC. As a software engineer making high income, I still almost got homeless in the apartment hunting. Insane!
"creating a pipeline with different tools and languages to scrape data from a website with the use of Puppeteer, load data into a SQLite database using Go and visualizing all the data with Python and Folium."
So much sophistication based on at least partly incomplete, filtered/distorted, and inaccurate data. So much important aspects that cannot be in an ad or intentionally left out and can only be extracted by painstaking efforts. Even the initial filtering out could be unreliable then.
This reminds me of the small Go utility i wrote to help when i was in a similar situation. My focus was entirely on checking the location though, and see how much time it'd take to get to my various points of interest ( work addresses, places where things happen, favourite restaurants and bars, etc.). (It was just one of the things we were looking at with my SO, but it was a factor).
It's all Google Maps with arguments so i should probably open source it, someone might find it useful.
Kinda funny side story: I tried scraping Homes (the Japanese equivalent to this site) for a small subsection of data, basically instantly got IP banned and could not use the site consistently until I outright moved to a new location and made a new internet contract
How many major cities in the world do we have right now where prices of rent did not go up?
I'm really curious to know the breakdown of home/building ownership in the world, and how much the person makes in that city. Curious how much property is owned by individuals vs companies.
If I have time, I always prefer to walk around the neighborhood I want to live in an see if there’s signs or hear from people if a place is available informally. That’s where you get the best deals.
If you find the seller, you usually get a better deal. If the seller finds you, …
May be I am too old fashioned but I have never been able to select apartments like this mathematically . Sure I have used methods like this to shortlist the apartments, but I still need to visit them and feel for the place before commiting to one place.
I've received a PR suggesting just that [0], my concern is
> Only thing that concerns me is that it will spawn n parallel requests to the website which could consume a lot of memory since we're talking about spawning a new tab with Puppeteer (Chrome :sigh:) for each link.
I didn't have an easy solution to grab that JSON variable directly by scraping HTML. So I chose Puppeteer because it simulates the browser (yes, chrome) and therefore I could just get the JSON object with a bunch of lines as if I was using the browser's console. It's slow, but I think it's the most straightforward way to do it. But if you could elaborate more on your approach I will definitely try that out!
I had a similar problem that I solved with goquery and otto. You can use goquery to traverse the DOM and otto to execute the script fragment. Then just grab the data from otto's VM.
Your scraping being slow and using Chrome might be a blessing in disguise though. If you aren't careful you can get detected as a bot and banned from the site.
Looks to me like procrastination from doing what he actually needs to do. Which is to click on those links to find an apartment. Tech won't solve that.
I partially agree! But atm I’m using the tool a lot, it’s working well and as expected. Let’s say that for the time it took me to develop it was definitely worth it
And you are in the beginning of your career, can just as well post it to make yourself attractive for future employers. Better solve an actual problem that you had than to post some made-up nonsense-projects on Github just because people think you gotta have "a Github" when applying somewhere.
You submit precious personal info over to a random stranger along with payment to check your credit. Last time I checked, this fee is $25-50 USD. You've now given them enough information to steal your identity. You also may need to do this multiple times if you're rejected. The person receiving this information rarely has identity protection measures in place and you receive no guarantees that they've destroyed this information.
A much better solution would be to develop an escrow service which acts as a middleman between renters and landlords. Aspiring renters could sign up with the escrow and submit their information once. Their one-time fee would cover a fixed number of queries by potential landlords. The queries could include such information as income validation, criminal history, job history, rental history, etc. But these queries could be approved by the renter in advance. This sort of query process allows the renter to control the types of information given out. The queries could be structured to allow pass/fail results only, so that landlords do not receive intimate details but only know whether or not a renter is acceptable based on their criteria.
Once you have a system like that, you can open it up to all sorts of validation queries. Think of it as a general personal information escrow service.
This idea has been bouncing around in my head for over a decade now but I'm more of a low-level systems engineer so I never got around to trying to implement it. Seems like a decent startup idea though.
A service similar to what you’re proposing called Smartmove already exists. The renter pays them for the application + credit/background/etc checks. You don’t have to handle payment or take SSNs. Unfortunately pass/fail doesn’t work due to nuance in income/credit/rental history.
Zillow does have a product like this: pay one application fee and you can reuse the background check across as many apartments as you apply to in a month. The biggest downside is that it only works for landlords that chose to accept the service.
Why don't you just ask someone if they have an apartment? We rely so much on technology, when we could just ask a friend. Currently living in Belize for $200 a month, beachfront propety.
> TL;DR: I hate apartment hunting, I’ve tried to make it as interesting as possible by creating a pipeline with different tools and languages to scrape data…..
Grow up, learn to take on boring tasks without playing with toys
Government should provide apartments, free, in the center of the city, for anyone who needs one. Housing should be a human right. You can pay extra for a better place, but then landlords would actually have to compete for your business instead of the other way around.
The UAE are in a uniquely blessed situation where they have a small population, not a lot of land, and a ton of valuable resources. Not every country could pull this off without significantly reworking how it's tax/budget/administration work.
That being said, there's subsidised, including to 100%, housing in France too. And cities are legally required to build affordable housing for those schemes, so it's not "it exists on paper, but nothing has been built in decades so nobody can get in".
Are you insane? Then what, abolish poverty, stop human exploitation, and maintain the ecosystem that allows our species to survive? You dangerous revolutionary.
Its actually interesting to read about housing in the Soviet block. Housing was a right, but in reality long waiting lists, multi generations in cramped apartments and lots of dense blocks. But it kinda worked. I'm not sure how private stand alone housing was allocated.
Conclusion: it hasn't changed much at all.
It's still necessary to search on a myriad of sites. Most of the time, by the time you inquire, a place is gone. Or the ad was misleading and that "$2000 2BR" is actually "$2000 for the cheapest Studio we have, but we also have 2BR for $3.5K".
Finding places that match my criteria still requires careful inspection of the description. Checking a filter box for "Parking" excludes places that only mention parking in the description.
With all of the modernization and streamlining that has happened in other parts of our lives, apartment hunting remains unstructured, messy, frustrating, stressful.
Aggregators like Padmapper helped for awhile, but these days, it seems like most listings are posted by scammy middlemen just trying to make a commission on the listing, and now the site actively makes the process even harder.
Is hunting for an apartment doomed to be this messy forever? It seems like an area that is ripe for disruption or a better option, but what would it take to actually solve this?
---
With all of that said, this post is exactly what I needed right now. I've been toying with a very similar idea to improve the efficiency of my search process and this is good inspiration.