
Machine Learning-Powered Search Ranking of Airbnb Experiences - jamesjue
https://medium.com/airbnb-engineering/machine-learning-powered-search-ranking-of-airbnb-experiences-110b4b1a0789
======
benjaminwootton
Fantastic that they share this information and an enjoyable read.

This said, users can’t currently order AirBNB search results by price
ascending or descending.

[https://community.withairbnb.com/t5/Hosting/Sorting-
listing-...](https://community.withairbnb.com/t5/Hosting/Sorting-listing-by-
price/td-p/559404)

I think we should remember to empower the humans a bit more than the machines?

------
s_Hogg
The headline screams red flag to me. Calling something "machine learning-
powered" isn't a guarantee of improved performance at all. Their baseline is
already a fairly complex model, so there's no way to gauge how useful having a
model _at all_ is.

The partial dependency plots they show have gigantically wide error bars on
them - they talk as if they offer a really nice confirmation of intuition, but
the real relationship could be anywhere in those intervals, so it's probably
weak evidence at best. The only one of the three where you could say with
confidence that there's a relationship is the higher price -> lower score one.
Add it all up, and you've discovered that people prefer cheaper stuff, with
maybe a whisper of it being more complicated than that. In which case, their
stage 1 model likely amounts to "show them the cheap stuff first", without the
developers involved realising it. This is one of the problems we have with
modelling - the model could be doing something simpler than we realise.

This gets more worrisome when you consider the fact that they're doing A/B
testing to check for performance improvements. How are they able to claim that
they improved bookings by 13% with their stage 1 model? Improved relative to
what? More importantly, how do they know that this isn't a dataset-specific
effect given that the data used for the stage 1 model is "small"?

I didn't continue reading after this, the whole article looks like
confirmation bias to me.

~~~
orasis
The A/B test was against a random ranking.

------
syspec
I find the searches on Airbnb to be pretty ineffective. Often times I’m trying
to find a listing I was previously viewing and it’s nearly impossible to find
it (referring only to the cases where I eventually do find it, not when it’s
no longer available)

~~~
pheug
It seems to be working as designed:

"experiences that were clicked but not booked (which we treated as negative
labels)"

~~~
moritzmeister
Too bad that they don't provide any reasoning for this labelling. Also it
isn't clear to me over which time span they were looking at individuals to
make the label decision, i.e. are they considering sessions or just single
clicks with subsequent bookings?

------
sbmthakur
Related: [https://blog.acolyer.org/2019/10/09/applying-deep-
learning-t...](https://blog.acolyer.org/2019/10/09/applying-deep-learning-to-
airbnb-search/)

------
dwoozle
Look at the authors. China is really going to put the screws to the US and
it’s going to be sooner rather than later.

