
Why did infinite scroll fail at Etsy? - danso
http://danwin.com/2013/01/infinite-scroll-fail-etsy/
======
MattRogish
Boy, I really hate infinite scroll. I've never found it enhanced my experience
over plain old paginated results for anything product related (sure, maybe it
works for Google or HN home page).

I like knowing:

* how many items are in the results

* how many pages there are (a derivative of the first, sure, but helpful to know)

* where I was if I accidentally close the tab and re-open it, or follow a link and need to hit the back button, or share my position with a friend (yes, most paginated results aren't stable in the long run, but usually are in the short run)

* easily jump to either end of the list (or, near to the end of the list, e.g. < 1, 2, 3, ... 98, 99, 100 >)

For example, I follow way too many people on twitter (1700!). I know I'm
getting tweets throttled so I miss out on people I care about.

I also know that when I first picked up twitter years ago I made some noob
mistakes and followed things I, well, shouldn't have (@Tide? I think a friend
was working at P&G or something...).

Anyway, "People you follow" on the web UI is an infinite scroll. Even worse,
it's buggy so sometimes you can trigger it to not register you're at the
bottom and it won't load. AWESOME. I just want to get to the LAST PAGE. BUT I
CAN'T.

And since it seems to load about 10-15 at a time, given that it takes about
1/2 a second to scroll down and wait for it to load, that means it'll take at
least 60-85 seconds to reach the bottom - IF it doesn't crash (a reload takes
me back to the top). Which means I've never been able to do it.

I had to pay one of those "show the folks you follow that haven't tweeted in
n-months" just to try and prune the list, which helped me go from 2,000 to my
present number.

Yes, this could be solved by sorting and filtering, but in the truest MVP sort
of the world, why do all that extra work just for the "infinite scroll" fad?
Switch back to pagination and I could accomplish all I needed and have a nice
pruned list. And I bet it's less effort and has far fewer bugs than the
current implementation.

I hope that in a few years enough data against infinite scroll will have
cemented it as a generally accepted bad idea, only working in a few particular
cases.

~~~
firefoxman1
I only enjoy infinite scroll on touch screens. I think the perfect solution
would be paginated results, but if the user ontouchstart pulls the page past
the bottom, it would automatically load the next page (like reverse of the
facebook app's pull-down for new items). It's much more convenient than trying
to tap _just_ perfectly on the "next" arrow, but paginated is just easier with
a mouse.

That mindset of design I think is the next step after "responsive" design.
Rather than just scaling the visual design of the page, how about truly
_designing_ the user interaction to fit multiple scenarios. If it were up to
me, I'd call it _Symbiotic Design_ (responsive 2.0)

~~~
MattRogish
That sounds like a great idea. I do agree, touch seems to fit infinite scroll
pretty well - pagination navigation is a pain on touch devices.

------
oconnore
> But the A/B tests showed various negative effects of the feature, including
> fewer clicks on the results and fewer items “favorited” from the infinite
> results page. And curiously, while users didn’t buy fewer items overall,
> “they just stopped using search to find these items.”

How is this a negative effect? The amount of stuff that users are buying
should be pretty much the _only_ metric you care about. And they are probably
not using search as much because browsing is much nicer with infinite scroll.

It sounds like they just did infinite scroll wrong.

If you read the comments page on the forum, the biggest complaint is that you
lose your place in the scroll when you return to the scroll page. If they had
listened to their users and made it so that you return to the spot where you
clicked on the item, perhaps it would have been a success.

~~~
gfodor
I helped out with this project at Etsy and obviously our implementation
intended for the back button to work as expected. Our initial rollout worked
fine in most browsers but certain older versions of IE were discovered to have
issues. Of course, the experiments that Dan referenced in his talk were in
regards to known-good implementations that did not have issues with the back
button.

Anyone who has worked on infinite scroll knows that the back button is a
"interesting" technical challenge.

~~~
gerwitz
I'm curious how seamless that implementation was. I've seen a few hacks that
load-then-scroll which is, of course, just about the worst of all worlds.

~~~
gfodor
IIRC we ended up using a cookie (in non web history api browsers) so when you
hit back the page would render the entire necessary result set to get you back
where you were. Ie, no Ajax load to wait for. (Don't quote me on this :))

------
georgemcbay
Contrary to Dan McKinley's half-apology as quoted in the article, infinite
scroll _is_ stupid on your website.

Infinite scroll flies in the face of the way the human brain works with groups
and sets, makes it virtually impossible to usefully search within the current
page using the browser's search feature, etc. And the positives are? Nothing,
other than novelty.

It is one of the many recent examples of webdev/designers doing something
because it is possible and trendy and new rather than because it adds any
value.

Paged results with a well designed indication of where you are plus good
server-side categorization and server-side search to filter results is far
preferable to infinite scroll in every practical situation.

Keep the infinite scroll for those purely arty non-commercial story-telling
sites, if you make me try to use an infinite scroll interface to buy stuff
from you I will buy nothing.

~~~
gfodor
Of course your opinion may or may not be right but the point is to figure out
a way to test it. The reason infinite scroll was attractive at Etsy, among
other reasons, is because it is reminiscent of the experience one has at a
large craft fair where you quickly scan through hundreds of items on tables
looking for one that jumps out as worth zooming in on.

~~~
georgemcbay
I don't think those experiences are at all comparable. I'm a compulsive
scanner, I can usually impress people by my ability to quickly scan pages of
text without really processing any of it while finding a specific word or
phrase. Infinite scrolling is nothing like that experience for me. For one
thing it is impossible to involve peripheral vision with any type of web page
scrolling and peripheral vision is absolutely essential to that type of
scanning. And moving your eyes to scan something that is fixed vs. keeping
your eyes still to scan something moving are also vastly different experiences
in my opinion, not really at all alike.

~~~
gfodor
Sure, this is a anecdote based argument levied against the hypothesis. Its a
fairly weak way to make final decisions though. Again the point was that there
was intuitive reason to believe this concept _may_ be fruitful, the problem
was we devised too large of an experiment and spent too much time to get to a
point to validate the core hypothesis.

To be perfectly honest opinions like yours provide guidance of what to focus
on but in general this type of intuition usually turns out to only capture
part of the dynamics when something is actually used by millions of people,
and in many cases intuition turns out to be the polar opposite of the reality
the data tells you.

------
ComputerGuru
Infinite scroll really annoys me on most sites where you're searching to find
something (vs carefully reading each and every result). It's frustrating to
not really have any sort of "progress" indicator.

For example, try "scrolling" this list of acrylic sheets on inventables:
[https://www.inventables.com/categories/laser-
cutting/acrylic...](https://www.inventables.com/categories/laser-
cutting/acrylic-sheets)

Makes me want to find whomever implemented the system and shoot^H^H^H^H^H err
explain to them how unusable it is.

~~~
stan_rogers
Infinite scroll annoys me, period. I spend a lot of my time on 3G mobile, and
data costs me money. When I scroll down to the bottom of the page, it's
because I want to read what's there, not because I want to give my mobile
provider more money. When we're all on flat-rate, unrestricted hyperspeed
connections, maybe. (I wait with bated breath.)

~~~
1amzave
Infinite scroll annoys me, period, too, but for entirely different reasons
(ones that aren't going to change with improving network infrastructure, more
related to ComputerGuru's reasons).

When I start browsing the sort of page that web developers are prone to
applying infinite scroll to, I usually begin with a sort of semi-subconscious
assumption that I'll just read down to the end of the page, or just the first
N pages (for some smallish N) or something. I start doing so, but for a while
don't realize that I'm never going to reach that point because the page is
continually growing as I read it, and thus lose my way of quantifying
(admittedly very vaguely) how much time I've wasted^W spent on it and where I
should stop. My usual solution is to hit the "end" key (or its moral
equivalent) and scroll up from there, ignoring whatever loads beyond that
point.

Now from the developers point of view this may be a good way to get me to
spend more time on their page, but if they're actually "gaming" me like that,
it really just makes it enormously _more_ obnoxious than it already was to
begin with. Gaming intended or not, it strongly disinclines me to continue
visiting the site in the future.

------
nodata
> Users want more results per page.

For me, I cannot use Etsy because there are simply too many products. I have
no way to narrow down the volume of products to something which anywhere near
approaches my ability to make a choice.

(At least with Amazon I can filter by department, then filter by four stars
and higher...)

------
lubujackson
Here's why I hate infinite scroll:

\- the little scroll bar changes sizes randomly and moves places in the
browser. Users can't predict this or know where it is, so they have to keep
hunting for it when stuff loads in the background.

\- if I browse a lot of items, scroll down a ways and then decide to scroll
back up, it is much harder to find items that otherwise I would remember as
being on page 3.

\- The scrolling gets jagged as my browser barfs trying to shove more things
in the list. In other words, even if it's faster it FEELS slower and less
responsive.

~~~
lubujackson
\- also, after a few loads I am a million miles away from the top of the page
which is really bad if I want to go back up to the search box or anything else
at the top of the page.

------
gav
I think it's a bad idea to look at what Google does with their search results
and then try to apply it to product search.

Users have a very different set of goals, with Google we know that the falloff
from the top search result to the bottom is huge, people expect that the
relevant results are the top couple.

With product search there's different goals, for example browsing an minimal
set of criteria looking for a results--you're looking a pair of jeans and
starting with 36x32; or the opposite you just want to know the price of a pair
of New Balance 990s in a 11 4E.

In the latter case infinite scroll or not doesn't come into play, but the
browsing case it does, and from the point of view as an Etsy customer, I have
the most difficulties.

One of the things I've found the most is people do better with a task if they
can understand how long it's going to take. If you know there's 100 search
results over 5 pages, you can decide if that's too much to go through. As an
aside, this is true of a lot of things, you'll do better with pull-ups if I
tell you to do 20 and give you a count, than just tell you to keep going until
I say "stop".

With infinite scroll you've got no idea how much effort looking through the
total search results is going to take. Etsy don't make things easy, as an
example Art > Custom Portraits[1]. I have no idea how many results there are,
I can scroll all the way to the bottom and find out there are at least 8
pages, but that's it.

The search results themselves are pretty snappy, so I don't see a huge
advantage of infinite scrolling. I do think the search results are pretty bad
and the lack of filtering is a problem. From Art > Custom Portraits I can
filter on just Pets or Silhouettes or More (which I assume is everything
else). It's not obvious what these links do either, they are just floating at
the top.

Some better filtering, e.g. by price, would help. I'm not sure what other
metadata Etsy have that would be useful for an great faceted search.

As an aside, I've been working on (product) search for over a decade and I'm
close by the Etsy offices; over a coffee I'd be happy to discuss search. Part
of my New Years resolution is to do a better job of networking, so I'm happy
to extend this offer to any other NYC based e-commerce shops.

[1] [http://www.etsy.com/browse/art/custom-
portraits?ref=br_nav_n...](http://www.etsy.com/browse/art/custom-
portraits?ref=br_nav_new_2)

~~~
elimgoodman
Hi! I actually worked on that exact feature. I definitely know what you mean
about lack of refinement options on that page, though you can refine (roughly)
by price. Also, email's in my profile, so hit me up if you wanna swing by the
office.

~~~
chaddeshon
Just a heads up. Your email is not visible in your profile. The "email" field
isn't visible to other people. If you want us to be able to see you email, you
need to add it to the "about" section.

------
brownbat
I remember hearing Aza's push for infinite scroll on search results... and
that guy is wicked smart, but I feel like it only works if we assume I'm
already engaging the scroll bar.

Clicking is so much faster than dragging, (barring terrible page load
times)... a part of me wishes no one ever invented scrolling.

------
karolisd
I've come across similar issues and I thought I coined the term "monolithic
testing" but clearly great minds think alike.

I'm the biggest proponent of avoiding monolithic tests and having clear and
testable hypotheses. I'm glad there's a high profile example to point to now.
Thanks.

------
Spiritus
I don't get all the hate towards infinite scroll. Personally, I like it!

\---

Q: Hey John Doe, check out page 3 at site.com, I totally want to buy item 3
and 6.

A: Obviously by the time John Doe checks the site/link item 3 and 6 might be
something entirely different. People send direct links to items.

\---

Q: Oh no, I don't know where I am in the scroll list.

A: Who gives a rats ass? Your content is right there. Interesting items you
found along they you open in a new tab so that you don't have to backtrack.

\---

Q: But I want to quick jump to page 45?

A: Why 45? You just pulled that number from your hat (read: ass) anyway. What
you really should be looking for is the search field.

\---

Q: But I never get to see the footer!

A: The what...? I don't think I've looked at a site footer since '94.
Navigation is at the top, and bullshit at the bottom.

\---

I just don't get peoples need to "know where they are", like it made a
difference?

------
abalone
Here's a theory:

1\. People only care about the first few search results.

2\. Infinite scroll exists solely to make it easier to scroll past the initial
view.

3\. Therefore, infinite scroll made it easier to scroll away from the results
people care about.

There you go.

By the way, I don't get why Google's "instant results as you type" was cited
as a reason to pursue this infinite scroll feature. Those are totally
different things.

"Instant results" is for faster display of the first few results, which is
great for search. "Infinite scroll" is for scrolling through long streams of
information -- great for newsfeeds and timelines, but not for search results
where you only care about the first few.

Google doesn't even implement infinite scroll in their results.

------
tobyjsullivan
This reminds me of the age-old advice of not overwhelming customers with
options. I.e., if you know the customer wants your product (say, a cell
phone), you're much better off giving them one or two options to choose from
(think iPhone). If you overwhelm them with 14 different models each a little
different in some small (or big) way, you're more likely to lose the sale
altogether.

This concept plays off the simple fact people are fundamentally bad at
choosing between many options.

I'm curious if this is the factor that caused negative results with the
infinite scroll. I'm also curious what would happen if you started only
returning 5 results/search...

------
dirtyaura
My guess it has something to do with the paradox of choice. Seeing more
options makes it harder to decide what to look at more closely, let alone to
make decision to buy.

Infinite scroll is a good solution, if you are building lobster traps - sites
where you want people to spend endless amounts of time.
([http://gizmodo.com/5586337/pandas-and-lobsters-why-google-
ca...](http://gizmodo.com/5586337/pandas-and-lobsters-why-google-cannot-build-
social-applications)). It clearly works for Facebook, Twitter and Quora.

Although Etsy has some aspects of lobster trap, it still is more transactional
than FB and other social time killers.

------
mikecane
Wordpressdotcom added Infinite Scroll to blogs. I turned it off but it still
has it when you want to see older posts. How much I hate it! I spend more time
staring at a damn spinning wheel than seeing more posts when Search won't
bring up what I need (due to not recalling an exact keyword). I wonder if this
was the case with Etsy too? The only Infinite Scroll I've seen that works
properly has been with Twitter. Even Bloglovin (which I use for RSS) has a
one-second or so stutter that can be very annoying.

------
marknutter
Here's a dissenting opinion: I hate pagination. Especially when I need to
browse through a lot of content. When I scan for something I hate being
interrupted every 10 seconds by having to click a tiny pagination target,
especially if that target jumps around from page to page. I understand that
most infinite scrolling implementations leave out critical functionality like
back button support, but that doesn't mean infinite scrolling can't be done
well.

~~~
epsylon
"View all". Problem solved. Or "View first X00 items", where X is digit (if
you're worried about performance).

Just like many people here, I hate infinite scrolling with passion.

~~~
jasonlotito
Judging by the reasoning people are providing, I only see problems with poor
implementation of infinite scrolling, not infinite scrolling itself, solved by
better implementation (like your "View all" suggestion).

In truth, your view all suggestion as a way to fix pagination is akin to me
making a paginate button to solve infinite scrolling.

------
pearkes
We (a friend and I) built a simple application[1] that gave folks relevant
clothing for the weather in their location.

We shelved the infinite scroll at the time out of haste.

It turned out to be a happy accident, as active users refreshing would
generate a ton of pageviews.

Not only that, but the act of clicking refresh to "get more" is a nice way to
get the user to engage.

[1] <http://wevther.com>

------
debacle
Someone could point out that Etsy's search is relatively poor, which was only
made worse by infinite scroll.

I tried to buy something that I knew was on Etsy during the holidays, and
after about an hour of looking (their category breakdown reminds me of reading
through the yellow pages - remember those?) I gave up and bought from Amazon
instead.

------
jbrooksuk
I wonder why this failed? I've seen infinite scroll work on some major
websites, Facebook and Twitter, but then perhaps it's related to search only?
FB and Twitter are loading a feed, a search page should give users what they
expect to see, not randomly see more elements pop on screen, perhaps?

------
chadyj
Seems like infinite scroll is a solution in search of a problem. Users don't
want to see more results. They want to see the right results. Etsy seems to
fail in this regard and has dozens of pages of irrelevant results. A better
move would be to rethink search relevancy and discovery.

~~~
gfodor
This is a false dilemma and while Etsy's search engine can always improve a
massive amount of work has been done over the last several years to improve
the relevancy of results. The vast majority of these have been small wins and
the large net effect would not have been possible without the methodology Dan
outlined in his talk.

------
Benferhat
Infinite scroll + History API is sexy, I'd encourage them to give it another
shot, this time without breaking the back button.

------
msutherl
Why not do both and allow users to select whichever they prefer? You could
then A/B test which is the better default option.

~~~
SoftwareMaven
There is cost associated with keeping code alive. It is magnified when that
code will have follow-on effects for how other code (search results format,
etc) gets designed.

If the code is not providing real value, it should be thrown out.

------
jfarmer
Warning: long comment ahead! Read at your own risk. If you ask me for a tl;dr
I'll kick you in the shins.

I'm so happy to see them measuring "median item impressions" rather than "mean
item impressions." Many of the underlying variables describing consumer
behavior aren't normally distributed. Talking about "average number of friends
invited" when 50% of your users invite 0, 25% invite 1, 13% invite 2, etc. but
3 users invite 5,000 will necessarily lead you to propose bad product ideas
and decisions.

However, I'm curious about the experimental design. For example, was this
tested on new users, old users, or both? Changing a fundamental part of a
site's experience like this will have some cost as users acclimate. I'd wager
Etsy's audience is less technically inclined, too, so it might take them
longer to acclimate.

They also commit a small fallacy when they talk about how they should have
done it instead, and IMO it's a fallacy that frequent A/B testing encourages
people to commit. They suggest that instead they should have determined first
whether more items are better and faster items are better.

On the most surface level, perhaps there's something about more items AND
faster items that outperforms either one or the other in isolation. That's
easy enough to accomplish, technically. You use different statistical tests,
but it's possible at the cost of perhaps a larger sample size.

On a deeper level, you're providing the users with a fairly different overall
user experience. Their sense of where things are placed, what they're supposed
to do when they want to "see more," how they know they have the opportunity to
"see more," etc. are aspects of the infinite scroll design that aren't
encapsulated in either rendering more items or rendering those items more
quickly.

For example, can users bookmark specific search result pages under the current
design? Can they still do the same thing under the "infinite scroll" design? I
imagine there are lots of little things like this and that the UX difference
alone would have a larger impact on the results than just changing the number
of products per page.

To get more meaningful results from this, I'd run this experiment under the
following assumptions.

1\. Assume that existing users will be more impacted by this change than new
users. Therefore the cost of "failure" for existing users is higher.

2\. Assume that at the end of the day the #1 thing Etsy cares about is "dollar
throughput" of the Etsy platform. Engagement, favoriting, searching, etc. are
all positive indicators of an increased dollar velocity.

3\. Assume they have information about what aspects of a users' first visit
are indicators of their long-term ability to contribute to Etsy's dollar
throughput.

4\. Assume that eventually every user will have the same experience, new or
existing alike.

So, I'd start by running the experiment with new users only. Over the course
of a week or a month I'd put a % of the users who joined each day into the
"infinite scroll" bucket. I'd then run the study as a longitudinal study.

Assumption (3) can guide us as to whether we need to cut off the experiment
early. The length of the study would be determined by the particularities of
an Etsy user's life-cycle, e.g., maybe given a cohort of users, we care about
the length of time it takes 75% of the eventual purchasers to make their first
purchase.

Because of assumption (4) we know that if the "infinite scroll" design is
terrible for new users, we never have to bother testing it on existing users.

[1]: Non-technical users, in particular, are sensitive to sudden change. I
forget where, but I read a research paper once that implied that the worst
thing you can do to harm a person's user experience is change the placement of
links, buttons, etc. You can change the color, text, icons, etc. but if you
change the placement, they essentially have to "re-learn" the interface.

IIRC, the users were given a task (e.g., "create a document") and they
measured two core variables: time to accomplish the given task and time until
their "performance" at a given task was equivalent to the control user
interface.

Changing the placement of a certain action in the UI had a deeper and longer-
lasting impact on users' ability to perform tasks than changing anything else
about the UI by a large margin.

~~~
kaliblack
I got the impression from the article that their experiment had very little
planning from the statement about it being to prove it was good and then
celebrate. This probably means it was a straight X% of traffic A/B test and
the results were only analysed deeply when it wasn't a positive result. This
is speculation only based on the tone of the article.

~~~
danso
This is the tongue-in-cheek slide that McKinley used:

[http://image.slidesharecdn.com/continuous-
experimentation-12...](http://image.slidesharecdn.com/continuous-
experimentation-121202174027-phpapp02/95/slide-29-1024.jpg?1354491839)

I didn't rewatch the entire presentation but McKinley does discuss the user
makeup in the experimental groups...yes they do account for different kinds of
users, and the most drastic difference between user behavior are between
sellers and non-sellers. Someone from Etsy would have to talk about how much
slicing-and-dicing of the demographic that they do...but even if infinite
scroll was good for some users (new users without pagination-related habits)
and not for others, it's probably not a good idea to have two kinds of search
experience in the hopes that the "oldies" eventually figure it out...based
solely on how hard it is to implement infinite-scroll in the technical sense.

~~~
mcfunley
It's pretty common for us to look at new vs. returning users and Etsy sellers
vs. others (sellers are obviously really engaged users, and behave
differently). Occasionally one group will stand out, but not in this case.

------
mintplant
The article mentions that infinite scroll is "prominent among popular Tumblr
themes, Pinterest, and of course, at Facebook and Twitter". All of these are
content consumption sites, and for at least Pinterest, Facebook, and Twitter,
clicking on an item in the list opens it in-page -- for Pinterest and
Facebook, it's in a modal box; for Twitter, it's expanded within the stream.
The content you're browsing is all there; there's no problem with clicking on
an item in the list to get a more detailed view.

At Etsy, introducing infinite scroll resulted in "fewer clicks on the results
and fewer items 'favorited' from the infinite results page". On an Etsy search
results page [1], clicking on an item bounces you to another page. And on
sites with "infinite scroll", this is typically a _very_ uncomfortable
experience, particularly when trying to get back. Depending on the
implementation, you're wind up back at the beginning of the result set; even
if you don't, it's usually a fairly bumpy ride, with the time taken for the
page to reload its data and the jumps in scrolling as everything loads in.
Even if it technically works, the kind of sensation this brings about is
enough to discourage someone from actually clicking through. There's an
negative association that develops with clicking on these items, the
foreboding feeling that you'll end up losing your place, such that one tries
to do so as little as possible -- in line with what was observed from the Etsy
experiment.

On the Etsy search page, you can indeed "favorite" items without going into
the item's actual page. But that's not something one is likely to do based on
a tiny little thumbnail -- one would usually first click through, see a bigger
picture, and possibly read the description below. It doesn't help that, on the
results page, the "favorite" button is but a tiny little icon, that only
appears if you directly mouse over the thumbnail; meanwhile, on an actual item
page, it's right there under "Add to Cart" [2]. Perhaps users weren't even
aware that you could do this from the search page.

With a site like Etsy, where lists are a means to an end -- a way to get to
information on other pages -- it's no surprise that infinite scroll performs
quite poorly, as opposed to content consumption sites, where browsing is a
self-contained experience of its own.

(As an aside, it's fairly silly to compare infinite scroll with Google's
Instant Search. Instant is well-liked because it gets you to your search
results faster; this being Google, the user isn't there to hang around and
enjoy the scenery, but to get to the information they're looking for as
quickly as possible. And Google's results pages themselves still use
pagination, despite their experimenting with infinite scrolling back in 2011
[3] -- a change that, quite clearly, didn't make the cut.)

[1]
[http://www.etsy.com/search?q=test&view_type=gallery&...](http://www.etsy.com/search?q=test&view_type=gallery&ship_to=US)

[2] [http://www.etsy.com/listing/65114535/12-boracilicate-
glass-t...](http://www.etsy.com/listing/65114535/12-boracilicate-glass-test-
tubes-with?ref=&sref=)

[3] [http://searchenginewatch.com/article/2103479/Google-
Experime...](http://searchenginewatch.com/article/2103479/Google-Experiments-
with-Infinite-Scrolling)

