> But the A/B tests showed various negative effects of the feature, including fewer clicks on the results and fewer items “favorited” from the infinite results page. And curiously, while users didn’t buy fewer items overall, “they just stopped using search to find these items.”
How is this a negative effect? The amount of stuff that users are buying should be pretty much the _only_ metric you care about. And they are probably not using search as much because browsing is much nicer with infinite scroll.
It sounds like they just did infinite scroll wrong.
If you read the comments page on the forum, the biggest complaint is that you lose your place in the scroll when you return to the scroll page. If they had listened to their users and made it so that you return to the spot where you clicked on the item, perhaps it would have been a success.
When I use these things (and even pagination sometimes), I might find something I like, but it's so easy to browse ahead that I'll skip over it in the hope that I'll find something I like even more further on.
At that point, there's no easy way to get back to what I originally liked unless I had the forethought to open it in a tab or remember it exactly.
As for the metrics, it depends on how the business wants to position itself. It's evident from the quote that they care more about just the sales and would prefer to encourage more engagement with their site.
> How is this a negative effect? The amount of stuff that users are buying should be pretty much the _only_ metric you care about.
Items ordered is most helpful for making decisions when in context. It's awfully difficult to gauge health or make decisions with in isolation. A superficial short-term return may not be as beneficial on other levels - strategically, operationally, vendor ecosystem...
The article and other commenters have pointed at some, a few more more basic things worth knowing:
- return visit behavior. If infinite scroll diminished visitors favoriting items, what's the relative conversion for visitors returning to a list of favorites vs not? (features like this, Amazon Wish List etc, usually perform pretty well).
- avg order value. Visitors are buying the same number of things, but are they making the same $$ per order? Preserving avg order value but increasing item count also increases operations overhead (which could be worth it to encourage a broader vendor pool...)
- Clicks to items helps Etsy identify interesting/trending items&sellers independent of sales (move your data knowledge upstream!). Also increases the data size on individual users to build a taste profile and surface more items of interest at an individual level
> The amount of stuff that users are buying should be pretty much the _only_ metric you care about.
Generally yes, we would trade an increase purchases for anything else. The test was a negative effect because we forced people to stop using search, which was the feature we were experimenting with. Making buying less efficient, even if people want to buy so much that they eventually succeed, isn't a great result.
> And they are probably not using search as much because browsing is much nicer with infinite scroll.
No for two reasons. First, browse pages didn't have infinite scroll back when this test was done. And second, the point is that people seeing infinite scroll purchased less from search as compared to the control. This was a controlled A/B test.
As gfodor says, yes the back button did return you to your place (or it did, once we ironed out all of the problems with it in early incarnations).
Given that I've never yet used an infinite scroll implementation that had reliably-working 'back' button behavior, I'm skeptical of the claim that yours worked. I'd put some money on the hypothesis that it worked under ideal conditions, or worked in testing, but failed an irritating fraction of the time in actual real-world usage.
EDIT: In fact, even if by some miracle it didn't, it probably still triggered a twitch of stress and irritation in users who assumed it would.
I helped out with this project at Etsy and obviously our implementation intended for the back button to work as expected. Our initial rollout worked fine in most browsers but certain older versions of IE were discovered to have issues. Of course, the experiments that Dan referenced in his talk were in regards to known-good implementations that did not have issues with the back button.
Anyone who has worked on infinite scroll knows that the back button is a "interesting" technical challenge.
> the back button is a "interesting" technical challenge.
Especially when some A/B testing seem to show that it is a bad UX idea (for shopping goods).
Side thoughts: I wonder if they did A/B test "My apps" page on Google play (https://play.google.com/apps), it manages to have all the wrongs of infinite scrolling (no fixed ground, no sane back button) while not being infinite scrolling.
IIRC we ended up using a cookie (in non web history api browsers) so when you hit back the page would render the entire necessary result set to get you back where you were. Ie, no Ajax load to wait for. (Don't quote me on this :))
> The amount of stuff that users are buying should be pretty much the _only_ metric you care about.
I agree if Etsy is a 100% commerce focused, but they aren't. Key to their success has been their ability to build up a community and sell stuff. Even if you weight down the impact of clicks and favourites the effect is still negative with less exploration and interaction happening. What would be the long term impact of sales of this reduction?
That's a good point, about whether it was really a failure if those users still bought things...certainly it didn't seem like a benefit to search at all...but maybe most people just end up buying through other avenues anyway, such as following a shop
Also to your point, Dan talked about how they also thought that opening results in new tabs would be a sure fire winner (because power users open up multiple tabs to do comparisons), but it was also a significant negative effect. So clearly Etsy users back up when venturing into results and apparently expect to be at the exact same state they were before clicking through
> So clearly Etsy users back up when venturing into results and apparently expect to be at the exact same state they were before clicking through
Which makes sense, given that the multiple-tabs-at-a-time is often a power-user only thing. Etsy's users are much more "typical" in their usage of the browser; similar to Tumblr (who also ditched infinite scroll).
Infinite scroll that remembers your place vs infinite scroll that doesn't. Now that would be an interesting test. Although I think I know which one would win.
I have this problem frequently on many sites; whether its from a UI deficiency or my own memory error. I have trained myself to always open listed/grid links in a new tab as a result.
When building sites now, I usually contemplate using target=_blank on any lists just to aid this kind of pain point.
How is this a negative effect? The amount of stuff that users are buying should be pretty much the _only_ metric you care about. And they are probably not using search as much because browsing is much nicer with infinite scroll.
It sounds like they just did infinite scroll wrong.
If you read the comments page on the forum, the biggest complaint is that you lose your place in the scroll when you return to the scroll page. If they had listened to their users and made it so that you return to the spot where you clicked on the item, perhaps it would have been a success.