
Steam needs to stop asking its customers to fix its problems - smacktoward
https://www.rockpapershotgun.com/2017/02/14/steam-curation-user-reviews-fixes/
======
researcher11
I'm an applied researcher in search science. This is a very typical domain
specialized search problem. These days it uses Big Data, Machine Learning,
constant experimenting / prototyping, A/B testing. I get to geek out each and
every day. It is my dream job :)

I consider it one of the main reasons Apple lost so much content market share
to Amazon despite Apples first mover advantage - Amazon has better search and
you can actually find the stuff you want. AFAIK Apple tried to start up an
applied research group to fix the problem but they refused to pay enough and
ended up with the dregs. They tried to hire me 3 years ago for $130K. It
highlighted that they didn't understand the importance of the problem and
we're likely to fail. I typically make ~$240 as FTE and more as a contractor.
Apple also has a culture of secrecy where as search science is pretty open.

Anyway; I checked out Steam. The search there is clearly very poor and can
easily be improved. The overreliance on tags is usually a good indication that
they are not digging into their usage data for their recommendations.

~~~
mrskeltal
Can you recommend some readings for developers who aren't familiar with search
science?

~~~
researcher11
It's basically the union of Information Retreival, Big Data, and Machine
Learning. There is a lot of good and bad info out there.

It's best to tailor your learning around a problem so you can get feedback on
what works. Building bespoke search engines is expensive and often not worth
it until the problem is big enough. Machine Learned optimization is even more
expensive.

So in the likely instance that your problem is small; I'd stick to an off the
shelf Lucene. If you need more specialization write plugins for it. If you
need more speed then DIY OkapiBM25 in native (maybe Rust these days). If you
need Big Data I'd use Spark. If you need ML then GDBT in R. If you need
advanced NLP then Deep Learning. At each stage it's usually diminishing
returns. So quit once you start losing money on the additional effort.

Edit: As a fun aside. Page Rank doesn't work very well and AFAIK Googles major
advance was from creating 'meta documents' using anchor text and search
queries. Google has a habit of sending out red herrings to guard their
important ideas. So if some blogger is waxing lyrical about Page Rank you know
they're full of it.

~~~
Retric
Page rank was very useful when they started, 19 years ago. It was killed by
their own success as normal people stopped posting link sites, and bad actors
added their own.

~~~
researcher11
The people that I know at MS and Yahoo told me that Page Rank was never as
good as meta-documents. And meta-documents was more than sufficient to explain
the improvement in relevance.

Of course this is anecdotal and I would be happy to see evidence to the
contrary.

------
mjolk
Core complaints of article (please add any that I missed):

1\. Steam receives dozens of releases a day and it's difficult for
developers/publishers to make their title stand-out without performing
marketing.

2\. Author perceives that Steam's display algorithm results in display of
outliers (very good/bad).

3\. Very low quality titles are released/being sold by Steam.

4\. Reviews are a very poor metric as reviewers can be shills or rate content
poorly out of pettiness or malice.

5\. Author thinks that staff-curated titles are the solution.

My responses:

RockPaperShotgun is a game review site. _Reviewing games_ is how they manage
to make ad revenue (well, besides click-bait articles). If the defense is that
more content comes out than RockPaperShotgun can review, then RockPaperShotgun
should hire more people. If that's cost prohibitive, and there's market demand
for it, then why are they asking Steam to do it?

1\. Welcome to selling anything.

2\. I don't know if this is true; I'm not rejecting this claim, but I'd love
to see the data.

3\. Quantifying "good enough to sell" is a risky game. I'm okay with spending
$5 (most of a beer at a bar) on something kind of buggy if it's amusing for 30
minutes. For someone else, that amount may be double or $0 on principle.

4\. If this subproblem gets solved without background checks on reviewers, you
have a serious business that Amazon/Yelp/etc would love to buy from you.

5\. Beyond catching content that simply _does not do what it says on the box_,
this is all personal taste.

And as an aside, there's a "curious" market that exists in low-quality games
that have DLC/content that ends up on the Steam Community Market (Steam gets a
% of each transaction), so the "how did a broken game end up on Steam?"
question has more depth than a casual observer would guess.

~~~
ramy_d
Asking review sites to sift through the dredges in the spirit of "more
business is better!" is naive. Jimquisition comments regularly on this and it
has turned into a longstanding and ever constant critique of Valve and its
"Whatever goes" attitude to getting on their Steam platform. At 20 or 30 new
games _a day_ consumers don't have time to go on review websites and see
what's on the "do not touch list". What a monumental task for consumers! If
steam would block games that, out of the gate, did not work, that would be
great. As the author stated:

    
    
      This morning in half an hour I got through three interesting-looking games that didn’t have functioning controls.
    

You can't even play them! How are unplayable games on Steam!? Additionally
they should make sure that the min spec IS an actually min spec that run the
game in a playable state. No studio should be immune to such a level of
scrutiny (see Rocksteady's Batman Arkham Knight).

~~~
Retric
Paying someone 10$ an hour to do a quick look at a game for 2 hours * 30 new
games a day ~= 220k/year, double that to pay for games and your looking at
~450k/year which is not free, but that expensive either.

Really this could stand on it's own as a product either as a review site or an
industry group that supports several review sites.

PS: An yes that seems like a high estimate the average is less than 20$/game
and most of this junk does not need 2 hours.

~~~
ramy_d
You argument is that its cheap to review. Sure, but why should all review
sites duplicate the work? Why can't valve do the review, stop the crap at the
source.

Stop apologizing for shitty content making it to the store shelves.

~~~
Retric
> _or an industry group that supports several review sites._

So, no I don't think any one site needs to take on this cost rather anyone can
get it off the ground for a few months and then sell a feed to other people
thus defraying the costs.

Sure, steam could have higher standards, but IMO there is value in letting
poor games into their ecosystem that may be improved with time. UnReal World
for example been in development by one guy for 26 years and right now it's a
solid game though dated game. But, 20 years ago it was a buggy mess that would
only really interest a tiny group of people even though he had been working on
it for 6 years.

~~~
ramy_d
There's "Early Access" (which wasn't even touched upon!) for titles like
those.

------
kelvin0
I've been using Steam for many years and I play 5-10 hours a week. I simply
find it great, to purchase games and play them (unlike EA origin and all it's
clones out there, last time I checked)

The way I decide to purchase a game is simple: 1) I see/read a review about a
game 2) If it piques my interest I go check out some videos and read a few
more reviews 3) Add it to my wishlist 4) Purchase it when it's on 'special'

Finally, I almost never buy games unless they are at least 1 year old (save a
few exceptions). This article seems mostly to have a contention about new
releases and the 'suggestions' made by Steam (which I almost never look at
anyways).

Hope this can help others with their Steam purchases.

~~~
SixSigma
try Gog.com

~~~
kelvin0
Cool, but how is it different\better than Steam? Thanks for your suggestion.

~~~
SixSigma
No login needed to play games once downloaded

All games DRM free

Games are tested before being listed

------
slantyyz
I understand that the reviews and discoverability are legit problems for
smaller developers trying to sell their games, but I have to wonder what
percentage of people using Steam are simply "in and out" customers who are
looking for the known titles anyways. Those people (like me) don't care so
much about user reviews or discoverability.

I'm an "in and out" customer - I love games but I basically only buy big name
titles (old behaviour from my console days). I know I'm missing out on some
other great titles, but I only have a limited number of hours to put into
games.

The only time I might get a lesser known game is when it got a lot of buzz
outside of Steam, but the most likely scenario is via a heavily discounted
bundle that contains a big name title that interested me.

Of the ~100 games in my Steam library, only about 20% are games I specifically
wanted the buy. The rest were basically freebies that came along for the ride.

------
gwbas1c
Uhhhh... Steam needs a decent startup time. I only play computer games about
once a month. When I click on the link on my desktop, Steam always takes 5-10
minutes to update itself.

This is ridiculous. I never have this problem with games on my Nvidea Shield.

~~~
Shivetya
I just wish to understand why they need an application update so often to what
should be a simple distribution product. Do they hard code item availability
or such?

~~~
spatulon
There have been 12 updates in the last 12 months - do you really consider that
to be "so often"?

Anyway, there's a detailed changelog[1] for all the updates, so you can have a
look at the kinds of things they change.

[1]
[http://www.steampowered.com/platform/update_history/index.ph...](http://www.steampowered.com/platform/update_history/index.php)

~~~
zerocrates
A fair number of those updates go out 2 or 3 times with little fixes but the
same "release notes," so that's not a particularly accurate count of the
number of times Steam will want to update.

------
drivingmenuts
I'd rather Steam get back to the core business of selling games, rather than
trying to be some sort of oddball social network.

Drop the achievements (do a large number of players actually pay attention to
that?), drop the trading cards (I still don't know WTF those are for). Drop
everything that isn't about selling a game to a player. Then hire some
competent UI people to simplify and clean up that abhorrent interface that
tries to do too much.

In fact, separate Steam from Valve entirely. Let Valve focus on producing
games and let Steam focus on selling them, but put a wall between them.

~~~
striking
The trading cards are part of the Steam economy. It's a money-making ploy. I
don't know if it works or not, but it probably does.

If they dropped everything that isn't about selling games to players (Steam
Matchmaking, social features, the Steam economy especially wrt CS:GO and Dota)
I don't think Valve would be able to sell and host games as cheaply as it
does, and I think more people would pirate games rather than getting them on
Steam.

~~~
chrisdbaldwin
Ding ding ding! Each one of those trading cards facilitates trade. When there
is more trade, there are more transactions for valve to skim. It's simple
broker economics, and it works great for everyone involved.

~~~
elcapitan
Except for the people who get annoyed by a message in Steam for every new
achievement for jumping from platform a to b (like me).

------
smacktoward
The problem with Steam is Valve.

Valve is famously organized as a place where everybody gets to pick the
problems they want to work on
([https://www.bloomberg.com/news/articles/2012-04-27/why-
there...](https://www.bloomberg.com/news/articles/2012-04-27/why-there-are-no-
bosses-at-valve)). Manually reviewing dozens of (mostly crappy) games every
day in the hope of finding a few that are worth recommending is the kind of
dull, grinding work that nobody would ever pick for themselves to work on.

------
legitster
I don't get what the ask is. Steam should be even more heavy handed? They
should hire people to playtest other people's games? I don't see any
alternative reality where they have already taken these actions and are less
hated for them.

~~~
elthran
I think the author wants to suggest a solution where anyone can submit a game,
but there is still some sort of quality check before it goes on sale.

Currently, we have floods of garbage hitting Steam, with no curation at all.
Pre-Greenlight, Steam was a closed shop that was hard to get into.

What is needed is a healthy balance, that doesn't have zero chances for an
indie game to be seen because it's buried under all the AAA titles, yet
doesn't allow trash like Slaughtering Grounds to be sold.

The article suggests getting Valve employees to playtest any submission -
however, I feel this still wouldn't work due to Valve's 2 main issues. Their
flat structure, where people work on what they want - who wants to look after
content curation? Secondly, they take a cut of every game sold - regardless of
how awful it is. They are financially incentivised to keep subpar (or even
scammy) games on the store.

I'm not going to pretend I have a solution, but I feel all the issues with the
Steam store currently are issues of Valve's own making - the problem is, they
don't (seem to, at the very least) care.

~~~
jessaustin
_Their flat structure, where people work on what they want - who wants to look
after content curation?_

Are you asking, who wants to play video games for pay?

One expects that someone could be found...

~~~
falcolas
Personal Anecdote from a former videogame QA tester and min-spec discoverer:
Playing video games as a job is not playing video games. It's software testing
a GUI.

Basic "does it run on X platform" is even more boring (but more automatable).

------
JamesBaxter
I think you could use Steam Achivements and playtime to recommend games to
people.

If I've played a game to completion on the highest difficulty presumably I
like it. Find me other games that people with the achievement also play and
have the equivalent achievement on and you've got something basic.

~~~
vocatus_gate
Max or close-to-max achievements for a game also is an indicator of skill,
could be used to weight a games scores more towards "recommended by skilled
players" vs. "recommended by casual players" ?

~~~
falcolas
I would have a hard time equating achievement coimpletionism as skill, to me
they are more of a measure of time spent in a game. Yes, some achievements in
some games do require a mediocrum of skill, but most of them do not; they're
used primarily as a form of feedback for the developer on how people play
their game, and secondarily as a lure for completionists who want the points
on their profile.

~~~
dkns
I agree. See for example achievement 'Astronomically Low Odds' from Faster
Than Light:
[http://ftl.wikia.com/wiki/Achievements](http://ftl.wikia.com/wiki/Achievements)

"Fail to evade 5 shots in a row with a full powered and upgraded engine."

It's almost pure luck. Wiki mentions ways that you can increase odds for you
but still it's mainly luck.

------
arca_vorago
Honestly I have been thinking more and more about the problem with steam, and
I realized that even as a foss, anti-drm proponent, steam is one of the few
things I have bended for on those positions because of the quality of the
marketplace. If they continue to fail, I truly hope something like Lutris or
Desura or GoG steps up to the plate and open sources a game store (bonus if
DRM free like GoG is). I mean, does anyone remember the days of hating games
that required you to have a CD in the tray so you would get a noCD warez just
to play? Steam solved those issues, which is what propelled it to it's current
state. I don't think that's a problem with modern games though, so perhaps
steam is weaker than Valve realizes if people ever actually thought about it.
Now it's main function is as a game library stored in a cloud.

That makes me really sad to realize too, because I feel like Valve is one of
the only companies that has the resources to really push linux gaming into a
new zone, but this hasn't been delivered on yet. I highly suspect they are
waiting for Vulkan, but the time can't come soon enough to offer many people
the last reason they need to switch to linux from windows: gaming.

I have also noticed this sort of let the customers do the work for us attitude
far too often lately. For example, I am loving working in Unreal Engine 4, but
they promised a native linux editor and then ingored it for a long time, only
once linux native users submitted enough patches to their own fork(s) before
Epic finally spent some time to push those changes to the official repo. I
think sometimes open source users who want things do work to get them but some
companies abuse this process. In Epic's case, we still don't have a
marketplace launcher on linux, and there are tons of editor bugs, such as I
can't even edit skeletal animations/rigging without crashing the client.

There is a balance to be had between user help and the companies that need it,
but that pendulum seems to have swung far too far over to the user lately, and
it needs to readjust.

------
TorKlingberg
People love to demand that Valve or Google hire lots of people to do manual
curation or phone support. Yet that costs money, and people would buy their
games elsewhere it it's two cents cheaper.

~~~
majewsky
I don't know. There might well be websites or apps where $game_that_i_buy is 2
cents cheaper, but I don't even bother with anything else than Steam. I've
just decided to buy into the ecosystem. The question then is: How common is my
behavior among Steam users (or gamers in general)?

