
How Samsung inflated its performance scores - antimora
http://www.theguardian.com/technology/2013/oct/13/samsung-benchmarking-apps
======
sharpneli
"We've already started work on making sure that all future versions of
benchmarks we get will come with unique package names." \- Anandtech

After that Samsung will just figure out some other identifying feature to
recognize the benchmarks. Naturally we might go overboard and do random
obfuscation on everything, that might solve the issue for the journalists who
use them one off.

However for benchmarks that are published in the Google play store that is
impossible, so there is no real way for benchmark developers to 'immunize'
their products for this sort of approach.

Disclaimer: I am employed in developing Mobile benchmarks, and our product is
something they cheat on. Avoiding devices that cheat is very much in our
interest.

~~~
masklinn
> After that Samsung will just figure out some other identifying feature to
> recognize the benchmarks.

Yep, reminds me of 3DMark2003 when NVidia and ATi "optimized" their drivers by
disabling, altering or swapping the benchmark's shaders for their own or less
complex versions

------
masklinn
> The phone giant has proved that benchmarking apps are vulnerable

The only thing they have proved is their willingness to game benchmarks. ATI
and NVidia were gaming 3DMark 10 years ago.

Also interesting to note, though Samsung is definitely more than willing to
game benchmark despite their protests (and looks like the worst offender by a
_pretty long shot_ ), Anandtech found out they're far from the only company
doing so: [http://anandtech.com/show/7384/state-of-cheating-in-
android-...](http://anandtech.com/show/7384/state-of-cheating-in-android-
benchmarks)

Edit: turns out this was mentioned in the article, but the way it's presented,
what the fuck Guardian?

> Anandtech, a site where you could get the (wrong) impression that benchmarks
> are the only reason to own a gadget

~~~
NatW
Anandtech's Article:

Cheaters: Samsung, LG, Asus, HTC

non-cheaters: Motorola, Google, Nvidia, Apple

See any patterns?

~~~
nyrina
Not really, care to enlighten me?

~~~
CaptainZapp

      Cheaters: Samsung, LG, Asus, HTC
    

Korean companies

    
    
      non-cheaters: Motorola, Google, Nvidia, Apple
    

US companies.

I for one can't help but see a slightly xenophobic pattern in the poster's
comment.

~~~
vetinari
ASUS and HTC are Taiwanese, not Korean.

~~~
CaptainZapp
I stand corrected.

Thanks

~~~
r0muald
Still... Korea+Taiwan vs US.

------
steve19
The funny thing is that despite being caught out doing this a while ago with
the S4, they have no choice but to continue doing it regardless how
embarrassing it becomes. If they don't, their next generation devices may
score only slightly more (or less than) than the current gen devices.

They painted themselves into a corner from which there is no escape. And they
only have themselves to blame.

~~~
bsullivan01
But it's not like the market punished them at all. They are still selling lots
of them.

Apple is the only that held them accountable in court, at least a little it.
These b*stards are shameless [http://cdn.mactrast.com/wp-
content/uploads/2011/09/Samsung-A...](http://cdn.mactrast.com/wp-
content/uploads/2011/09/Samsung-Apple-Cable-Copy.jpg)

~~~
nodata
Your proof that "Samsung are bastards" is because they have a connector that
is either a PDMI connector, or something that looks suspiciously similar?

~~~
rimantas
How is this standard called: [http://media.idownloadblog.com/wp-
content/uploads/2011/09/Sa...](http://media.idownloadblog.com/wp-
content/uploads/2011/09/Samsung-Charger.png) ?

~~~
nodata
A cube?

------
hclee
People still buys. In business sense, it is funny. They cheated and impressed
as much as they can and made really large volume of sale. Media reporting like
this does not seem to make much difference. Other than few
hackers/engineers/programmers, majority of consumers don't give a thing on
benchmarks. Too bad.

------
slacka
The CPU/GPUs keep getting faster but this idiotic PPI race is forcing
manufactures to cheat so they can appear faster than the previous generation.
On a 5" display anything over 720p is overkill.

~~~
masklinn
A number of benchmarks have an "off-screen" mode with fixed-resolution
rendering specifically to avoid native resolution artefacts. That provides
both the absolute hardware power in comparison to other devices and the
device's "feeling", how its hardware meshes together (aka if it has enough
power to drive its own screen). The second one is arguably the more important
one wrt user experience and the ability to actually use the device.

GLBenchmark has both onscreen and offscreen modes, and Anandtech posts both
every time.

------
badman_ting
All I can think is, imagine if Apple did this.

------
bsullivan01
Samsung is one shady company. They get away with murder in South Korea, given
that they make up a huge chunk of SK's GDP.

[http://en.wikipedia.org/wiki/Lee_Kun-
hee#Samsung_scandal](http://en.wikipedia.org/wiki/Lee_Kun-hee#Samsung_scandal)

They do make decent electronics, but I wouldn't trust anything they say.

~~~
belorn
So the chairman was found guilty of financial wrongdoing and tax evasion.

Tax evasion is indeed serious business, but murder? I guess I would like to
see several large business CEO's charged with facilitating murder by tax
evasion. Would make for some interesting times, and likely tax cuts for the
general public.

~~~
ceejayoz
To "get away with murder" is a very common way of saying someone is
successfully avoiding the consequences of their actions.

