
Mobile Benchmark Cheating as a Service - uluyol
https://www.anandtech.com/show/15703/mobile-benchmark-cheating-mediatek
======
drewg123
Towards the end of the article, they talk about high performance modes added
by manufacturers and say _" The phone essentially goes into a high-power mode
throwing away any attempt to be efficient; it’s a nonsensical mode that is
unusable in every-day use-cases beyond getting high benchmark scores"_

What makes it "nonsensical" ? Are they exceeding the TDP of the phone? If so,
then couldn't benchmarks just be made much longer, so that the phones would
throttle or fail if they were cheating?

Or is it just about battery life? If so, why is it unreasonable to have a
phone plugged in to get great gaming performance?

~~~
codeflo
The problem is basically a variant of Goodhart's law: "When a measure becomes
a target, it ceases to be a good measure."

Publications want to use these benchmarks as a convenient proxy for real-world
performance. Phones have "benchmark detection" and run their power management
with different settings when they detect that a benchmark is running,
defeating the purpose of running the benchmark.

BTW, I don't think this is about gaming vs. non-gaming workloads (there are
different benchmarks for those), or even running plugged-in vs. on battery. My
understanding is that some of these "benchmark modes" are so aggressive that
the phones would become too hot to hold in your hands. No sane phone
manufacturer could use settings like these for real-world usage.

~~~
lonelappde
Reviewers should simply say, "we've adjusted these numbers to account for
benchmark manipulation", and give those phones worse reviews.

------
DeathArrow
Most popular benchmarks used on mobiles are pretty much flawed.

I.e.Geekbench is highly optimized for Arm but it doesn't use AVX2 on PC,
making ARM seem much better than it is.

Geekbench runtime is too small. Good laptop reviewing websites like
notebookcheck use many rounds of Cinebench R15 to test if scores go down due
to thermal throttling and by how much and after what time.

Something similar can be done on mobile phones instead of just running
Geekbench once and writing down the number.

Yes, some manufacturers are trying to cheat but benchmarks used are bad and
many reviewers do a poor job.

------
pjc50
There's a long history of this kind of nonsense:
[https://techreport.com/review/3089/how-atis-drivers-
optimize...](https://techreport.com/review/3089/how-atis-drivers-optimize-
quake-iii/)

(Drivers detect the name of the executable as QUAKE.EXE and behave
differently!)

~~~
jimbob45
That was a really interesting blast from the past.

------
fxtentacle
Isn't that kind of exactly what GPU companies have been doing all the time
with game-optimized drivers? And since they cannot just skip work, but merely
re-prioritize things, these cheats should be easily detectable by a reduced
battery life.

~~~
ReverseCold
> game-optimized drivers

These generally actually make the games you're playing have somewhat better
performance (via game-specific hacks). Applying tweaks to a mobile phone
benchmark, but then not applying those same tweaks to other apps is just
misrepresentation of actual performance.

