
Huawei caught cheating performance test for new phones - tango24
https://techcrunch.com/2018/09/06/huawei-caught-cheating-performance-test-for-new-phones/
======
axaxs
This comes as no surprise. This is a company who has also been caught claiming
their phone took a photo - but left in DSLR exif data, claiming a phone had
UFS but shipped EMMC instead, boasting to work with dev community - only to
lock them out months later, among other shady practices. They make nice phones
at a good price, but is not a company to be trusted.

~~~
sschueller
I am not surprised as others have been cought doing the same. Anyone remember
the DSLR in the reflection in a Nokia ad that was supposedly shot a Nokia?
Have you seen the rig Apple uses? doesn't look much like a phone anymore but
technically is still an iPhone.

Sadly the P20 pro has an incredible camera and would stand on its own without
forgery.

All I really care about is that they unlock their bootloader. The marketing
crap is always half truths anyway. Retina, you mean 4k?

~~~
PinkMilkshake
> Have you seen the rig Apple uses?

I cant find this, do you have a link? I'm quite interested in what they do.

~~~
dingaling
They mount DSLR lenses onto stripped-down iPhone chassis via custom adapters.
So it demonstrates what the Sony sensor + Apple software can do, but isn't
reproducible in the real World.

~~~
noir_lord
They'd have to be careful in some EU countries with that trick.

------
dspillett
> we have decided to delist the affected models and remove them from our
> performance rankings

I would be inclined to be more aggressive than that. Leave them listed, but
add support in the site or other publication system to display "deliberately
broke the rules of the test" as a negative, perhaps displaying in a negative
looking colour and in any graphs show as a if they scored, say, 20% worse then
the worst other score.

If presenting some arithmetically derived overall rating of benchmarks and
other properties, balloon this further. Then if the rest of the phone is
demonstrably brilliant they still have a hope of not looking terrible, but the
cheating hurts the result otherwise.

Unfortunately in these litigious times such an inclination would probably
cause me some financial trouble, so it is probably for the best that I don't
run a device comparison site! This may be why simply delisting was the chosen
way of handling the situation here.

------
foobaw
I used to work at an OEM that did this. When the VW Emissions Scandal was
publicized, we removed almost every piece of code that did things like these
from all our phones.

Not sure if it was ever reverted eventually but performance test might not
matter to HN users but it is something that typical consumers do look at when
making purchasing decisions, based on our research.

~~~
maxerickson
Your second paragraph is completely ethically bankrupt.

That a lie increases sales is at the top of the list of bad reasons to lie to
customers (and they are all bad reasons).

~~~
orf
> That a lie increases sales is at the top of the list of bad reasons to lie
> to customers

I think that in the society we've built it's actually one of the best reasons
to lie to customers. Advertising is pretty much built on that.

~~~
maxerickson
Replying to that piece of text out of context is really boring.

~~~
orf
I'm not sure how it's out of context, your entire comment was "lying to
customers to increase sales is bad!".

------
EZ-E
I'm gonna get down voted but that's why I buy Apple phones. Comparing
processors, ram, test scores means nothing compared to the actual experience

~~~
drampelt
And that's why I buy Google phones. The processor being slightly slower
doesn't matter to me if I prefer the overall experience

~~~
bitL
If you don't mind being stuck storage-wise with no SD card slot... 4k videos
fill up internal storage pretty quickly.

~~~
euyyn
When you buy a Google phone you get unlimited backup storage in Google Photos.
You never think again about things like that.

~~~
TeMPOraL
Unlimited backup storage _for low-quality, highly-compressed_ photos. If you
want to store photos the way they were shot, you'll quickly run out of space
and will have to pay up.

(That's what made me pay Google directly for the first time; extra storage for
backing up photos.)

~~~
flatb
The photos are neither low-quality nor highly-compressed. I personally can't
tell much of a difference 95% of the time and I'm a technical person. Here´s a
comparison and a closeup, left image is the original, right one compressed
with the Unlimited Storage setting:

full picture
[https://images.huffingtonpost.com/2016-01-01-1451631423-7665...](https://images.huffingtonpost.com/2016-01-01-1451631423-7665610-Beach.jpg)

cropped detail
[https://images.huffingtonpost.com/2016-01-01-1451631515-2060...](https://images.huffingtonpost.com/2016-01-01-1451631515-2060334-BeachCrop.jpg)

Also, I believe Photos storage is unlimited even in Original quality if you
own a Pixel phone, which may or not may be the case, but still.

~~~
bitL
The first picture has a very noticeable color shift (left is warmer, right is
colder) - I am looking at it on a professional 10-bit 4k display which might
be a factor, but I would run away from any storage that would cause that :-O
Detail looks horrible on both, difficult to say which one is worse and how
well do they match. On better photos that would probably be noticeable (my
color vision is 100%).

~~~
TeMPOraL
I'm looking at it on a Dell U2715H, which is not a professional display, just
a consumer one with above-average color quality and good calibration out-of-
the-box, and I could spot the temperature shift immediately as well. Also, as
you say, the details are bad.

I'm wary of services that auto-compress images, because a) I want my bits to
stay the way they were originally captured, and b) every now and then I have
to print a photo I make, and then suddenly all those details matter for the
print quality.

~~~
euyyn
It's a non-issue with a Google phone (a Pixel) in any case, as you have
unlimited storage of photos in their original quality.

------
yason
This is just what is expected: the vendors do this because they need to shine
in these meaningless benchmarks because that's merely the name of the game.
The tech magazines are in between: they publish results in reviews and phone
comparisons but it seems that not many people really buy phones based on
benchmark scores. They might give some overall idea of whether the phone is
middle-end or high-end, but differences within one category tend to be small
enough to not matter for the general buyer.

To fix this, there should be different runs for these benchmarks. A max-clocks
run with the phone set in an actively cooled cabin to see what the hardware
can theoretically do at its very best, and another run in non-actively cooled
room temperature with the test running for about an hour to rule out any
benefits from temporary boosts or ignoring thermal limits to get an idea of
continuously supported performance.

Even those results wouldn't really tell much to the end-user. I always suggest
my friends and relatives to buy a phone with excessively large memory and
storage space because lack of memory and storage really is what turns phones
slow after a few rounds of application and system updates.

The performance edge on mobile is thinning out pretty much what happened with
PCs. Ditto for memory and storage. Early on, vendors competed on who has the
highest MHz cpus but somewhere between 1-2GHz the performance got high enough
in nearly any case. 95% of people could buy about just any PC or laptop and it
would be "fast enough". There's gigabytes of memory in even the sloppiest
laptop these days, and enough SSD to make things fly. The same will happen
with phones which makes it impossible to buy a phone that is too slow. At that
point components with lower performance and less capacity will become more
expensive due to the lack of volumes, that no vendor will bother any longer.

------
komaromy
I don't see why an earnest benchmarking firm would let manufacturers know
details like the name of the testing app.

~~~
halflings
Transparency. Everybody can run the benchmark on their own device [1] and
check the results, that way they can't deny the results saying that it's
malicious, a conflict of interest or a configuration error.

[1]
[https://play.google.com/store/apps/details?id=com.glbenchmar...](https://play.google.com/store/apps/details?id=com.glbenchmark.glbenchmark27)

------
thisisit
My first reaction was reaction isn't this a dupe. But then the earlier one was
about Huwaei's camera cheat:

[https://news.ycombinator.com/item?id=17805027](https://news.ycombinator.com/item?id=17805027)

------
wenbert
Can these be grounds for refund?

~~~
Joakal
In Australia, can return for a full refund that it's not working as
advertised.

~~~
sjwright
Only if the benchmark results were marketed by Huawei. Otherwise you would be
limited to a claim under _misleading or deceptive conduct_ which is still
technically possible but you'd have bear the huge expense of taking them to
court with the real possibility that you might lose and be bankrupted.

------
k_sze
How ironic that Huawei uses the word "Honor" in some of its models.

~~~
baybal2
Well, they sure missed the dis* prefix

------
jaclaz
>however, when an unlabeled version of the benchmark test was run, the phones
were unable to recognize it and, as a result, displayed lower performances.

>In other words, the phones aren’t so smart after all.

If I may, also the fact that benchmarks are normally/usually "recognizable"
through some "label" doesn't seem "smart" to me.

------
da_murvel
If something is cheap in terms of the price you pay for it, you're going to
pay by something else. Be it poor quality components, information collection
or unethical manufacturing processes. Yes, of course I am generalizing. There
are good cheap(er) products and bad expensive ones.

~~~
Bayart
AFAIK Chinese companies make money (in China) with the software they ship on
their phones, and the associated Internet services. They all have a slew of
what we'd call bloatware, but apparently Chinese people expect it and enjoy it
for it being "free stuff".

I don't really know what's their roadmap for Western markets considering
nobody wants to let off their Google world. Even though the cynics among us
will placate it as the Party bankrolling it for spying.

------
chrisper
That's a shame. I got a P20 Pro and I really like it (even the software).

But I don't think I'll get another Huawei next time, because the company
itself is really weird.

~~~
yjftsjthsd-h
Using "weird" as a euphemism for unethical, or some other problem?

~~~
chrisper
Weird as in they keep doing shady things even though they don't need the shady
practices.

------
on_and_off
Huawei caught cheating at a meaningless 'test' nobody should care about.

I spoke a fair bit about performances with a Googler in charger of it.
Apparently he is appalled that OEMs have been optimizing their phones in order
to score high on these benchmarks.

They don't reflect real use at all and are not what should be optimized
against.

In the end, he documented the work he did on the various terminals he worked
on so OEMs could do what they should have been doing for years..

------
ezoe
It reminds me of the old GPU vendors when we were still using AGP bus(or pre-
AGP era I forgot). Good evil days.

~~~
taneq
Are those days gone? Benchmark-specific hacks in graphics drivers, I mean, not
AGP. :P

Also all the outrage when people discovered that a Radeon 9600 was just a
Radeon 9800 running in crippled mode, and that you could bridge the tracks and
re-enable it in 9800 mode. (Or something similar, anyway, that was a long time
ago... :P )

~~~
cehrlich
It was the 9500 that could be modded into a 9700. Early versions could be
softmodded, this was disabled in later revisions which could only be modded by
soldering the additional rendering pipes back on. No idea why I remember this
useless piece of information.

Also on a completely different note, the 9700 Pro which was the flagship card
of that generation, cost about $300. Times have changed.

~~~
Geforce8472
$399 was the price on launch in 2002, which is around $550 adjusted for
inflation, current flagship consumer GPU from Radeon is the Vega 64 priced at
$599 RRP. Times haven't changed as much as you might think

~~~
opencl
AMD's flagship is the $700 Vega 64 LC. NVIDIA's flagship 1080 Ti is $700 and
their soon-to-launch 2080 Ti is $1000 or $1200 for the Founders Edition.

------
lysp
> For the Huawei case, the rules are actually a little fuzzy. Phones are
> permitted to adjust performance based on workload, which results in peaks or
> dips in performance for different apps, but they are not permitted to hard-
> code peaks in performance specifically for the benchmark app. Huawei
> reportedly claimed that the peak in performance seen during the run of the
> benchmark app was an intuitive jump determined by AI; however, when an
> unlabeled version of the benchmark test was run, the phones were unable to
> recognize it and, as a result, displayed lower performances.

Source code for their AI engine:

if (app == benchmark) { increasePerformance(); }

~~~
userbinator
What exactly does "adjust performance" even mean? If it's something like
frequency throttling, there should be a user setting to force maximums, even
at the expense of battery life, and likewise another option to force minimums.

~~~
Shank
No mobile device on the market runs at the same clock speed for any length of
time. They’re always scaling down the clock speed for a variety of reasons
(usually battery life and thermal management). If you ran at max speed all the
time, you’d have trash battery life and probably cause overheating.

~~~
vvanders
Each device has a pretty unique thermal profile(tablets dissipate heat better
than small phones, more metal = better profile, etc) so these things tend to
get tuned on a per-device basis.

We had one device I was helping bring up where you could run the GPU full-tilt
for ~60s. After that the device would reboot and fail to come up until it had
cooled enough. Was a fun process characterizing that device and our workloads
in 60s bursts. Couple weeks later they had it tuned in where you could run it
24/7 as the thermal limits kicked in pretty aggressively.

------
lamarpye
This is a change from their normal business practices, usually they would
steal performance results and publish them as their own.

------
fapjacks
Yeah, Huawei also sells shitty phones with locked bootloaders. Not interested
in any case, cheating or no.

~~~
chrisper
I disagree that they are shitty... hardware and software is great. Their
ethical standards maybe not.

Besides, if you want to install custom roms just get a phone that is supported
officially (e.g. Pixel). I don't care about custom rom scene so it doesn't
bother me that the bootlocker is locked. Though it was definitely a nice to
have while it still existed.

~~~
fapjacks
_You_ don't care about custom ROM scene, but by purchasing Huawei's products
when you have plenty of other highly-competitive choices, you _are_ supporting
their ethical standards. And thus by proxy you are declaring your support for
locked-down hardware and black-box software. As long as you are totally aware
and recognize that you do not support the principles of free computing, and
that you support companies asserting ownership rights over something after you
have purchased it, then I have no problem with your reply. There is no hostile
intent with this comment, but I do wonder if you understand what you are
saying, when you say that you realize their ethical standards are not great,
but you continue to buy their products because it currently does not affect
you, despite having plenty of excellent alternatives in this market.
Incidentally, I think you have a cool username.

