This comes as no surprise. This is a company who has also been caught claiming their phone took a photo - but left in DSLR exif data, claiming a phone had UFS but shipped EMMC instead, boasting to work with dev community - only to lock them out months later, among other shady practices. They make nice phones at a good price, but is not a company to be trusted.
I am not surprised as others have been cought doing the same. Anyone remember the DSLR in the reflection in a Nokia ad that was supposedly shot a Nokia? Have you seen the rig Apple uses? doesn't look much like a phone anymore but technically is still an iPhone.
Sadly the P20 pro has an incredible camera and would stand on its own without forgery.
All I really care about is that they unlock their bootloader. The marketing crap is always half truths anyway. Retina, you mean 4k?
It's all about lighting, and doesn't have to have some complicated rigs. These guys have done a bunch of videos on getting professional shots out an iPhone. For example:
They mount DSLR lenses onto stripped-down iPhone chassis via custom adapters. So it demonstrates what the Sony sensor + Apple software can do, but isn't reproducible in the real World.
Actually doesn't even showcase the apple software. The disclaimer in their own and says additional software is used. So I assume they are postprocessing with color correction and who knows what else.
I don't doubt that is the kind of setup they use, but on this video I see a link to buy the lense advertised but it is not really implied that is what Apple used. Do we have a real knowledge of what hardware they really use for their materials? It would really be interesting to know at least for an information point of view.
> we have decided to delist the affected models and remove them from our performance rankings
I would be inclined to be more aggressive than that. Leave them listed, but add support in the site or other publication system to display "deliberately broke the rules of the test" as a negative, perhaps displaying in a negative looking colour and in any graphs show as a if they scored, say, 20% worse then the worst other score.
If presenting some arithmetically derived overall rating of benchmarks and other properties, balloon this further. Then if the rest of the phone is demonstrably brilliant they still have a hope of not looking terrible, but the cheating hurts the result otherwise.
Unfortunately in these litigious times such an inclination would probably cause me some financial trouble, so it is probably for the best that I don't run a device comparison site! This may be why simply delisting was the chosen way of handling the situation here.
I used to work at an OEM that did this. When the VW Emissions Scandal was publicized, we removed almost every piece of code that did things like these from all our phones.
Not sure if it was ever reverted eventually but performance test might not matter to HN users but it is something that typical consumers do look at when making purchasing decisions, based on our research.
I agree that it was unethical - practices like these are one of the major reasons I left.
It's nearly impossible to be "ethical" in large OEM companies because of the extreme competition and complexity of the organization. Most of these large OEMs are so political that even the smallest changes would require way too much effort.
I switched to Apple after my Nexus 5 stopped getting updates about a year after I bought it.
Bought an iPhone 6S second hand for $320. Still getting updates now, and looks like I will for the foreseeable future.
The core feature iPhones are missing is USB Type C- I won't upgrade to a new iPhone if I cant use the same headphones on my laptop and phone (either 3.5mm or TypeC)
Google phones tend to use whatever the current high end SoC is. So when you see the same SoC scoring lower in a Google phone than on everyone else's phone is it really that the processor is slower, or is it just that Google/AOSP doesn't engage in benchmark detection and you're seeing the true performance of the SoC as a result?
Unlimited backup storage for low-quality, highly-compressed photos. If you want to store photos the way they were shot, you'll quickly run out of space and will have to pay up.
(That's what made me pay Google directly for the first time; extra storage for backing up photos.)
Google's phones (ie: Pixels and their XL variants) get full size photo and video backups included. Even if you don't have a Google phone, the free-of-charge backup only compresses photos over 16MP which is still pretty adequate for most people taking snapshots on their cell phones.
For the hobbyist or pro photographer backing up their full frame RAWs, sure, you want a more professional (read: paid) backup solution but that doesn't seem unfair to me. For the 95% of users who want to free up space taken up by vacation snaps and selfies, compressing photos over 16MP is often acceptable when it doesn't cost extra.
The photos are neither low-quality nor highly-compressed. I personally can't tell much of a difference 95% of the time and I'm a technical person. Here´s a comparison and a closeup, left image is the original, right one compressed with the Unlimited Storage setting:
The first picture has a very noticeable color shift (left is warmer, right is colder) - I am looking at it on a professional 10-bit 4k display which might be a factor, but I would run away from any storage that would cause that :-O Detail looks horrible on both, difficult to say which one is worse and how well do they match. On better photos that would probably be noticeable (my color vision is 100%).
I'm looking at it on a Dell U2715H, which is not a professional display, just a consumer one with above-average color quality and good calibration out-of-the-box, and I could spot the temperature shift immediately as well. Also, as you say, the details are bad.
I'm wary of services that auto-compress images, because a) I want my bits to stay the way they were originally captured, and b) every now and then I have to print a photo I make, and then suddenly all those details matter for the print quality.
Nice analysis and I admit I'm surprised by how good their compression is. I was expecting much worse, and that's why I immediately started paying for storage, so that I don't have to worry about bad-quality photos later.
Yea I kind of agree. I think most people, myself included, just don't really recognize minor performance differences, only relatively large ones. And I've always thought that these things improve so fast year-to-year, that it's kind of pointless to try and keep up. So, i shoot for a phone I like that performs pretty well, and just leave it at that (hello iPhone SE).
They actually do run x times faster. They almost always have faster chips than the previous model. So they’ll pretty much always be faster. Who would release a slower phone as a new model? It doesn’t make any sense.
point is its not as fast as they claim it to be, because they take edge cases as always. I never said the new models are slower, no need to strawman my argument.
> Comparing processors, ram, test scores means nothing compared to the actual experience
But Apple brags endlessly about their A11 "Bionic" chip and similar[0].
Arguing that Apple values experience over specs might have been a valid argument back with Steven Jobs, but Apple's presentations spend significant amounts of time talking about hardware specs these days.
I don't specifically mind that Apple does, but people need to stop pretending like Apple hasn't changed as a company with the shift in leadership.
Apple has always tailored their marketing to whatever their advantage is at that moment. If they have the best tech, they'll talk about speeds and feeds - if they don't, they'll talk about the experience or some other aspect. This is almost as reliable a tick-tock cycle as Intel's.
The A11 chip's CV module is the reason FaceID works as well as it does. And FaceID was their one of their major marketing points of the X and likely the phones being announced this month.
This is just what is expected: the vendors do this because they need to shine in these meaningless benchmarks because that's merely the name of the game. The tech magazines are in between: they publish results in reviews and phone comparisons but it seems that not many people really buy phones based on benchmark scores. They might give some overall idea of whether the phone is middle-end or high-end, but differences within one category tend to be small enough to not matter for the general buyer.
To fix this, there should be different runs for these benchmarks. A max-clocks run with the phone set in an actively cooled cabin to see what the hardware can theoretically do at its very best, and another run in non-actively cooled room temperature with the test running for about an hour to rule out any benefits from temporary boosts or ignoring thermal limits to get an idea of continuously supported performance.
Even those results wouldn't really tell much to the end-user. I always suggest my friends and relatives to buy a phone with excessively large memory and storage space because lack of memory and storage really is what turns phones slow after a few rounds of application and system updates.
The performance edge on mobile is thinning out pretty much what happened with PCs. Ditto for memory and storage. Early on, vendors competed on who has the highest MHz cpus but somewhere between 1-2GHz the performance got high enough in nearly any case. 95% of people could buy about just any PC or laptop and it would be "fast enough". There's gigabytes of memory in even the sloppiest laptop these days, and enough SSD to make things fly. The same will happen with phones which makes it impossible to buy a phone that is too slow. At that point components with lower performance and less capacity will become more expensive due to the lack of volumes, that no vendor will bother any longer.
Transparency. Everybody can run the benchmark on their own device [1] and check the results, that way they can't deny the results saying that it's malicious, a conflict of interest or a configuration error.
Have you ever thought about what the business model is for benchmark companies?
It probably won't come as a surprise to many of you, but there's a major conflict of interest when it comes to benchmark companies. They have "benchmark development programs" which companies pay tens or hundreds of thousands of dollars to join, and in turn the member companies get early access to beta versions of the benchmarks, and get to propose optimizations to the benchmarks.
Now the benchmark companies insist that they make impartial decisions when it comes to what to include or improve in the benchmarks, but you have to wonder just how much influence these member companies have over the benchmark companies, especially when it comes to new features that are in one but not all devices, or features that are not often used by developers but have different performance on different devices, what the decision process is to include tests for those features.
Only if the benchmark results were marketed by Huawei. Otherwise you would be limited to a claim under misleading or deceptive conduct which is still technically possible but you'd have bear the huge expense of taking them to court with the real possibility that you might lose and be bankrupted.
>however, when an unlabeled version of the benchmark test was run, the phones were unable to recognize it and, as a result, displayed lower performances.
>In other words, the phones aren’t so smart after all.
If I may, also the fact that benchmarks are normally/usually "recognizable" through some "label" doesn't seem "smart" to me.
If something is cheap in terms of the price you pay for it, you're going to pay by something else. Be it poor quality components, information collection or unethical manufacturing processes. Yes, of course I am generalizing. There are good cheap(er) products and bad expensive ones.
AFAIK Chinese companies make money (in China) with the software they ship on their phones, and the associated Internet services. They all have a slew of what we'd call bloatware, but apparently Chinese people expect it and enjoy it for it being "free stuff".
I don't really know what's their roadmap for Western markets considering nobody wants to let off their Google world. Even though the cynics among us will placate it as the Party bankrolling it for spying.
Huawei caught cheating at a meaningless 'test' nobody should care about.
I spoke a fair bit about performances with a Googler in charger of it. Apparently he is appalled that OEMs have been optimizing their phones in order to score high on these benchmarks.
They don't reflect real use at all and are not what should be optimized against.
In the end, he documented the work he did on the various terminals he worked on so OEMs could do what they should have been doing for years..
Are those days gone? Benchmark-specific hacks in graphics drivers, I mean, not AGP. :P
Also all the outrage when people discovered that a Radeon 9600 was just a Radeon 9800 running in crippled mode, and that you could bridge the tracks and re-enable it in 9800 mode. (Or something similar, anyway, that was a long time ago... :P )
It was the 9500 that could be modded into a 9700. Early versions could be softmodded, this was disabled in later revisions which could only be modded by soldering the additional rendering pipes back on. No idea why I remember this useless piece of information.
Also on a completely different note, the 9700 Pro which was the flagship card of that generation, cost about $300. Times have changed.
$399 was the price on launch in 2002, which is around $550 adjusted for inflation, current flagship consumer GPU from Radeon is the Vega 64 priced at $599 RRP. Times haven't changed as much as you might think
Thanks! I wasn't in the market for one at the time (I don't think I had a job yet) but remember the furore. The idea that someone would sell hardware that deliberately performed below its potential was shocking to a lot of people. I guess it still is now, for instance the Tesla "60kWh" cars which actually had 70kWh packs and had extra range enabled with an OTA update ahead of a hurricane.
> For the Huawei case, the rules are actually a little fuzzy. Phones are permitted to adjust performance based on workload, which results in peaks or dips in performance for different apps, but they are not permitted to hard-code peaks in performance specifically for the benchmark app. Huawei reportedly claimed that the peak in performance seen during the run of the benchmark app was an intuitive jump determined by AI; however, when an unlabeled version of the benchmark test was run, the phones were unable to recognize it and, as a result, displayed lower performances.
As someone who spent a ton of time in the smartphone GPU/CPU space from '09-'13 we saw every single vendor do this.
They'll happily ignore thermals, run custom shader microcode and any other tricks that'll boost their benchmark scores on common apps.
We always kept a set of internal benchmarks that we didn't share with vendors and used that as a part of SoC evaluation. It had the benefit of not being gamed and also lined up with our normal workloads so we had a decent idea of where we expected the whole system to land.
Apple plays this game, but in a different way. They know that apps like Geekbench test for only a short duration which won't necessarily hit their thermal throttling if the phone is cool enough. Detecting 100% load for a few dozen seconds and giving it full performance isn't hard, that is what Huawei and other vendors should have been doing.
Well, short high load sounds like typical usage for some smartphone tasks. Like open a browser, load heavy website, check something and shutdown. If phone can boost CPU, I can load website faster. And if benchmarks want to test for a prolonged duration, surely they can just run longer?
Nope, not implying that at all. For some games that may make sense, but I think that'd fall under the sustained heavy CPU/GPU usage category for most games. Same for compiling large apps, interactive HTML5 PWAs and such.
Intel has been doing this for ages. It may even be the primary reason why they came up with Turbo Boost. It's gotten a lot worse these days, as Turbo Boost can add another Ghz on its own to the base clock, which allows Intel to keep base clocks very low, yet still claim its chips are as fast as if they always run at the peak Turbo Boost speed all the time (and that turbo boost running time has been shrinking every new generation, too).
Probably because they don't need to. Their mobile chips are so ludicrously far ahead of everyone else in the world, it seems possible they discovered a crashed UFO and integrated alien technology into their designs.
Shader replacement was extremely popular for years - see AMD's infamous Doom3 shader replacement: https://www.anandtech.com/show/1488/6 . NVidia drivers did the same thing although they were (generally) a bit smarter about it, relying on shader fingerprinting rather than application fingerprinting to prevent embarrassing "rename the EXE" expose articles. They were notoriously caught using a benchmark's camera path as a fingerprint: https://www.extremetech.com/computing/54154-driver-irregular... .
GPU drivers are still loaded with such replacements, either to work around bugs or to optimize games where a developer partnership was impossible or otherwise rejected. I think that with the shift to lower-level GPU APIs this practice is beginning to fade, but it's basically been par for the course since programmable shaders became a thing.
There's absolutely nothing wrong with shader replacement. Quite the opposite, as it shows the OEM cares about performance. They have two options: convince the game developer to use a better optimised shader or just replace it themselves. The latter is obviously better.
The latter is obviously better? Shader replacement is fairly fragile and circumvents whatever QA process the game developers had in place - if the shader could be incorporated into the actual product I would think it would be better than shipping thousands of patches.
However, I agree that shader replacement is great for the consumer especially in situations where developers have agreed to vendor lock-in/specialization agreements. It's only questionable to me in the sense of benchmarking applications.
Doom 3 was selling title for Nvidia back in the days. Carmack and Co. didnt wanted to do any optimizations for ATI cards because it wont favor Nvidia cards and they had a contract with them back then.
TLDR: ATI did the same thing with Quake 3 Arena which was commonly used as a benchmark game for testing new video cards of that era and which many game news sites were using.
That's not an entirely accurate TLDR here as what ATI did back then is now considered standard practice for GPU drivers and is welcome. It is a bit different than the mobile space as GPU drivers are doing game-specific optimizations rather than benchmark-specific ones. The end user experience is a net win for the major titles, and as long as there's no visual cost it's considered fair game.
It did go too far, even to the extreme of Nvidia hardcoding in their GPU driver the exact camera path that 3dmark took ( https://www.extremetech.com/computing/54154-driver-irregular... ), which is more like what the mobile space is grappling with now. That was solved through a combination of scolding and by just changing the benchmarks to be what people actually do. You don't see 3dmark show up in GPU reviews much, you instead see performance results in a handful of major games. If mobile follows suit and shifts to doing things like benchmarking the average/90th/99th percentile of a game of Fortnite or something then these cheating practices become less feasible as a whole.
> what ATI did back then is now considered standard practice for GPU drivers and is welcome.
Yep. That was the problem at he time. There were no shaders/etc so you’d just taking a demanding game and see how good it ran. No one optimized specific games in their drivers.
So what ATI did made it appear the GPU was stronger on average that it was.
If it was disclosed it would have been a feature and might have been welcomed. But it was secret.
I remember seeing discussions about whether the kind of optimizations we see today are cheating. They sort of are. But why wouldn’t I want the faster frame rates?
I forgot about the nVidia “micro optimization for a specific benchmark” one. No good excuse there because no one actually plays it.
> So what ATI did made it appear the GPU was stronger on average that it was.
Well, not really. It really did run Quake 3 at the speed the benchmarks were reporting. The problem was it got that speed at the expense of a drop in image quality, and a somewhat substantial one at that.
But if you launched the game and just played it you did get the exact same performance that the benchmarks were reporting.
> I remember seeing discussions about whether the kind of optimizations we see today are cheating. They sort of are. But why wouldn’t I want the faster frame rates?
Desktop GPU optimizations or mobile ones? Desktop GPU optimizations these days are not considered cheating because of 2 key points: 1) it improves the actual game experience (this is why reviews focus on game performance now) and 2) it doesn't impact image quality (periodically you'll see image quality comparisons to ensure both sides are still being honest)
The mobile things that Huawei are doing though are cheating because it does not translate into improvements in actual user experience or gameplay. It's tuning the thermal & power consumption to hit the throttling point at the exact duration of the benchmark. This doesn't translate in any way to any real-world improvements. Thus, cheating.
Reminds me of the time way back where Futuremark confirmed Nvidia was cheating on synthetic benchmarks by doing something similar except in reverse where they would degrade texture quality and use other tricks to artificially inflate scores in 3dmark. I'm certain ATI was caught doing so as well at some point.
Shader replacement and fingerprinting is par for the course in the GPU industry. The ethics are arguable when it comes to applications that aren't benchmarks - is it really wrong for the vendor to apply post-facto optimization to give end-users a better experience? Certainly, though, in the case of benchmarks it's a slimy practice.
It is now, and because it improves things for gamers it’s hard to argue with.
Quack.exe was for Quake 3, Before shaders even existed. At that time drivers didn’t have special paths for certain games, which is what made it a scandal.
If ATI had said they were doing it, perhaps it would have been fine. But they hid it and looked like they were doing something underhanded (true or not).
What exactly does "adjust performance" even mean? If it's something like frequency throttling, there should be a user setting to force maximums, even at the expense of battery life, and likewise another option to force minimums.
No mobile device on the market runs at the same clock speed for any length of time. They’re always scaling down the clock speed for a variety of reasons (usually battery life and thermal management). If you ran at max speed all the time, you’d have trash battery life and probably cause overheating.
Each device has a pretty unique thermal profile(tablets dissipate heat better than small phones, more metal = better profile, etc) so these things tend to get tuned on a per-device basis.
We had one device I was helping bring up where you could run the GPU full-tilt for ~60s. After that the device would reboot and fail to come up until it had cooled enough. Was a fun process characterizing that device and our workloads in 60s bursts. Couple weeks later they had it tuned in where you could run it 24/7 as the thermal limits kicked in pretty aggressively.
If you enjoy a phone that runs hot, but only for a few minutes, depending on whether an empty battery or thermal protection kicks in first, sure.
Power management and thermal budgets are a very real thing. The times where all device’s components would run at full speed are (thankfully) over for quite some time.
20 years ago PCs had multiple incredibly loud constantly running fans, that I think hardly anyone would put up with anymore (and would obviously not work for phones anyway). Today’s phones are extremely more powerful than those PCs in many ways, and yet consume a fraction of energy per time unit. Does not work without complex power management.
20 years ago, PCs had much smaller fans than they do today, and the cases usually had no separate fans of their own - the only fans you'd find would be in the power supply and on top of the CPU. And passive cooling for the CPU wasn't difficult to achieve with a big heatsink.
The industry didn't focus much on quietness, that's true.
(I used to assemble and sell custom-built PCs around 1996 to 2000.)
1.4GHz Athlons were a thing around 2000, almost 20 years ago, and had anything but "small fans" to be cooled. I used to build PCs too, and once I pretty much instantly burned up the die of one of those Athlons because I forgot to apply the heat paste.
Even with computers that could be run all passively at that time, there is just no comparison to today's smartphones, which sit in our pockets and are orders of magnitude more capable along almost every axis.
I disagree that they are shitty... hardware and software is great. Their ethical standards maybe not.
Besides, if you want to install custom roms just get a phone that is supported officially (e.g. Pixel). I don't care about custom rom scene so it doesn't bother me that the bootlocker is locked. Though it was definitely a nice to have while it still existed.
You don't care about custom ROM scene, but by purchasing Huawei's products when you have plenty of other highly-competitive choices, you are supporting their ethical standards. And thus by proxy you are declaring your support for locked-down hardware and black-box software. As long as you are totally aware and recognize that you do not support the principles of free computing, and that you support companies asserting ownership rights over something after you have purchased it, then I have no problem with your reply. There is no hostile intent with this comment, but I do wonder if you understand what you are saying, when you say that you realize their ethical standards are not great, but you continue to buy their products because it currently does not affect you, despite having plenty of excellent alternatives in this market. Incidentally, I think you have a cool username.
