Huawei is one of the strong players in the smartphone industry. They have even surpassed Apple in shipments to become the second largest smartphone manufacturer in the world. With all that great things happening, they’ve been getting into a lot of controversies lately.
ALSO READ: Here Is How Samsung Is Self Destructing Itself [TDAnalysis]
The most recent one being, a number of smartphones from Huawei and its sub-brand Honor cheating on benchmark scores in order to obtain an upper hand over their competitors, at least on paper. The whole thing was brought out by reporters of AnandTech and the response they got from Huawei officials were rather interesting. Let’s take a deep look into what exactly happened here.
What Actually Happened?
As you all know, Huawei and Honor released a slew of devices powered by their flagship Kirin 970 SoC this year, that too in various price points. Huawei P20 Pro, P20, Nova 3, Honor Play to name a few. On testing these devices, specifically to evaluate the GPU Turbo capabilities, AnandTech noticed some anomalies with popular GPU benchmark software – 3DMark and GFXBench.
On further testing, they found that new Huawei/Honor devices were employing an underlying algorithm which detects and whitelists certain benchmarking applications and breaks the thermal and power limits set for normal use cases while running benchmarks. So the SoC would perform consistently at its maximum potential while running benchmarks resulting in a higher score, but at the expense of more heat and power draw.
That is not the case in real-world scenarios. When the SoC is pushed by normal applications, it performs to its maximum until the temperature hits a certain point. After which the clock-speeds decrease to keep the temperature and power usage under control. The efficient throttling of SoC is extremely important in handheld devices like smartphones which use batteries as a power source.
So, removing such restrictions just for benchmark software will result in a better score, making at least some potential customers think that it would translate to real-world usage and make their purchase decision based on that.
ALSO READ: Huawei Kirin 980 Vs Qualcomm Snapdragon 845: What’s The Difference?
AnandTech did the comparison using an insider version of benchmarking software which could bypass Huawei’s detection mechanism and work just like any other normal application. The difference in results was astonishing and was more prominent in budget devices like Honor Play.
3DMark Graphics Scores (Image Credits: AnandTech)
In that respect, comparing and making claims based on benchmark scores obtained under different testing conditions in launch events and advertising materials are unethical and outright misleading. You can check out the detailed benchmark comparisons here.
Not A New Thing!
Altering CPU and GPU performance just for benchmarking software and obtaining higher scores is not a new thing. It dates well back to the dawn of benchmarking and even the PC market was once facing it. On the smartphone side of things, OnePlus, Samsung and many other companies were accused of benchmark cheating in the past.
According to AnandTech who have a great history of exposing benchmark cheatings, such activities have been slowing down recently as more and more people became aware of such issues and understood the meaning of benchmark scores. But, it is still happening out there in the wild as proved by the new reports accusing Huawei.
ALSO READ: What Is Huawei GPU Turbo Technology?; Smartphone Gaming Revolution?
Huawei’s Response And The Events Followed
When AnandTech contacted Huawei officials to comment on this, the response they got was really interesting.
On responding to AnandTech, Huawei’s President of Software, Dr. Wang Chenglu stated that synthetic benchmarks were moving further away from real-world usage and don’t show the full experience. He also said, Huawei is doing this because a number of other manufacturers were doing the same in China, and they had no option but to follow.
Though that kind of response from the world’s second largest mobile manufacturer would definitely be frowned upon, Dr. Wang admitted that there is a need for standardization on benchmarks. Later, Huawei issued an official press release saying, they’re planning to introduce a “Performance Mode” which users can choose if they are ready to trade in lower temperatures and better battery life for that extra performance.
ALSO READ: Are Gaming Smartphones Really Worth It? [Explained]
In light of current events, 3DMark delisted Huawei P20 Pro, Nova 3 and Honor Play from their database. The company stated that benchmarks are meant to run just like any other applications. Latest reports say that Huawei has reached out to the creators of 3DMark and discussed the matter. As a result of the discussions, Huawei ensured transparency in the way benchmarks run on their devices.
Benchmarks Are Not Everything
At this point, we as general consumers should understand what benchmarks are for.
Benchmark scores don’t necessarily represent the real world performance. They just give a vague idea of how superior the chip is and how better thermals are managed compared to other devices. Though that will usually translate to real-world usage, software optimization plays a huge role here.
Also, other applications or games might not be utilizing or allocating resources similar to the way the benchmarking application does. That itself will result in a completely different output.
If you remember, some of the deca-core processors from MediaTek absolutely crushed multi-core benchmark scores back in the day but didn’t translate well on to the real world. That’s because the software underneath couldn’t allocate resources efficiently to the ten cores resulting in much worse performance and high temperatures.
Why Companies Do This?
Smartphone manufacturers know that getting higher scores in benchmarks by overthrowing all thermal and power limits are not at all a practical representation, especially in the case of smartphones. That might not be the case with desktops where power is drawn from the outlet and high temperatures can be handled by upgrading the cooler, both of which aren’t possible with smartphones.
But still, manufacturers do this just to get additional marketing points, especially in markets like China, where benchmark scores are considered as an important performance metric. At the moment, this is the only reason that comes to my mind.
ALSO READ: Samsung Teases Foldable Smartphone Launching Later This Year!
Conclusion
Huawei is getting into a lot of controversies lately. Just a few weeks ago, they were accused of using pictures taken with a DSLR to advertise the Nova 3 selfie camera. Now, the benchmark cheating issue is just adding fuel to the fire. At this point, the performance claims made by Huawei at the Kirin 980 announcement should be taken with a pinch of salt since we don’t know which metric Huawei used to evaluate the chipsets.
Though this was not expected from a leading smartphone manufacture
