Research suggests OPPO and HUAWEI cheating benchmarks (Update: OPPO response)

Update, November 2 at 10:54 a.m. ET: OPPO has released a statement to Android Authority regarding claims that the company was caught cheating on benchmarks to inflate the performance scores of its Find X smartphone. OPPO’s statement can be found below:

In the public version of the 3DMark test, the benchmark scores of the Find X are similar to other smartphones that also use the Snapdragon 845. The results all score between 3300 to 4600. Today, the SoCs adopted by mainstream smartphones fully take into account the different needs of users for performance and power consumption, and can dynamically adjust the clock speed. Thus, the tradeoff between performance and power consumption is achieved.

Since 2015, OPPO has been conceiving and promoting system-level resource allocating optimization in order to better meet users’ needs for performance and battery life. In 2016, we extended the system optimization scheme to both Qualcomm and MediaTek platforms. We developed the underlying optimization technology together with chip manufacturing partners.

We want users to be able to get the full performance of the CPU and GPU when they need it, and to reduce system power consumption as much as possible when they do not need it to achieve longer battery life. When we detect the user is running applications like games or running 3DMark benchmarks that require high performance, we allow the SoC to run at full speed for the smoothest experience. For unknown applications, the system will adopt the default power optimization strategy. The specific strategy is — after the user has not actively operated for 5 to 10 seconds, the device limits the system performance to 70% to 80% of the maximum performance (according to different platforms), without affecting the user experience, reduce the standby and operating power consumption of the system. When there is a user operation, it will immediately cancel the performance limit to ensure that the user experience is not affected. So, in a private version of 3DMark, the performance score of Find X will decrease significantly because the phone is using the power optimization strategy at this time. You could validate this optimization strategy by touching the screen within 5 seconds (similar to how you play a game). At the same time, we are working on upgrading the system, and strive to distinguish between the requirements of undetected apps or the subjective needs of users.

Okay, let’s unpack that. OPPO claims its in-house developed performance optimization is the culprit for the disparity between benchmark scores. When the phone recognizes graphic-intensive apps (games/benchmarks), it allows the phone to run at its normal level of performance — not at an overclocked rate. Instead, when the phone fails to recognize a game/benchmark app, it runs in its power-saving mode, hence the lower benchmark scores when the Find X was tested against the unlisted app.

Original article, September 28 at 6:23 p.m. ET: A few weeks ago, we learned that HUAWEI installs software onto its smartphones that detect when a benchmark test is conducted. When a benchmark app runs, the smartphone pushes its processing power over the limit in order to make the score go higher than it ever would in real-life scenarios.

When HUAWEI got called out on this practice, the company admitted that the allegations were true. However, rather than agree to remove the software tweaks, it said it would make the overclocking tweaks available to the end user, which would make its previous benchmark scores valid.

However, the results of a new study by a research team at TECH2 suggests that Oppo is also artificially inflating benchmark scores of smartphones by huge margins using the same technique. What’s more, the team found that several other smartphone manufacturers are not inflating their scores by conducting the same set of tests.

The TECH2 team conducted benchmark scores on the following devices:

• HONOR 10
• HUAWEI P20 Pro
• Nokia 7 Plus
• OnePlus 6
• OPPO Find X
• Pocophone POCO F1
• realme 1
• realme 2
• Redmi Note 5 Pro
• Samsung Galaxy Note 9

Instead of using the UL Benchmark-created 3DMark app that’s available on the Google Play Store to conduct these benchmark tests, TECH2 used a private version of the app that it got from UL. This private app is the exact same app as 3DMark — except the name of the app is different.

Since the name of the app doesn’t match with any recorded benchmark app, a smartphone that would overclock itself in the presence of a benchmark app wouldn’t start the overclock process for this private version of 3DMark. In other words, the private version of 3DMark used by TECH2 will get real-world benchmark scores for a smartphone running a processor at the same rate that it would during any other normal function.

Here are the results for five of the devices. The black bars are the advertised benchmark scores and the yellow bars are the scores TECH2 obtained using the private, unbranded app:

As you can see, there are some wild discrepancies between the advertised scores and the actual, real-world scores. In the case of the HONOR 10, the real-world score is nearly half of the advertised score.

Now, for comparison, here is the same comparison made by performing the same tests on the five other devices:

In the cases above, the advertised scores are almost perfectly in sync with the scores obtained using the private app.

What does this all mean? It proves that two phones from HUAWEI and three phones from OPPO artificially inflated benchmark scores, while one phone from HMD Global, one phone from Samsung, one phone from OnePlus, and two phones from Xiaomi did not. But it also suggests that advertised benchmark scores from some companies are more trustworthy than others.

We’ve reached out to both OPPO and HUAWEI about these results, but we did not hear back from either company before press time. We will update this article accordingly should we receive statements from either company.

NEXT: HUAWEI phones delisted from 3DMark