Report: several major Android makers cheat in benchmarks

October 2, 2013
68
96
52 31 13

benchmark cheating cb_agulto

It should come as no surprise, but gaming benchmarks is a common practice in the Android ecosystem.

Anand Lal Shimpi and Brian Klug of AnandTech just released a damning report on the issue, revealing that, with a few exceptions, devices from every major Android OEM are “optimized” to deliver top results in benchmark tests. This involves setting the device’s CPU to top speeds whenever certain benchmarks are running, thus artificially increasing scores. Some device makers apply similar tactics to GPUs.

Since it broke the story about Samsung gaming benchmarks on the Galaxy S4, AnandTech tested a host of devices and found that other manufacturers are, more or less, doing the same thing. Specifically, Asus, HTC, LG have at least one device that exhibits the same behavior, affecting at least one benchmark.

There are a few “clean” devices – AnandTech found no sign of benchmark gaming on the Motorola Razr i, Moto X, Nexus 4, Nexus 7, and the Nvidia Shield.

anandtech benchmark cheating AnandTech

Some devices are worse offenders than others. For instance, the LG G2 and the Asus PadFone Infinity (Snapdragon 800 variant) only kick into high gear when AnTuTu and Vellamo are detected, with five other benchmarks running without interferences. At the other end of the scale, the Galaxy Note 3’s “cheating mode” is active on six out of the seven benchmarks tested.

The shady behavior is independent of the maker of the SoC, so the blame is squarely on the OEMs’ shoulders. As Anand notes, the silly thing about this whole affair is the benefits of optimizing firmware for top benchmark scores are minimal, in most cases just a few percentage points.

The sad takeaway is that, for dubious benefits, some Android manufacturers are willing to mislead customers and the media. For more details, we highly recommend that you read the full report.

Comments