Mobile benchmarks are a mess, everyone seems to agree. They are imprecise, distort the reality, and fuel the unhealthy obsession with specifications that some in the tech world seem to suffer from. Worse, unscrupulous manufacturers looking for an edge in public perception can easily manipulate them.
British startup Gamebench wants to change benchmarking with a different kind of test, that’s supposed to be impervious to manipulation and better at measuring the often invoked “real world performance”.
Gamebench works differently from conventional benchmarks like AnTuTu or Quadrant, as it doesn’t measure the performance of a synthetic test run over a limited period of time. Instead, the app runs along games and measures several factors that include frames per second and power consumption. The test runs with some of the most popular games in the Play Store, including Real Racing 3, Dead Trigger, and Minion Rush, and the app assigns a final weighted score based on all the measurements. The idea is to offer a better idea of how good a device is in real life, judging from real tests, with an emphasis on gaming performance.
Gamebench is currently in beta, and its developers told Engadget they plan to release the app in the Play Store in the first part of 2014. The team offered a preview of how Gamebench will work, pitching against each other the Galaxy S4 and the HTC One. Both devices are powered by a Snapdragon 600 processor, though Samsung’s is clocked slightly higher. Unsurprisingly, the S4 did better in the FPS measurement, at the cost of increased battery drain, except for the Dead Trigger test, where the S4 was better in both the frame rate test and the battery drain test.
For comparison, a GPU benchmark from the GFXBench suite assigned the two devices the same score.
Besides a more nuanced view at device performance, the Gamebench team claims that the app is “uncheatable”. That’s a bold statement that the company didn’t elaborate, so take it with the proverbial salt.
Back in early October, several manufacturers were found to artificially boost benchmark scores by setting devices to run at full speed when benchmarks were being run. It’s not clear if this technique would affect Gamebench as well. We’ve contacted Gamebench for more info and we’ll update this post as soon as we hear back.