Mobile benchmarks are a mess, everyone seems to agree. They are imprecise, distort the reality, and fuel the unhealthy obsession with specifications that some in the tech world seem to suffer from. Worse, unscrupulous manufacturers looking for an edge in public perception can easily manipulate them.
British startup Gamebench wants to change benchmarking with a different kind of test, that’s supposed to be impervious to manipulation and better at measuring the often invoked “real world performance”.
Gamebench works differently from conventional benchmarks like AnTuTu or Quadrant, as it doesn’t measure the performance of a synthetic test run over a limited period of time. Instead, the app runs along games and measures several factors that include frames per second and power consumption. The test runs with some of the most popular games in the Play Store, including Real Racing 3, Dead Trigger, and Minion Rush, and the app assigns a final weighted score based on all the measurements. The idea is to offer a better idea of how good a device is in real life, judging from real tests, with an emphasis on gaming performance.
Gamebench is currently in beta, and its developers told Engadget they plan to release the app in the Play Store in the first part of 2014. The team offered a preview of how Gamebench will work, pitching against each other the Galaxy S4 and the HTC One. Both devices are powered by a Snapdragon 600 processor, though Samsung’s is clocked slightly higher. Unsurprisingly, the S4 did better in the FPS measurement, at the cost of increased battery drain, except for the Dead Trigger test, where the S4 was better in both the frame rate test and the battery drain test.
For comparison, a GPU benchmark from the GFXBench suite assigned the two devices the same score.
Besides a more nuanced view at device performance, the Gamebench team claims that the app is “uncheatable”. That’s a bold statement that the company didn’t elaborate, so take it with the proverbial salt.
Back in early October, several manufacturers were found to artificially boost benchmark scores by setting devices to run at full speed when benchmarks were being run. It’s not clear if this technique would affect Gamebench as well. We’ve contacted Gamebench for more info and we’ll update this post as soon as we hear back.
Like this post? Share it!
the ratio of Performance/power consumption is in favor of the One in most cases, since the performace doesn’t really scale up with clock speed , hmm interesseting, I’m wondering aswell of subpixel has something to do with performance or not
Yeah – but look at the framerate on the one – 19fps on minion rush? Unplayable
median isn’t an average.
median is (max+min)/2 .
maybe the slow parts were on the beginning, and from then it worked fine…
No, median is the centre number: out of [4,7,12] the median is 7.
right. sorry about that. it’s getting late, and i make more mistakes at night (it’s now 1:28), and i finished my degree a very long time ago.
median in this case is usually quite close to the average.
odd that they showed it instead of average. i think it’s the first time i see a media value in benchmarks.
Median values are much more useful. They’re a much more robust statistic, as they aren’t artificially inflated unless the data set is more than 50% contaminated (read – spikes, both negative and positive). Averages can be affected by a single spike or drop in the dataset.
but when there are many samples, the noise is redundant, and so both converge to pretty much the same value, no?
Medians are also not affected by outliers
watch youtube videos and see for yourself….don’t take anything you see as the truth
I’ve played it, on both devices. I can believe the 19fps is accurate.
wow amazing cara memperbaiki hardisk kesalahan sektor
weird because I m not having that low of fps.
“Mobile benchmarks are a mess, everyone seems to agree.” the mess isn’t limited to the mobile space!