Futuremark, maker of the 3DMark cross-platform benchmarking app, announced it removed from its scoreboard several Samsung and HTC devices “suspected of breaking [its] rules.”
The devices in questions are the Galaxy Note 3 (Exynos and Snapdragon versions), Galaxy Note 10.1 2014 (Exynos and Snapdragon versions), the HTC One, and the HTC One Mini.
Established in 1997 in Finland, Futuremark is one of the most respected benchmarking software developers, having built a reputation for itself with benchmarking tools for gaming PCs. Earlier this year, the company released 3DMark for Android, iOS, and Windows, as one of the few benchmarks that allows for comparisons between devices on various platforms.
Futuremark did not specify the reasons it delisted the HTC and Samsung devices, but hinted at the practice of detecting the presence of benchmarks and artificially increasing CPU and GPU performance to obtain top scores.
People rely on Futuremark benchmarks to produce accurate and unbiased results. That's why we have clear rules for hardware manufacturers and software developers that specify how a platform can interact with our benchmark software. In simple terms, a device must run our benchmarks without modification as if they were any other application.
In fact, the devices that Futuremark delisted are exactly the same devices that AnandTech accused of cheating 3DMark in a report from October. Samsung, HTC, Asus, and LG were all found to be rigging some benchmarks by AnandTech.
What’s the consequence of this delisting? The devices will appear unranked, and without scores, at the bottom of the 3DMark scoreboard.
In the bigger picture, this kind of public shaming could make manufacturers think twice before tampering with benchmarks. Futuremark is the first benchmark maker to delist devices due to suspicions of rule breaking, but others may follow suit.
Like this post? Share it!
great move :)
My favorite benchmark n.n The place to go to check if a device is worth buying, specially when you are asked to recommend a device.
Good they delisted these devices.
Benchmarks mean nothing. Period. Besides, all car, truck, motorcycle, and the likes thereof do the same thing with torque and horsepower readings. It’s the game they ALL play. So if futuremark wants to delist…go right ahead…it won’t impede my decision whatsoever.
Benchmarks are good for the ones who have learnt to interpret them.
You’ve been downvoted :D
Why down vote? Finland is great!
Isn’t the whole point of doing benchmarks to find out what the full potential of the hardware is capable of delivering in performance?
If yes, then what’s the problem by letting the benchmarks use the speeds that the phone comes default with then?
If i run 3DMark 11 on my computer, it will use the speeds that my GPU and CPU have. So why should benchmarking be different on smartphones than it is on computers?
Because unlike desktop computers, that extra performance is unreachable and practically unusable. It sacrifices battery life for the sake of inflating scores. To make that extra performance available all the time would mean significantly reducing the quoted “talk time” (as if that term matters any more). Sure, you can have top performance, but does it really matter if it only lasts 2 hours before the phone dies?
Benchmarks is not about saving battery at any points. It’s to squeeze every penny out of the hardware at the speeds the hardwares comes with.
My Samsung 9-series laptop (15-inch) have really great batterylife. It lasts for around 10 hours. However, if i run any types of benchmarks apps on it, the battery on it will be gone in 1 hour.
I agree. But the thing is, you actually can access that performance. For the sake of argument, let’s say your laptop is capable of running Crysis on maximum settings at 60 FPS but only allows you to reach that performance when benchmarking. When you play the actual game you only get 45 FPS. That fundamentally changes your purchasing decision. It becomes even more of an issue when dealing with time-sensitive tasks. Phone manufacturers are fixated on benchmarks because they know us geeks use them as purchasing decisions as well as we are the ones who write about them on the Internet. In reality, it’s other things that grab the average user’s attention. Samsung is heavy on software features. HTC focuses on build quality. Apple on the simplicity of its user experience. Blackberry on well… staying afloat. Benchmarks are important but should not be the only deciding factor. I personally use a Note 3 and don’t care that the processor is handicapped for normal use. The large screen size, stylus, AMOLED technology far outweigh the performance “gimp”. The practice of inflating scores is wrong, but that’s only because the performance is not fully accessible by the user at will.
I respectfully disagree, my opinion is that the sole reason for a benchmark is to get high scores. Is it the ultimate factor for a buying decision? Defnitely not.
Suppose I’m in the market for a super sports car and I’m choosing between two. One says it goes to 300km/h, the other goes to 280km/h max speed. Did they use special high octane gasoline? Did they mess around with fuel injection? I don´t know, but the fact is one car was faster then the other. Even though I know I’ll never reach these kind of speeds and I know that it should not be the sole deciding factor of my purchase, it will influence it anyways. Every market category has this kind of “cheating”.
I think delisting those devices for getting higher score because they had higher clocks doesn’t make any sense, but that is just my opinion.
But what if you purchase the 300km/h car and then, no matter what you do, you can’t reach that speed, because it has a governor that caps speed at 250km/h?
What does that 300km/h result mean to you when it is completely impossible to attain in the reality, no matter what you do?
That’s what’s going on with the phone benchmarks.