On this edition of the Friday Debate, we discuss benchmarks. There’s a bit of a scandal going on in the Android community, following reports that several large manufacturers are artificially boosting their scores in benchmarks. Some say it’s a normal practice, others that it’s downright cheating. Our questions for this week – do benchmarks matter? Do you care that Android OEMs are rigging them? Is the whole issue overblown?
Join us in the discussion, vote in our poll, and sound off in the comments!
This whole benchmark scandal was going to happen. It was just a matter of time. No reviewers really base their final judgments on a device on how well they do in benchmarks, but a lot of people do. Maybe not insofar as the scores themselves, but the graphs that get shown. People see devices like the Note 3, the HTC One, etc are much, much longer (or taller, depending on the graph) than the competitors. So people think that the device is that much better proportionately.
Now here’s the kicker. I don’t think that this should be a scandal and I don’t think people should care at all. Even if these OEMs code it into the OS to kick the processors into high gear, it’s not actually cheating. The chipset the device comes with can only do so well, so why not let it go nuts? Obviously, the Snapdragon 800 is better than last year’s quad cores, so what does it matter if they’re both maxed out for their graphics test? It’ll invariably show the same information at the end. That one is better for graphics than the other.
We already know these aren’t good (or even reasonably accurate) representations of the real world performance of these chipsets and devices. However, we do one thing. If the OS maxes out the CPU for the tests, we know where the ceiling is. If the OS is treating an intense graphics benchmark as any other app, then the CPU will adjust accordingly, which knocks off points and ends up somewhere in the middle. With the CPU maxed out, the scores max out, and we can see just how high these devices can go. In either case, we still don’t know the real world performance, but if we let them max out the CPU, we’ll at least know the performance ceiling. So we may actually walk away from graphics benchmarks with a fact to give people, instead of conjecture like we do now.
So what should the media do? Nothing. Most of us already add the standard “it’s a graphical benchmark which doesn’t show the real world performance of this device” boilerplate in all of our articles. The only difference is when we do it now, it’ll actually be more accurate that it was last week!
We’ve all known for a while that benchmark scores don’t directly translate into real-world performance, and I can’t say that I’m shocked to hear that some manufacturers have been fiddling around to improve their scores. But at least if every manufacturer is optimizing their components and software when it comes to benchmarking, then the playing field is level and the benchmarks still perform their function.
However, optimization is one thing, but overclocking parts only when running benchmarks verges on misrepresentation of a product. People benchmark different devices with similar hardware because clock speeds, other pieces of hardware, and software optimisations can affect the results. But if you’re testing a secretly overclocked device then all comparisons instantly go out of the window, as these levels of performance cannot be obtained in the final product.
There are good reasons why components are locked at certain frequencies, usually for stability, but also to control power consumption and heat levels. In the Galaxy S4’s example, it’s very doubtful that the GPU would be able to run at 533MHz for a long period of time without draining the battery or causing substantially more heat than the stock 480Mhz clock, and consumers are being misled if they expect to see similar levels of performance and the advertised levels of battery life on their own handsets.
Looking at PC hardware for reference, stock and overclocked versions of graphics cards are openly tested and compared, and this lends itself to decent performance comparisons. Attempting to conceal or alter clock speeds based on the testing conditions defeats the point of doing a comparison.
Hopefully this experience will lead to more diligence and accurate testing in the benchmarking media, more scepticism on the part of the consumer, and the rightful calling out of those who attempt to cheat.
It’s a pretty sad move on Samsung’s part, especially since its devices would have apparently outperformed the competition without them gaming the system (just not by as much). If you’re altering the performance just for benchmarking apps then you’re creating a false impression because people will never get that level of performance in the real world. I guess if everyone cheats you’d have a level playing field again, but it would obviously be better if everyone didn’t cheat.
It reminds me of the mileage stats for cars. Manufacturers run them in vacuum conditions that will never be matched in the real world, but because it’s understood everyone does this, the comparison still has some value for consumers, even if it doesn’t reflect real world performance.
I’m not convinced many consumers check up on benchmarks before buying a new phone and you’d have to be really naïve to be shocked by this, but it is still shoddy. Not just putting your hands up and admitting it is actually more annoying, a Samsung spokesman said “This was not an attempt to exaggerate particular benchmarking results. We remain committed to providing our customers with the best possible user experience.” What was it then? A weird decision to optimize for specific benchmarking apps helps customers how?
I suspect I’m like many of us; easily dazzled by bar charts and graphs. Though, it’s nice to see how high these numbers can go, right? We’ve reached the point where its just a matter of further optimizing apps and the Android OS itself. Oh, and perhaps a few manufacturer overlays could be (at last) laid to rest, permanently. Benchmarks, while important – certainly – there are other elements that people truly legitimately care about like like battery life, real world performance, etc.
If you recall the Lenovo K900 we saw with the world’s first Intel Clovertrail+ SoC it scored very high in Antutu, but the phones battery life under those conditions was shockingly poor. My advice – look at the big picture. I highly doubt most consumers look at benchmarks, and I’m not surprised in the slightest that they are ‘gamed’. Technically, they aren’t gamed. Rather, the hardware is pushed to its limit for a short period of time, and its documented. This is akin to an engine being properly tuned for a quarter mile speed test. Of course, the logic nerds and semantic specialists could debate this to no end, but we’ve better things to do. Is it right? Arguably not. Do I care? Nope.
First, maybe I’m too intransigent but I don’t think we should give tech companies a free pass to cheat customers. Because no matter how you look at it, that’s what’s going on: tech companies lie to customers so they can make more money. I know that’s how the world works, but that’s not important.
Also not important:
While phone makers are blameworthy for doing what they do, what I dislike the most about this whole affair is the patronizing attitude of some members of the Android community. I get it – you knew all along that benchmarks are rigged, kudos to you. But that doesn’t mean we should all just ignore the issue now that it’s in the open.
What do YOU think?
Join us in the comments and vote in our poll.
How important are benchmark scores for you when you decide on a device?
Do you care that Android manufacturers cheat in benchmarks?
Like this post? Share it!
Only thing I care about is battery life ._.
In that case you are going to love the upcoming technology within batteries. They are experimenting with powerful batteries that last 3-4 months and take 25-30 seconds to charge.
This stuff has been on for years.. but you still don’t see them implemented in consumer products.. Philips had the tech years ago to charge any device up to 80% in a matter of seconds.. But still you don’t see these things implemented..
But just like 4K UHD televisions, isn’t that largely because the companies do not yet have the ability to mass-produce the batteries?
That might be true.. but do you remember computer harddrives a few years ago. They had the tech to create TB’s in drives but they still went on with drives of 100/160GB or 200GB (not even starting with the MB harddrives). It is because they want to exert whatever they had back then. Look at it now, the past 5 years or so how much TB are they putting on a harddrive? 3 or something. Where are the 5 or 10TB? Some might say no one needs it. Just like the Ghz vs Cache wars of Intel and AMD years ago. The only consumer that really cared about ghz were nerds like some among us. but the avarage consumer just wanted to be able to play games and watch movies :)
Don’t know if you remember the price fixing scams of huge manufacturers on Televisions. So why has Display tech jumped so much in a matter of few years.. until 2000s it was a matter of how large your TV was and the screen frequency rate, now it is how many pixels you can put on a small screen.
True, Nokia is developing a battery capable of recharging by using cell waves…It might take years because they still can’t mass produce it
Can’t say it’s something that bothers me when i’m researching a device but it’s quite reassuring if the device has everything you are after and happens to be near the top of the benchmarking table
It irritates me that they try to hide it. It is an ethical issue. Do I ever bother looking at benchmarks in a review, no. So for that reason when it comes to buying the tech it is a non issue for me. I’m guessing that is what they hope for, but honestly whoopie-d-freakin’-do. If they make a good product it will sell, if it bombs it will be for reasons other than benchmarks.
Only when they get caught doing it.
Benchmarks are good for nothing except boosting egos. If I’m rooted and able to control my CPU you bet I’m going to tweak it so it runs at maximum performance while benchmarking. I know it has no impact (or is probably even detrimental, i.e. battery life) to real wold performance, all I’m concerned about is how big of a number I can get. It seems like Samsung has the same philosophy, and if other companies want to use benchmarks as a selling point it is stupid for them not to do the same.
It irritates me that they lie about it. Not that they do it.
When reviewers makes artificial tests, like calculating the nth fibonacci number, part of the review, it only makes sense that vendors optimizes for these artificial tests.
But they should be open about it, and be honest about their motivations for boosting benchmarks. Preferably a little boost-icon should light up in the statusbar when a benchmark is run, and the device is sped up. It really isn’t cheating to go faster in a speed-test. But it is fraudulent to pretend that you are not.
Speed Test, for me, are a somewhat important, because I’d like to know that the flagship phone I’m about to drop some serious clams, can keep up with the rest of the competition. At the same time I don’t really care after I get it, because it’s a huge waste of time, and dineros to get the fastest phone there is. Besides phones are becoming faster and faster, eventually we’ll be counting the milla-seconds it takes for one phone to out race the other. How about for next year, lets not focus on the speed of our devices, but better connectivity, better battery life, better cameras, and badass curved screens.
I don’t care. As long as the devices really are powerful, they’re cool.
The benchmarks are not the most important feature of a device. There is the version of the Android on it, as well as how much crapware is on it. I’d rather have a slower phone with plain Android than one with lots of customized apps and useless features but a fast processor. My next phone should be one of the Nexus ones, for this reason.
I agree with Bogdan.
I don’t base my decision on benchmarks, but they are interesting to look at, I base it how well can it connect to the network, does the battery last all day, and when I talk to someone are the speakers loud enough, as what Android version its running.
Should Manufacturers optimize hardware and software for benchmarks, only if they are going to be open about and not overclock the processor, unless they are going to sell a overclock device. It is nice to look at benchmark and see where device land especially if you are stuck between two devices.
It’s only a serious issue for me if they turn a last spot performance to the first spot. Not what they did with the note 3 where it was already the top spot and was just boosted like 5%. Also, if the gpu automatically does that in games, it is not cheating.
If manufactures are cheating on benchmark, what else are they cheating on? Since high end smartphones cost about $600-$800 I expect to invest in a company that is honest about their products!
of course, and you should extend it all the way. like apple sells their $2 cables for $20, monster got called out for their supposed “scam” so why isn’t the same thing happening ? those phone accessories ? samsung or others charges like $25-50 – that’s plain highway robbery for a stupid case/ cover.
The purpose of benchmarking should be to evaluate GPU performance and optimization in certain ROMs, especially new alpha/beta builds. Comprehensive tests such as Antutu (however unreliable it has become), serve to test stability.
Although Asus, LG and HTC all run CPUs are their max freq during CPU tests, the fact that Samsung does that in GPU tests as well ruins the whole point of benchmarking. Samsung has turned it into a game, a competition. Neither am I pleased with their security suite, Knox. Next thing you know, Samsung will be ditching Android and taking up Tizen to become the next Apple.
In all fairness, I believe that lying about performence by gaming benchmarks should not be allowed. Just like the Truth in Savings and Truth in Lending acts for banking allow consumers to compare apples to apples to get the best rates, consumers should be able to compare benchmark performance in the same way.
No. I care about specs in general and battery life.
The overclocking of the GPU in the S4 Octa was way over the limit for acceptable, but disabling powersaving isnt that bad. I used to do PC overclocking, with high end watercooling and whatever, benchmarks were somewhat important back then. It was no secret that Nvidia and ATI worked overtime to optimize 3DMark scores. Im ok with that, but overclocking only in benchmarks is in NO WAY ACCEPTABLE, thats cheating.
I may add that it was common practice to disable EIST and stop services and so on when doing PC benchmarks, a lot of the famous benchmarkers over at extremesystems.org did that and more, running at a bare minimum with liquid nitrogen to squeeze every possible microsecond out of SuperPi. In no way were those systems stable for anything else than, they could not even run multithreaded benchmarks without crashing, those rigs were specially built just to run SuperPi for less than 10 seconds. We used to call it “Pi-stable clock”.
I don’t care about benchmarks, all I care about is real life performance, no lags and battery life.. the balance of the software and hardware
Well, i can’t really agree that it is “Cheating”. Since benchmarks are supposed to decide and show how strong your device is…, If they went the way that you wanted we would never know how the new Cpu’s and Gpu’s preformed at max Capacity.
If cpus and gpu also goes at max in most graphically intensive games and apps so doesnt matter
AA is like scrambling to clean up their favorite OS’s and OEM’s mistake right now. Lol…
It irritates me more that reviewers don’t do a proper set of “real world tests” exactly the same on each device, so we can all see the reality of the devices.
For example. . . all the bitchin’ about the Moto X not having quad cores when in reality cores 3 & 4 rarely ever get used on an Android device, thus all “real world tests” of the Moto X Vs HTC One Vs S4 showed the Moto X coming out on top — let’s not argue over “future proofing” at the moment.
“Real world tests” will ensure that manufactures stop the total marketing BS that geeks love — more cores, more ram = must be better — and make them focus on their devices actually performing well. . .
Let’s rather create a test for real life performance that we can run somehow.
I’m not that into benchmark.but I have seen ppl including my friends who always says
Look that phone tops all benchmarks, that phone is the best phone in the world !!!
For me it’s crap talk cuz ik tech n I read tech
But not majority of the world…
So ya,oem must stop cheating !!!
While I don’t take benchmarks too seriously, its still ridiculous that large companies like Samsung cheat and lie on these tests. If it were Apple that was found guilty of cheating on benchmarks this whole site would be littered with negative articles and commentaries, instead of the writers trying to downplay the situation with “do you care?” editorials.
“If it were Apple that was found guilty of cheating on benchmarks this whole site would be littered with negative articles”
They didn’t cheat. Show you what?
read my reply again. the quote is part of the reply.
but since you have missed the obvious. cite me some articles showing android authority behaves like apple trolls, railing against them. pretty please with sugar on top.
android authority are blatant apple trolls in the exact same way that appleinsider are blatant android trolls; it’s essentially the reason both sites exist – to serve the fanboys, because lets be honest the general public couldn’t give any less of a sh*t.
I really don’t care enough about winning this argument; do your own research, since the apparent nature of the website is not as obvious as your ill constructed reply where the quote being part of the reply doesn’t make it any less incoherent.
… you acted like a doucheb*g first.
you are pissed cos you cannot cite some articles . buzz off loser. I dont troll in apple sites, unlike you guys here
What do you say about closed source IOS?
Dont you think they are cheating as well?
cheating or optimizing? There is a difference. DO you think iPhone 5S wasn’t optimized for crayola motion sickness?
no one gives a flying *uck about benchmarks, other than feeling good that your device is powerful that others, bragging rights. at the end of the day, 99% of the people want a fast device, which works very well with a long lasting battery.
I’ve seen replies like “no one gives a fuck about benchmarks”.
If that statement is true, why cheat at all? food for thought…
doesn’t mean they don’t need to do it. they’d like to carry it on regardless. it’s samsung for gods’ sake they have a $10Billion ad budget and yet they stoop to ridiculous extents to prove they are ahead in pissing contests. the other manufacturers wouldn’t want to get caught behind, and so they enter the game. PC benchmarks had plenty of it. I do not know if you have tracked it – I tracked all of it (cpu, gpu, ram, hdd, smps etc) at anandtech, tomshardware, etc for a long time now. and this one really doesn’t seem too much away from all of the crap that used to go on. of course, people used to call out the offenders, but nothing earth shaking as it is made out to be
this is NOT some fiasco of some sorts as this article is made out to be. big deal. get over it.
not really. i only use test to check changes onmy system and occasionally to compare other tablets that are the same model.
Cheating makes them look bad and wastes their time and effort. I don’t really care all that much, but it is annoying.
Benchmarks will get updated to better reflect real world usage over time, so eventually optimizing for the benchmarks will just mean optimizing for certain actual usage patterns, which is really what they should be doing anyway.
What you do know about a company has to stand as a proxy for all the rest of the stuff you can never know. So whilst I am not especially interested in the benchmarks of a phone, knowing that the company is out to deceive me can only make me trust them less in general.
So now that I come up against as issue about which I do care, region-locking, I have no trust in the blandishments that Samsung puts out about how it is all for my own good (somehow?) and that I can in any case remove the region-lock by activating first with my home-region sim.
So a device that increases power when it is confronted with a laboring task is called a device that is cheating the system. And benchmarks are important on a mobile device because you need all this raw power for what? Do I really care about bench mark results, not overly. If the HTC One beats out my Galaxy SIV by a few points so be it. I can only type a message so fast with my thumbs, I can only send a message as fast as my networks allows. My laptop CPU ramps up when I play a game or do some video editing maybe it is also cheating the system. Yes these comments are somewhat on an unorthodox level but it’s a mobile phone running a mobile operating system that in the end doesn’t need and enormous amount of raw power, and hey it’s not like the CPU is creating artificial power.
Why this cheap publicity? If your product is good enough, why not allow it to stand on its own merits? This shows that the manufacturers themselves don’t have faith in their own products, and they are tricking the customers into thinking they have bought something better, which in reality is lower graded than advertised. Shame on these so called innovators. This is what you innovate? Fanboys are trying to cling on to their investments by saying “benchmarks don’t matter”, but they fail to acknowledge that this is the same reason why customers face rotten experiences with aftermarket services, when they need them. It is because the same culture, same ethics gets propagated throughout the company.
I care about a phone that has removable battery, microsd slot, plastic removable back. I´m with Sammy as long as they do it. All the others I don´t care and do you know a bigger cheater than Apple?
xperia zr has all the things you listed
Really cheating has become routine. We’ve heard of high schoolers, and college students cheating for better grades. Here in Montgomery, Alabama some teachers were involved in cheating (selling grades). Students cheat, teachers cheat, athletes cheat, corporations cheat, politicians . . . well, you get the picture. We probably shouldn’t be surprised.