Tech savvy readers know that benchmarks have little significance in real life. They are a relatively good way to compare devices in relation with each other, but other than that, benchmarks tell us quite little about the real life performance of a smartphone or tablet.
Here’s another reason we should collectively give up our obsession with benchmarks and leaderboards – they can be gamed.
Gaming benchmarks has been a time-honored tradition in the PC GPU industry, but so far, we haven’t seen the practice adopted in the mobile world. Until now, because it looks that Samsung has just been caught red-handed.
Here’s the story – a user on the Beyond3D forum noticed that the GPU of the Exynos 5 Octa processor in the Galaxy S4 runs at a higher frequency when running benchmarks than it does when it runs games or other regular apps.
Screenshot of AnTuTu benchmark with the GPU frequency in the background (Credit: AnandTech)
The folks over at AnandTech investigated the claim and their conclusion is that Samsung does indeed set the GPU of the Exynos 5 Octa version of the Galaxy S4 to run at 533MHz during benchmarking, as opposed to 480MHz during normal operation. The benchmarking apps that trigger the behavior are GLBenchmark 2.5.1, AnTuTu, and Quadrant.
AnandTech spotted a similar behavior when it comes to the CPU – when running GLBenchmark 2.5.1, AnTuTu, Linpack, and Quadrant, the cluster of Cortex A15 cores @ 1.2GHz kicks in. When running GFXBench 2.7 (a benchmark that wasn’t whitelisted in the firmware), the lower-powered A7 cluster clocked at 500MHz is used. Moreover, it appears that the CPU of the Snapdragon 600 version of the Galaxy S4 is also set to maximum speeds when benchmarks are ran.
In addition to these empirical observations, AnandTech also analyzed the firmware of the Exynos version of the Galaxy S4 and found the hard code that instructs the GPU and CPU to kick into high gear when specific benchmarks are running.
While Anand Lal Shimpi and Brian Klug of AnandTech elegantly call the behavior “benchmark optimization”, it’s clear that we’re looking at Samsung’s underhanded attempt to make the Galaxy S4 look better in benchmarks.
The moral of the story? Don’t ever make buying decisions based on benchmark scores.
Like this post? Share it!
Many low level oem do it to gain popularity but Samsung-number one player
Shame on you people !!!!!!!
Samsung shouldn’t be doing this, but they weren’t the first ones to do it and won’t be the last one.
Plus, the epitome of sports car makers, Ferrari, does this as well. Ferrari do not let car magazines test their cars without optimizing their cars(changing suspension settings,etc) on specific test surface to be tested.
Ferrari, Intel, Nvidia, etc,,,,have done same things as well, not just samsung.
Doesn’t matter who does it, it’s immoral behavior.
Another reason why I wish samsung wouldnt “use” android
true. but don’t call out only one of them, expose all of them. btw, I am no samsung fan.
No one else has been confirmed to be doing it so far.
I was using it in context of the OP, who had mentioned similar behaviour from other companies. gaming benchmarks has been staple fare like ages now. that’s one of the reason i dont care about cores or their benchmarks too much
I’m not just blaming Samsung just bcuz they got caught
Just that you get a bad feeling, when you see top oem do something very crap like this :-(
I’d still be happy to have a Ferrari that can only do 60mph as opposed to 220mph
Same here…As long as I can “adjust” the Ferrari so it can go faster. :-)
Shamscum doing dishonest things? Why I never… They don’t get any more corrupt and evil than this tech monolith of South Korea willing to stop at nothing.
No surprises here. Anyone who uses benchmarks to decide on a device deserves to get duped.
Amen brother. Yet so many of the clueless fools here do, as they do only considering tech specs.
The thing is this kind of behavior from samsung makes from some free advertisement for them to mass the crowd to them, ‘oh looky we have the biggest numbers we are the fastest in the world, you don’t need to go far to find such threads AA has tons of them, I’m not suprised really Samsung does everything to sell their product heck they even employ people to post negative comments against competing brands such as HTC
Samsung deserves to get “duped” for being dishonest again* not the users. I don’t know if your trying to defend them. No one said they are buying a phone just because of the benchmarks.
but its get view point , and many people believe that because of scores its fastest device and others get bad rep like htc and sony.
I don’t see the problem. Those max. frequencies are being tested by test suites without restriction. That’s the point of benchmarking things…
Benchmarks should reflect the reality. If clock speeds are limited to 480MHz for every app except for benchmarks, how is that reflecting reality?
Those benchmarks reflect reality:
“GPU duties are handled by a PowerVR SGX 544MP3, capable of running at up to 533MHz.” –Anandtech
White-listed test suites have access to those 533MHz. What we don’t know are the technical reasons behind the 480MHz limitation. Heat? Battery life? Performance between different SOCs in a same model?
No, they do not reflect reality when the GPU is running at a higher clock than normal.
That would be the same as if I superclocked my Xperia Z to 1.9ghz and ran a benchmark while the phone was under ice water to prevent overheating. According to you that benchmark reflects reality, but it does not because the phone will not be able to operate normally with that clock.
They’re the ones who got caught… I guarantee they are not alone in this. Besides benchmarks are jokes, if you buy a phone Thanks to a high benchmark alone you’re a fool…
Well, you are fool if you are buying any products from a thief. Shamesung is nothing but a shameless copycat. This article right here tells nothing new. Everyone knows what morale they have.
You are fool buddy
Owww man BUSTED!
everyone “optimizes” for benchmarks. not surprising at all. it’s the same with browser wars, or GPU/ CPU benchmarks, everyone has their own favorites. I’ll just move along.
This isn’t optimization, it’s a hardcoded hack.
lol I was gonna say that
I don’t understand what you’re trying to say(@Bogdan Petrovan). When other companies do it, it’s optimization, and samsung does it it’s hardcoded hack?? What samsung did is not right, but don’t single samsung out, because other companies have done it and some worse than samsung.
That’s what i was getting at – gaming the benchmarks is nothign new. In this case samsung went too far I guess
No, in the case of GPUs and CPUs for desktops they give you a benchmark score for a stable clock settings. If the device can’t have 533mhz on the GPU without overheating or being generally unstable then they shouldn’t use 533 in a benchmark.
When you optimize your computer you overclock it to the best settings that is still stable, optimizing is not overclocking just for a benchmark score.
agreed. i used the word optimize with sarcasm, if you missed it.
I guess I did, sarcasm is hard to detect in written text. >.<
Samsung = immoral & corrupt. Its as simple as that.
join microsoft , intel, oracle, amazon, apple, sony, google, etc. to the list (yeah google! they turned their back on net neutrality recently!)
all the big corporates are pretty much that.
Disappointing behavior from Samsung. Playing with their customers.
Whats next – talk sht about the competition … oh oops happened already
cheaters and snakes are gonna be cheaters n snakes.
so, is there a way we can manually toggle that state-change ourselves for gaming? its nice to have power, if you can use it. ps> yea, that does suck … just saying, maybe there is a silver-lining.
I won’t be surprised when samsung decides to not carry android anymore because tizen is good enough to take it to the next level and ditches android. I see it happening. Fuck Samsung
I seriously don’t ever see that being successful. If they try it, the joke will be on them.
You must learn the concept of bench mark .(Max stress )
comparing a qual 800 with galaxy active(qual 600) .
complains about benchmark test ? . There are 6 to 7 variants in S4 right ?
waterproof mobile with same specs and normal mobile with out compromising price ? . Then they need to Down grade specs .
Benchmarks do not reflect real time usage. They are designed to test out max performance possible. A super benchmarked phone need not be super optimized for day to day usage..
Take example of a supercar say a Koeniggsegg Agera R.. On testing there is no other car capable of beating it.. But that doesn’t translate to optimized running on day to day life.. A 0-100mph time of 5 seconds means nothing when daily usage will not permit speeds above 70 max!
What it means is that given the proper conditions, the car is capable of superior performance. Similarly benchmarks account for the extreme situations. Some might call it “Future Proof” meaning when today’s ultimate performance becomes tomorrow’s standard. In that event, hardware will improve and allow for further enhancements! That is the rule of technology!
I am not supporting Samsung. But the very purpose of running benchmarks is not to imitate daily usage. The superior clock may become standard tomorrow for which today’s device has the power to remain in contention.
But realize this.. When 533 Mhz for GPU becomes the standard.. We will no longer be thinking about the GS4. We would all be lining up outside Best Buy waiting for GS8 to release and have this discussion all over again – the Adreno 6000 runs benchmarks at 2.3 Ghz whereas apps runs at 1.9 Ghz!!!
I am gonna save this comment in dropbox and copy paste it when I find this discussion again in 2016!!! :-D :-P
Which is besides the point.
The issue here is about Samsung programming their device to game benchmark tests, and not about its relevance to real-life usage.
Kudos to you Sir/Madam, for pointing out the obvious!
I had absolutely no idea I had digressed from the topic of discussion! Thank you, Thank you, Thank you!
Doesnt the HTC one has similar benchmark result as the S4 with the snap 600? If so then HTC is doing the same.
“AnandTech spotted a similar behavior when it comes to the CPU – when running GLBenchmark 2.5.1, AnTuTu, Linpack, and Quadrant, the cluster of Cortex A15 cores @ 1.2GHz kicks in. When running GFXBench 2.7 (a benchmark that wasn’t whitelisted in the firmware), the lower-powered A7 cluster clocked at 500MHz is used.”
You know, just throwing this out there.. but the first benchmarks listed under using the A15 cores actually, you know, benchmark the SYSTEM.
GFXBench is just as it says.. a GPU benchmark written to test OpenGL and such, which should be hardware off-loaded via the GPU.. so not taxing the CPU, which means WHY would the A15 cores kick in if the A7 cores could handle the load just fine? Shesh
Are you guys serious right now? I mean really, who gives a flying f*ck if they hard coded their GPU to run a little faster during a benchmark. If anything I’m happy that they did it so now when I get an S4, I will be that much cooler than an HTC One user when I score higher benchmarks. Besides every company has been doing this for years, it’s not cheap it’s called good marketing. They also have a faster 1.9ghz Snapdragon 600 compared to the standard 1.7ghz in the One and other devices, nobody seems to be asking about that, everyone seems to be happy about that little “optimization”.
but its immoral and misleading and because this other brand gets behind like htc , sony , lg ,stop giving excuses for your beloved company.
“…and during his press conference today to address the ‘benchmarking’ scandal, Samsung’s CEO simply responded with ‘Who gives a fvck?!’ Immediately afterwards he was seen getting into the back of his Maybach with his 19 year old model girlfriend and a Magnum bottle of Dom Parignon.”
I just read the original article.
so.. what I understand is, samsung TwDVFSApp check running app name (like com.aurorasoftworks.quadrant.ui.. etc), then enable its boost_mode accordingly ?
while TwDVFSApp also allow any benchmark app to request boost_mode through broadcast receiver ?
hmm.. probably any app/games can request that boost_mode to get real 532 mhz gpu. just be careful to not shorten gpu life :D
I always assumed the benchmark apps cranked everything to the max anyway. What’s the point of benchmarking if not?
Of course, Samsung shouldn’t be secretly doing this behind the scenes, but I’d expect the benchmarking app to do it since the point is to see how fast the hardware runs and you can’t do that without maxing everything out.
Benchmark apps do not overclock the device, and they should not.
A benchmark app is telling you how strong compared to other devices yours is, if your device overclocks only for the benchmark and gives you higher values that’s like cheating at a test and tell everyone that you are smarter.
I agree, but I’m talking about maxing out the CPU(s) and GPU(s) to the advertised max available, not overclocking. If they’re overclocking and no other apps can do that, then that would indeed be cheating. But my comment is about benchmark apps maxing out everything, as they should, as I would if writing one.
Samsung doing it behind the scenes is definitely not right, especially if they’re overclocking them… a feature not available to other apps.
｀y__////＿jニニニニニニニﾆfi- – - – - – - – - – - – - – - – - – ⊃ SAMSUNG
finally people are reliasing that quadcore processor and benchmark doesnt mean anything in real life performance, but i am also surprised those are the people who are saying that it doesnt have quadcore processor its bad device and benchmark are low so its bad device even because of this htc and sony device got low score because of fake benchmark scores.
think about it a litle bit
if the benchmarks are capable to get it
that means the device is capable to deliver those numbers
that means other apps should be able to do it too
the problem is with the variants all around of android its hard to make apps optimzed for all devices…..
The only reason those benchmarking apps can get 533mhz is because Samsung specifically coded the GPU to overclock when those apps are running. Unless you somehow make another app look like one of those benchmarking apps you will need get the overclock.
Ho ho hoooo!! Cheaters!!
Shame on you cheap copycat/scammer Shamesung. You are nothing but a thief, using “a stolen product”. Only good thing with this OEM and Android is, that all of these android customers are paying license fees to MS :)
Seems like a simple Troll comment…
LOL this Samsung’s practice shows the character of their home country.
Nothing wrong here it shows what the phone can do maxed out just like everyone else does and if you know what your doing you can max it out your self which is the purpose to show the hardwares capabilities although benchmarks really mean nothing the phone is still super snappy im satisfied with my purchase people shouldn’t get all up in arms cause the s4 can run circles around most high end phones instead buy one and stop trying to be a “cool kid” with your icrap and lastly dont be upset cause you iphone cant do as much as an android you bought it after all
well aint this weak..