Report: several major Android makers cheat in benchmarks

October 2, 2013
31 13 68

benchmark cheating cb_agulto

It should come as no surprise, but gaming benchmarks is a common practice in the Android ecosystem.

Anand Lal Shimpi and Brian Klug of AnandTech just released a damning report on the issue, revealing that, with a few exceptions, devices from every major Android OEM are “optimized” to deliver top results in benchmark tests. This involves setting the device’s CPU to top speeds whenever certain benchmarks are running, thus artificially increasing scores. Some device makers apply similar tactics to GPUs.

Since it broke the story about Samsung gaming benchmarks on the Galaxy S4, AnandTech tested a host of devices and found that other manufacturers are, more or less, doing the same thing. Specifically, Asus, HTC, LG have at least one device that exhibits the same behavior, affecting at least one benchmark.

There are a few “clean” devices – AnandTech found no sign of benchmark gaming on the Motorola Razr i, Moto X, Nexus 4, Nexus 7, and the Nvidia Shield.

anandtech benchmark cheating AnandTech

Some devices are worse offenders than others. For instance, the LG G2 and the Asus PadFone Infinity (Snapdragon 800 variant) only kick into high gear when AnTuTu and Vellamo are detected, with five other benchmarks running without interferences. At the other end of the scale, the Galaxy Note 3’s “cheating mode” is active on six out of the seven benchmarks tested.

The shady behavior is independent of the maker of the SoC, so the blame is squarely on the OEMs’ shoulders. As Anand notes, the silly thing about this whole affair is the benefits of optimizing firmware for top benchmark scores are minimal, in most cases just a few percentage points.

The sad takeaway is that, for dubious benefits, some Android manufacturers are willing to mislead customers and the media. For more details, we highly recommend that you read the full report.

Comments

  • Bone

    Surprise surprise… well not really. AnTuTu and a few others are useless anyway.

  • Bjajjull

    It’s not cheating… They just piush the hardware to its maximum which actually can damage the device, for example if it’s already too hot.

    Is it fair? No.

    • mobilemann

      it’s not cheating, it’s just cheating.

      /sigh

      • MasterMuffin

        “It’s not cheating, I mean the other guy did it too!!”

      • Joshua Hill

        Given that users can emulate what the manufacturers are doing by optimising their cpu govenor I think cheating is the wrong word.

  • Oli72

    I knew there was another article coming out someone is snitching on the others. Sad.

  • Amine Elouakil

    The big difference as mentioned in the Anand articale which you need to add I believe is that not only Samsung has said “Optimizations” but they OVERCLOCK their Exynos chips beyond their thermal limitation for benchmarks

    • Joshua Hill

      Not ‘beyond their thermal limit’. They change the thermal limit. This is only on Exynos chipsets and not Snapdragons.

  • NeedName

    And why you should always compare two devices via “real world tests” to see just how they perform in reality.

    • Balraj

      Sad part is, most review sites doesn’t do real life comparison test

  • Raj Singh

    You sure the word cheat is accurate? Don’t they “optimize” the processors to produce the desired results that are not typical of everyday use to see the theoretical maximum performance of said device/processor on specific benchmarking applications?

    • mobilemann

      if the benchmarks have access to it, and normal apps don’t. then it’s fucking cheating.

      • Joshua Hill

        Except I can make my over 2 year old SGS2 do the same thing. So if I can do it on a old phone it’s obviously not cheating. Optimising is a more accurate description. Gaming is also a more accurate description than your emotive and biased ‘cheating’.

        • mobilemann

          what games makes or the gnote go to 2.3ghz? What % of the general public root ? Install a new kernel? It’s sad some people are willing to go to such insane lengths.

          • Joshua Hill

            All games that have a cpu utilisation greater than the cpu govenors up threshold can get to the maximum cpu frequency state of a given chipset. Go read the Anandtech article it has some good graphs illustrating the cpu frequency over time both with and without the optimisations in place. For cpu intensive games the results would look similar to the non optimised benchmark results with the cpu possibly hitting maximum frequency for prolonged periods of time.

          • mobilemann

            Citations that the games get to use the same clock speed used for the benchmarking apps please.

            Please.

          • Joshua Hill

            Go cite yourself. Do you think max gpu frequency or max cpu frequency is more important for games? Why don’t you try and obfuscate the issue some more.

          • mobilemann

            You just said ars or Anand said the benchmarking profiles are the same speeds as the ones used for games. I don’t think they did. Cite your source please

          • Joshua Hill

            1) I never mentioned ars;

            2) I cited the Anandtech article linked to in this Android Authority article. I never cited them as saying ‘the benchmarking profiles are the same speeds as the ones used for games’. I cited Anandtech as showing ‘some good graphs illustrating the cpu frequency over time both with and without the optimisations in place’. Couple that with my description of how the cpu govenor in Android OS works ‘All games that have a cpu utilisation greater than the cpu govenors up threshold can get to the maximum cpu frequency state of a given chipset’. Using a little thing called logical reasoning I then concluded, hence no citation for this needed, that some games with heavy cpu utilisation could reach a given cpu’s maximum frequency.

          • mobilemann

            that’s just called assumption. the white listing of bench-marking apps also is quite specific.

          • Joshua Hill

            The benchmarks apps aren’t running at a higher cpu frequency than other applications have access to. They are just running at the max cpu frequency the whole time. It’s obvious you haven’t looked at or comprehended the Anandtech article. Just in case you missed the link in this article here it is: http://www.anandtech.com/show/7384/state-of-cheating-in-android-benchmarks

            I was very careful with the language I used so that the statements about games max cpu frequencies was all hypothetical. Therefore there were no assumptions. If only you would be so careful with your language instead of constantly mis-representing me.

          • mobilemann

            ok, well the article you love specifically says “Ultimately the Galaxy Note 3 doesn’t change anything from what we originally reported.”

            they also state:

            The CPU frequency optimization that exists on the SGS4, LG G2, HTC One and other Android devices, still exists on the Galaxy Note 3.

            in their original report, they specifically say:

            Running any games, even the most demanding titles, returned a GPU frequency of 480MHz – just like@AndreiF alleged. Samsung never publicly claimed max GPU frequencies for the Exynos 5 Octa (our information came from internal sources), so no harm no foul thus far. Firing up GLBenchmark 2.5.1 however triggers a GPU clock not available elsewhere: 532MHz. The same is true for AnTuTu and Quadrant.

            http://www.anandtech.com/show/7187/looking-at-cpugpu-benchmark-optimizations-galaxy-s-4

          • Joshua Hill

            Yep, except we were clearly talking about cpu frequencies. So once again, thanks for trying to obfuscate the issue and misrepresent me.

          • mobilemann

            read who i was replying to when you responded. I’ve been talking about overal SoC performance in GPU and CPU. Enjoy your bucket of dicks.

          • Joshua Hill

            The first time you mentioned speed was the 2.3GHz max CPU speed of the snapdragon 800 chipset in the Gnote 3 in a comment specifically directed at me. The GPU optimisation for benchmarks is not an issue on snapdragon chipsets only exynos ones.

            Do you realise how silly you look. Constantly mis-comprehending and mis-representing, then resorting to ‘eat dick’ insults.

          • mobilemann

            Wow, what an arrogant little asshole, kidding? you’re the guy who thinks GPU OC’ing is totally not cheating, desperate to defend what’s obviously a large corporation trying to mis lead people, again, i don’t care, and i responded 90% of the time mobile, my sincere apologies for not specifically getting into what parts i was talking about, at what points, but the point is there were performance enhancing over-clocks being done to a version of the note 3, that no games had access too.

            You, because you’re a prick, consistently try over and over again to confuse the issues; (all while saying i’m confusing them? kidding? You’re the one hung up on the SoC, while i’m talking over all) That they cheated, and got caught. Not made their devices looked better, they just cheated.

            I own a note 3 now; it’s a great phone. It doesn’t need self important kids who think they know shit defending them. Have fun updating your facebook.

          • Joshua Hill

            When’s your name changing to trollman?

          • mobilemann

            right after you change your’s to “bucket’s_O_Dick Hill”

            Also, welcome down off your hight horse, and to the internet. You took an entire comment longer than i thought you would have:D

          • Joshua Hill

            Well after others had already seen you burned and you decided to re-troll after the convo was dead for 5 days I figured what did I have to lose by stooping to your level :D

          • mobilemann

            it hurts man. look at all these 1 negatives for me, i wonder who could have done it? Maybe i’ll figure it out while i cry myself to sleep over some pre-teen chick freaking out. Have fun astroturfing and updating your facebook, you leet techno guy!

          • Anon

            Games and citations to said games have no point to this article when it’s simply about OEMs trying to get better reviews because they know sites usually benchmark the phones for their reviews.

          • Joshua Hill

            No, they have no point when @mobilemann:disqus is asking me for a citation for something he alleges I said which I just showed in the above post I didn’t say and hence didn’t need to cite. Your comprehension is apparently on the same level as mobileman so why don’t you two have a conversation. If you do have a reasonable question about what I have said I’ll be happy to continue discussing but if you’re going to misrepresent the situation find someone else to waste your time on.

          • Anon

            Hrm, if I offended you somehow, I apologize.

            Benchmarks, for the most part, are intended to represent or at least attempt to imitate real world applications so that you can make sure your phone can run certain applications well. That may not be the case for every benchmark and some may test for theoretical performance as well. In my opinion, the article says that OEMs are ‘cheating’ because the companies are trying to appear better than what they really are. Check out the following article from Ars about the Note 3 benchmark adjustments:

            http://arstechnica.com/gadgets/2013/10/galaxy-note-3s-benchmarking-adjustments-inflate-scores-by-up-to-20/

          • Guest

            Go cite yourself. Do you think max cpu frequency or max gpu frequency is more important for games? Why don’t you try and obfuscate the issue some more.

        • Anon

          What you have to understand is that the OEMs only ‘optimized’ the processors for the benchmark only. It’s not like they overclocked the processors to make the phone better overall. They simply set the processors to utilize all 2/4 cores at 100% when running benchmarks when it’s supposed to only run 1 core. If they overclocked the processors for the phone as a whole, this would not be a criticism.

          • Joshua Hill

            What you have to understand is that if you think benchmarks are indicative of real world performance you deserve to be duped. The phone manufacturers, not to be confused with the manufacturers of the chipsets who apparently don’t condone this (I don’t either, however I take objection to the inaccurate and biased use of the word ‘cheating’), are simply optimising their product to show it’s theoretical maximum performance.

        • http://islandinthenet.com/ Khürt L. Williams

          Ok. Call it gaming then. Can you at least agree this was “intentional”?

          • Joshua Hill

            Most definitely. I’ve never said anything that should cause people to believe I would think otherwise.

      • Raj Singh

        The industry doesn’t use “normal apps” to submit benchmark scores. They use specific benchmarking apps to do that. I’m saying that these companies optimize their processors for these tests; the word “cheat” is not accurate. It’s link bait… for the most part. Having said that, manufacturers should disclose that they do that, if they do that, but ethics is a whole other issue.

    • Anon

      It’s only optimizing the processors for the benchmarks. Not for anything else. Theoretical performances are just that – theoretical. No real world application. If they did processor optimization for commonly used apps, it would be a different story.

  • RaptorOO7

    I have never used these benchmarks as a deciding factor on whether or not to buy a device. It’s pretty easy to guess why, they can be gamed like anything else. Instead I look at overall specs, processor, storage, microSD slot, display resolution, RAM to run the device etc.

    Besides they are just taking cues from lots of other industries like the automotive mfg’s who cheat on the EPA #’s.

    • Balraj

      True but if you check any review site..all of them do benchmarking

  • Ben Edwards

    In all honesty, who really cares? I don’t understand why people are getting so upset about this. Benchmarks are little more than a pointless measurement of nothing. If an OEM was releasing a phone that was claimed to be a quadcore but in fact only had 3 cores, then it would be worth reporting. This, not so much.

  • zymo

    Quoting Mr Eric Cartmanez: “It’s OK to cheat as long as you cheat your way to the top. ‘If you cheat and fail, you’re a cheater!”

  • Jokesy

    Even I believe that Apple probably also cheats just that no one could confirm that yet due to the how closed iOS is. There is simply no logical reasons for a weaker processor to be getting way higher scores in benchmarks.

    • Nitinart Nunthong

      First I am not Apple fanboy. I am Android fan.
      As I know Apple A7 uses new GPU from Imagination Technology ‘s series 6. (A6 uses series 5 which is 544MP4 something like that). Also Apple can modify their SoC whatever they want (like Qualcomm do) because they have a license from ARM to do that.

    • http://islandinthenet.com/ Khürt L. Williams

      Did you even read the AnandTech article? Motorola and Apple are not doing this.

      • Jokesy

        There is no frigging way they can find out if Apple is indeed cheating except maybe on a jailbroken iPhone

      • Joshua Hill

        And Google. Motorola /= Google. Google owns Motorola, Google devices (i.e. Nexus’s) also don’t do this.

  • Fernando Asencio

    I dont get this, so the set their CPUs to go full speed when the benchmark software is detected, isn’t that what the benchmark does anyways? If you tell me me that they are rewriting the code on the benchmark to cheat i could understand the big fuzz. This is like taking your car to a dyno, you flood it to the max to get the best result possible, it does not mean you going to get that result on every day driving.

    • Anon

      The OEMs are essentially setting the CPUs to overclock during any benchmark. The function is exclusive for benchmarks therefore it does not determine real world usage. It’s like taking your car to get dyno’d and hooking up and using nitrous so you can show people how fast your car is on the dyno sheet (and never really using nitrous ever again).

      • Fernando Asencio

        Are you sure about the OC? I have not found any article that mentioned OC, all i have read said optimized, to me that means set their clock at their top clock, not allowing it to slow scale down to lower speeds while the benchmark is running, and that is not overclocking.

        • Anon

          Sorry, I misinformed ya! Here’s an article that will explain it much better than I ever can lol: http://arstechnica.com/gadgets/2013/10/galaxy-note-3s-benchmarking-adjustments-inflate-scores-by-up-to-20/

          • Fernando Asencio

            Very interesting article, now this brings me to my next question, are this benchmarks design to test top performance of the phones or to test if their power saving features work? Because performance test its supposed to show how powerful the unit really is, a multiple core systems is design to bring the power of the resting cores as need it, if these test only want to test 1 core of the unit then they should specify that, and that would not make it a valid performance test. Even in laptops you see high performance mode and power saving modes.

          • Anon

            There are multiple benchmark tests. 1) Raw cpu power – Apps that run mathematical calculations and spit out some numerical rating of the CPU (probably measured in MFlops) are measuring the capacity of the CPU to do work. They can also test single core or multi core processes. 2) Ram speed – We’re all caught up worrying about how much RAM our devices have, but the speed of RAM is just as important. There are a few tests that measure just memory performance because it is partially dependent on the chipset, but this data is often included in comprehensive benchmarks. 3) Game-centric benchmarks – These tests are used to push the SoC, and specifically the GPU to render 3D environments with as much accuracy as possible. This often involves tests at multiple resolutions and with different engines. GPU benches may also test 2D drawing performance. 4) Speed of NAND flash storage – Simply having a lot of storage isn’t going to do you any good if it’s so slow that it causes your system to lag. There are a number of comprehensive benchmarks that include I/O per-second and other measures of storage speed. 5) Browser benchmarks – This kind of test is heavily dependent on CPU performance and screen resolution, but it gives data about a specific use case.

            The truth is that some of these tests are little more than self-validation. The difference in a CPU score of a few percent is not going to make a huge difference to the way you use the phone. More than the CPU numbers, it’s the software optimization that makes a device smooth and easy to use. If the OEM has clogged up Android with poorly implemented features, it won’t matter how high the CPU benchmarks. The same goes for RAM speed, but to a slightly lesser extent. Some phones and tablets do actually have faster RAM, which can improve the experience. I would argue, however, that graphics benchmarks can serve a purpose. If you want to know how well a device is going to handle the hottest new 3D games, running OpenGL benchmarks is a good idea. This is the closest you can get to a real world test without playing a ton of games yourself. The storage benchmark situation is a little different. There is a real reason to check your storage speed, and it won’t even take you terribly long. The throughput of the NAND can slow over time, eventually dragging the system down (check out Androbench). Also, knowing a bit about web rendering and performance can be useful when the time comes to troubleshoot browser issues. If pages are loading slowly, you’ll want to know if it’s your phone, or simply network congestion. Standards conformity will largely depend on which browser you use, and Chrome is very good in this category.

            Probably more than you wanted to know but hey, there it is.

  • EvenInTheDarkestHour

    With benchmarks being what they are, and their place in anyone’s decision making process…does it really matter? The phone software is tuned and tweaked to whatever specs the OEM has set for it. My benchmark cheating Note 2 is still a fantastic piece of hardware, with more power than I’ll use. For whatever reason in the future, if more processing power is needed, Samsung can do an update to utilize more. Anyway, if someone doesn’t like the software, he/she is free to root and flash something else. That’s the beauty of Android.

  • Disasterpiece

    Well this does explain why my S4 lags at times, while the few times I played with my sister’s Moto X, left me wanting for more

  • Nitinart Nunthong

    According to the report , I am proud of having a nexus 4. (not sure about nexus 7 (2012) and galaxy nexus) . I am curious about Sony too , did they do the same thing?

  • Bryan Z

    This is disappointing yet I’m not surprised glad i have a nexus 4

  • Darktanone

    You forgot to mention that Motorola and Apple were the only ones found not cheating.

    • Joshua Hill

      Umm what about Google?

      • http://islandinthenet.com/ Khürt L. Williams

        Motorola = Google!

        • Joshua Hill

          No it doesn’t. See my other response to you.

    • Joshua Hill

      You forgot to mention that Google were also not found to be using this practice, which is not ‘cheating’. Makes you statement inaccurate. Were you trying to cheat?

  • Allan

    See, the thing is some of the manufacturers listed above probably clock their phones’ respective CPUs at max in order to reach their true potential in CPU tests.

    What Samsung’s doing is unacceptable; they’re running the CPU high in GPU tests in order to get a higher GPU score. That’s why (aside from the fact that they always want a higher binned SoC for their own phones) they always get a higher score than other phones with the same-clocked SoC.

  • abazigal

    Ah well, I guess this gives you Android smartphone users another feature that you can use to lord over us iphone users – faked benchmarks.

    “When is Apple going to invent faked benchmarks? Android has been faking benchmarks for many years now. OMG! Apple is late to the party again. Apple is doomed!”

    :D

    • toboev

      “Apple is late to the party again. Apple is doomed!”
      Er, no. Once Apple gets there they will claim it was their original idea and sue the pants of everybody else. As always.

  • Saif Merseysider

    I got a couple of inquiries I like to make. When benchmarks are done n results are posted, the website responsible for it would have indicated that it is in no way related to real life performance, right? So, what is with all these fuss? Are tech sites running out of idea to come out with something ORIGINAL to discuss? Second, isn’t it strange that the news about these ‘riggings’ are made with the exception to certain US made phones, coincidentally with the neverending rumours of how Google is trying to break free from Samsung’s dominance in the android playspace? Let me be forward about this, ok. I think this is just a smear campaign started to glorify american products and at the same time to end other’s reign. When I say ‘others’, I meant asia’s developing countries. Call me a conspiracy theorist if u like. But a couple of months ago, this site (Android Authority) n a few more bang a lot about Moto X and stressing how important experience is to specs. Now that the story has wind down a bit, u guys seem to find a new way to go around it. By saying the whole damn thing is rigged with the benchmarks and stuff. What the fuck?! Even a computer will not use it’s fullest ability while performing something and u guys know that. Stop with all these stupid discussions alright. No one is buying it.