Samsung stops skewing benchmarks results with KitKat update

March 4, 2014

    samsung galaxy s5 s4 note 3 2

    Recently Samsung and other Android manufacturers started to artificially boost their numbers in benchmark apps, but it appears that practice is starting to come to an end with Samsung’s KitKat update.

    In its version of Android 4.3 Jelly Bean on the Galaxy S4 and Galaxy Note 3 Samsung included code that forced all cores in the phones’ processors to run at maximum speed, skewing benchmarks higher than normal. After updating the phones to Android 4.4.2 KitKat, however, Ars Technica found the processors now run at normal speeds when a benchmarks app is launched.

    The publication was able to confirm its findings with John Poole of Primate Labs, the makers of GeekBench. Primate Labs put some detection code into GeekBench that triggered whenever phone maxed out its processors artificially when the app was running. While the Samsung phones running Android 4.3 Jelly Bean triggered the code, after the upgrade to Android 4.4 KitKat it did not.

    The change may seem small, but it is somewhat significant. Many smartphone reviews include benchmark tests as an easy way to compare the phone being reviewed to similar phones. Speed can be the one differentiating spec when you’re comparing two phones that are alike in almost every other way. Artificially setting processors to max during benchmark tests skews the results and makes it so they don’t represent real-world usage.

    Even after removing the code that skewed the results the Galaxy S4 and Galaxy Note 3 saw better benchmark tests under KitKat than when they ran Android 4.2 Jelly Bean. While the update does bring some new features, it also makes Android more efficient and faster, which is always nice.

    Samsung isn’t the only company getting rid of the artificial benchmark boosting. There are reports that Sony is getting rid of similar code in its KitKat updates for Xperia phones.

    Are you glad Samsung is no longer skewing benchmark results? Do benchmark results even matter to you?

    Comments

    • Grman Rodriguez

      For those who didn’t get how Samsung is cheating or think it’s fair to give maximum performance on benchmarks here is an analogy :
      Imagine I can walk at 10 miles per hour, but for a thirty minute race I can comfortably run at 20 miles per hour, however I can also run at 30 miles per hour but only for a short amount of time before I get tired and aren’t able to run as I need later on. If someone was testing the speed I’d use for the thirty min race, I’d have to run at 20 miles per hour, since that’s the speed I’ll take for the race, but instead since they are just measuring my speed I’m gonna run at 30 even if that’s not my actual running speed.
      Hope that clears some of the doubts.

      • MadCowOnAStick

        you can run 30 miles per hour? amazing

      • enloquecido

        The reality is that people who care about benchmarks are not only going to max out stock speeds but overclock it, so maxed out IS real world. If it can do it, then it’s real. Cheating is reporting a score that isn’t actually attainable (for some of us).

        • Grman Rodriguez

          I think a benchmark should measure real life performance, not just a hypothetical performance you can only get in that app or if you mess around with the threat of bricking your phone.

      • Tom-Helge Andersen

        Do you know what car manufacturers do with their cars when they are benchmarking their cars to see how fast they accelerate from 0 to 100 km/h and so on?

        Do they do that in the middle of the city streets where they can’t do the full acceleration at 100% full power?

        Or do you think they do it on a track to find the time while pushing the engine power to the limit.

        Take the Nissan GT-R R35 as an example. It uses around 2.8 seconds to go from 0 to 100 km/h. This is done when the engine gives out 100% of it’s power and not 70% of it’s power.

        And it’s the same for computer benchmarks and smartphone banchmarks to. They measures the performance when they run the CPU and GPU at full power.

        The meaning of the the benchmarks is to give you some numbers on how fast your computer or smartphone is, or how fast you car accelerates at full power.

        And because of this, it have never been taken as cheats as some butthurt Samsung hateboys claims. It’s normal that the benchmarks are allowed to run at it’s out of the box speed. It’s simply not possible that it’s considered cheat to run at it’s default speeds. It’s that simple.

        If the benchmarks had been run while the CPU / GPU are overcloked, then it’s cheat.

        Not only that, but every developers out in the world are fully allowed to optimize their applications to run better on the current hardware without being called cheats.

        • BD1971

          Its sad that false advertisement is now becoming the norm and consumers are defended the companies that do it. That’s just crazy.

          • Tom-Helge Andersen

            So measuring the acceleration with full power on cars is false advertimement?

            • BD1971

              Companies do all sorts of tricks to boost there products performance beyond what the consumer will ever see once the product is in there hands. As far as the over clocking of chips for benchmarking if you played games at that level your phone would probably become unstable. There not designed to run at that level for real use otherwise they would run at that level all the time. It’s not cheating is just not actually telling the consumer what he is going to really end up using once he buys the product.

            • Tom-Helge Andersen

              You said: As far as the over clocking of chips for benchmarking if you played games at that level your phone would probably become unstable. There not designed to run at that level for real use otherwise they would run at that level all the time.

              I’m saying: No, every phones are buildt to handle the tempratures the CPU and GPU are made to do.

              And also, you are never ever going to find out how a phone performs in real life performance without trying out the phone yourself to see how the real life performance is for YOU. What real life performance is for someone else might be totally different from what you sees as real life performance.

              So in short, to find out the normal performance or real life performance, you actually have to try out the phone your self to find that out. The benchmarks are there to give us some numbers that tells us all on how powerfull the phone is.

            • Grman Rodriguez

              Read my response

            • MrX8503

              You can accelerate your car at full power in any scenario. You can’t get the performance boost Samsung devices are getting outside benchmarks it knows to cheat in

            • Tom-Helge Andersen

              No, you can’t do the acceleration the GT-R R35 have everywhere. You need some straight roads and stuffs like that before you can do that.

              So no, it’s not cheat by any means to run at it’s out of the box speed.

            • MrX8503

              Then why did Samsung remove the cheat? You as a consumer will never see this performance boost in any usable application. How much is Samsung paying you?

            • Tom-Helge Andersen

              Samsung haven’t removed anything. Android 4.4.x just disabled the script (until Samsung activates it again) that is still present in the phones. So it’s just Google that have disabled it for the time being.

        • Grman Rodriguez

          Well the problem is that this max speed you are talking about isn’t the max speed the phone gets, it’s a max speed the phone only shows for benchmarks.
          Look at how they discovered the cheating: the changed the name of the Cinebench app to something else (I think CinĂ«bench) and, since the phone only recognized Cinebench, not CinĂ«bench, scores lowered a lot. This is different that what you are talking about because I can take the car to an open road at any time and obtain this speeds, but you can never obtain that “performance” Samsung claims on any other place different than the popular benchmark apps.
          I think the takeaway here is that no matter what OEMs say benchmarks suck and no one should use them

          • Tom-Helge Andersen

            And again, why crank in a gazillion of GHz on a processor in a phone if nothing is going to use it?

            The benchmark apps are some of the few apps that are able to tell us how powerfull the phones is. Doesn’t matter if other apps doesn’t even use 50% of the hardwares potential as long as i can see some numbers on how powerfull the phone is. That’s the whole point by the benchmarks to give us some statistics on how the different hardwares in the phones are performing at full power.

            Do you see 3DMark only use 70% of the GPU in our computers?

            No you don’t. They will use 100% out of the GPU to measure how powerfull the GPU is.

            So to sum it up. No, it’s by no means wrong or cheats to use the full potential of the hardware that comes with our computers, laptops, smartphones or even cars.

            • Grman Rodriguez

              The difference is that the computer or laptop uses 100% of your power, if your phone only uses 70% of its power even in the heaviest of tasks the benchmark should show only 70% of the real power, don’t you think?

            • Tom-Helge Andersen

              That’s not the point. The point is that there should be tools that lets us see how damn powerfull our smartphones is in the same way as gazillion of different benchmarking apps for Windows lets us do that for our computers.

              There is no differences on how benchmark apps should be runned on computers to how they are runned on smartphones.

            • Grman Rodriguez

              There should be if there is a difference on how smartphones use CPUs and GPUs

            • Tom-Helge Andersen

              But they don’t. They throttle the speeds depending on what the CPU or GPU are doing and are speeding up the speeds on them when they are active and so on.

              Both works the same way.

            • Grman Rodriguez

              Then how come Cinebench and Cinëbench show si different results if they are the same process?

            • Tom-Helge Andersen

              What the different benchmark apps do doesn’t have anything to do with what the whole point on what becnhmarking is and how the speed throttling technology works in the CPU’s and GPU’s today.

            • Grman Rodriguez

              They’re the same app, just with a different name (source: http://www.phonearena.com/news/Samsung-Galaxy-Note-3-apparently-using-benchmark-booster-to-fix-results_id47886)

              (http://pocketnow.com/2013/10/01/galaxy-note-3-benchmarks)

              The app was actually Geekbench and changed in name to Stealthbench, but it’s the same idea.

        • Andrew T Roach

          It’s still false advertising because in no other scenario on the device do all 4 cores max out on a consistent basis like they do with Samsung’s (and everybody else’s) benchmark optimizations. Multi-core devices are bursty and don’t use all cores for long periods of time for battery saving and thermal constraint purposes.

          The benchmark results under Samsung’s 4.3 are not representitive of the device people use daily. That’s the bottom line. Glad to see them cleaning up their act. Hope others follow suit.

          • Tom-Helge Andersen

            Again, you base the conclusion out of what other apps use or not use. You don’t take into the consideration that the benchmark apps are meant to use every cores at the maximum speed.

            So, go read about what the main reasons for doing benchmarks is and how they work.

        • MrX8503

          Anyone can enjoy the full power of the GTR on normal streets, albeit with plenty of tickets, or legally on the track. You can’t enjoy the full power of Samsung’s devices anywhere outside of benchmarks which is pretty useless. Thus your analogy is flawed.

          • Tom-Helge Andersen

            Everyone who owns the GT-R R35 can enjoy the launch control and get the full acceleration. But everyone who have an Android phone can enjoy the different types of benchmarks to.

            So there is no differences.

            • MrX8503

              GTR owners can enjoy that full power anytime. Your Samsung device is stuck in a benchmark. People don’t buy smartphones to enjoy benchmarks.

            • Tom-Helge Andersen

              Every users can still run benchmarks. However, they have to run them outside of their normal road / usage in the same way as GT-R R35 owners have to find a sideway or a speed track outside of thei normal driving roads to be able to use the full potential of the car.

    • Xavier_NYC

      At this point with the specs that phones are packed with these days, I don’t care about benchmarks as much anymore I care more about real world performance.

      • TheWay

        of course it doesn’t matter Samsung cheated in benckmarks and you’re happen to be a Samsung fan or paid by Samsung, However when it comes Samsung specs that in fact truly doesn’t matter nothing more than gimmicks

        • Xavier_NYC

          WRONG.. If you’ve actually read post on this site prior to this you’ll realized I don’t care for Samsung at all. I don’t even own a Samsung device, I actually have an LG G2… And if you were smart enough to scroll down you’ll see immediately after posting this I posted something about the LG G3.. Not everyone’s a fan boy. Have a good day sir

          • ratnok

            He’s not smart enough, but don’t worry about it man. You were on point.

          • TheWay

            sorry about that i like Apple but also Google and LG did a great job with Nexus 5 i think it looks better than HTC One design ..and yes Snapdragon 800

            • Xavier_NYC

              No worries. I had the HTC One but didn’t like it that much. Got rid of it after two months. I also wanted the Nexus 5 but the battery life was a concern for me so I stuck with the G2. I’ve also had every apple phone up until the 5. I was def an Apple Fan boy

        • Jayfeather787

          That doesn’t mean he works for Samsung. I hate Samsung, (well not really, but Nexus FTW) but I agree with him. It is a fact that benchmarks do not reflect real world performance. The Galaxy s4 scored higher than the Nexus 5, but in real life, the S4 was slower than the Nexus 5 by a long shot. It felt slow because of shitwiz. The Nexus 5 scored lower, but outperforms the s4, and I think that it also outperforms the Note 3.

          • TheWay

            Yes its faster than Note 3, Nexus 5 FTW!!
            Sony Z2, HTC One and LG G2 are great phones too but i like Nexus 5 more

        • Abdullah Abdulaziz

          I love your reply man you really hate Samsung dont you but why
          I’m same as you

          • TheWay

            Damn you got me lol!
            appreciate it

        • Mike – Construction Contractor

          Ouch. Love when immature children get shown the way.

      • Guest123

        The Moto X pretty much proved that specs are far beyond what we need for a fluid performing device.

        • Farbod

          I think iPhone did that long before the moto X

    • Xavier_NYC

      Looks like pics taken from the LG G3 has leaked online. The clarity is amazing.. http://bgr.com/2014/03/04/lg-g3-release-date/

    • chris125

      Still not sure why everyone made such a big deal. Benchmarks are meant to test what it can do. Sort of like gas mileage in cars yet people don’t get all up in arms about “false claims”

      • http://www.macinations.net moldor

        Gas mileage in cars is tested on a closed track at < 60kph, not in real word scenarios.

        Whether it's cars or tech, benchmarks are pretty much useless for decision making, unless you can be assured that all parties are running the same tests and haven't doctored their hardware to improve results. Anyone that relies solely on benchmarks to make buying decisions is a fool.

        • ratnok

          “Anyone that relies solely on benchmarks to make buying decisions is a fool.”

          Does anyone actually do that?

    • sandeep gore

      I have installed 4.4.2 0n my phone Galaxy 4, and I observed that there are only few changes have done in this version. There were no RAM utilisation difference I found. Hence I installed Kitkat 4.4.2 Launcher and I analyse that due to this launcher RAM utilisation decrease.

      Hence I would like to suggest to Samsung that if they can add this lsuncher inbuilt in 4.4.2 that would be great to see Samsung on Top.

    • filaos

      Are all tech sites going to re-run the benchmarks and adapt the ranking obtained by Samsung devices ??
      Well, I guess no. Evil is already done.

      • Dimitar Gospodinov

        did you read the article dude?

        “Even after removing the code that skewed the results the Galaxy S4 and Galaxy Note 3 saw better benchmark tests under KitKat than when they ran Android 4.2 Jelly Bean.”

        • filaos

          Maybe you could just read the source (ars technica) ??
          Especially where they say Geekbench scores went from 1800 (4.2) to 2100 (4.3), then 1900 with Kit Kat.

          • Dimitar Gospodinov

            Sorry I have not read that but aren’t most of Samsung’s devices banned from most of the benchmark apps anyway?

            • filaos

              Some of them only (3Dmark),and they weren’t banned yet when all the reviews were published.

    • Amine Elouakil

      “Samsung included code that forced all cores in the phones’ processors to run at maximum speed”

      Actually this is partially true, and if it was for this it would cheating while it is unethical it’s nothing to cry about as in theory the device can run at that condition in an extrem day to day usage. The relies in the fact that the Exynos version of Samsung device, OVERCLOCK it self when it detect a benchmark, which means not only the device runs in condition that will never achieve in day to day usage, but also that the SoC runs beyond it TDP limitation and everything that it can involve

    • KillEmAllx

      *Off topic*
      I can’t get over how much bigger the S5 is compared to the S4, while the screen size is almost the same.. What a shame.

    • jack

      they cheated on benchmarks (already proven) so why would you trust them again. dog-eaters.

      • Flip Jumpman

        Jack-ass has entered the building. What a fool you sound like.

    • Groud Frank

      “Artificially setting processors to max during benchmark tests skews the results and makes it so they don’t represent real-world usage.” benchmarks in general don’t produce real world results. I’ve got to say I’m liking Samsung more and more. They are toning it down and seemingly trying to take the right path. What matters to consumers is the user experience and not the specs.

      • BD1971

        Samsung is only toning it down because they got caught. If no one had found out they were doing it they wouldn’t have stopped. If they were really taking the right path they wouldn’t have done it to begin with. We shouldn’t kid ourselves into thinking ethics plays apart in business nowadays. It’s about money and what you can get away with.

        • Groud Frank

          What about the other OEMs that do the same thing but aren’t toning it down, like Sony. It’s only bad when Samsung does it, right?

          • BD1971

            No. Its bad when any company does things that are misleading. Its unfortunately become the norm nowadays when it comes to business practices. I just don’t think Samsung should be praised for changing only after there caught.

          • gord

            Sony did not cheat, and neither does Motorola.

    • ratnok

      After KitKat, benchmarks are irrelevant. All you need is a 1ghz dual core with 512mb of RAM and you’re smooth as butter.

      • K

        Not 512mb, but 1GB RAM and a dual core processor for a smooth performance.

        • ratnok

          Do you have any experience with KitKat at 512mb of RAM? It runs quite smooth on stock KitKat. Exactly as it’s designed to do.

    • dandroid13

      A Brazilian website is reporting that Samsung didn’t change anything, it’s a code change by Google that disables the “cheat”, is that correct?

    • V-Phuc

      Not even a bit. What matters to me is the TRUE AVAILABLE INTERNAL storage on a given phone! What difference does it make to take an app to launch between 0.2 and 0.5 sec? I’ve wasted so much more time being stuck in traffic, don’t you think?!!! Stop these meaningless results.

    • Steven Radcliffe

      Have a S3.. Have not upgraded bc heard the 3 is a better performer then the 4.. def need an upgrade.. whether samsung or not… plz recommendations considering price and performance..?

    • John Wm Rickman Jr.

      “My Dog’s Better Than Your Dog; My Dog’s Better Than Your’s…
      My Dog’s Better Cause It Runs Faster Now, And My Dog’s Better Than Your’s !!!”
      SO WHAT !!!

      • Timmy

        I see your schwartz is as big as mine

    • Scott Wood

      It’s a simple example of money over product. Samsung and apparently other manufacturers do this for revenue. It makes their phone standout on benchmarks, whereas in reality, it’s a fraud. New concept… money follows innovation, ease of use, and functionality. But it’s a profit driven economy, this happens to be the dark side of business. Although, if one advertises (willfully knowing) a false claim, it’s illegal. Go figure, Samsung doing something misleading or rather skirting legalities.

    • Be Cool

      LOL
      You wrote that other Android manufacturers did same, but always put Samsung in the title. Why?
      This was long before Note 3 to come, but Anand and the whole anti-samsung propaganda “discover” it just when Note 3 come out. Then when the job was done, and he was pressed to the wall with facts , he wrote article “All other do the same”. Then to wash his hands he wrote “Samsung didn’t need to do it, because anyhow , Note 3 beat them all. ”
      It’s not only Sony. It’s HTC also and all the others. And how could this be “skewing”, if the SoC works on full power when heeded???
      But may favorite is:
      “Are you glad Samsung is no longer skewing benchmark results?”
      ROTFL
      (now you can ban me)

    • Be Cool

      So you didn’t like my comment and just deleted it???

      Does it mean that I shouldn’t say the truth? Or otherwise I will be banned ?
      What was wrong in the my comment? If I was wrong- why not say , what? Smash me with arguments! Make me shut up, and feel silly.
      Did I break the rules of commenting?

    Popular

    Latest