Analyst says Intel beating ARM in benchmarks was a sham

July 13, 2013
77 32 9 28

    Intel logo Earlier this month we brought you news that Intel had destroyed ARM in benchmarks. It was a pretty big deal. Not just because it beat out chips like the Snapdragon 800 and the Exynos 5 Octa, but because it beat them out by so much. In the original benchmarks, Intel beat out the Snapdragon 800 by 11000 points and the Exynos 5 Octa by 16000 in the widely used AnTuTu benchmark. As it turns out, it may have all been a sham.

    According to Neil McAllister of The Register, analyst Jim McGregor didn’t quite like those numbers. McGregor went on an investigation and concluded that the benchmarks didn’t show Intel being better because they were actually better. It was actually an error with AnTuTu.

    During his investigation, he found that updates made to AnTuTu seemed to favor Intel chips over ARM chips. As he explains, the updates gave ARM processors a mere 50% upgrade in performance whereas Intel chips were seeing increases of up to 292%. This is what originally piqued McGregor’s interest to look more into the situation. It was a good thing he did, too, because he was completely right.

    So here is the long and short of the problem. Starting at AnTuTu 2.9.4, they began compiling using a unique compiler designed by Intel. This allowed Intel chips to bypass certain parts of the test to inflate their score. In other words, AnTuTu made a mistake and Intel’s chips cashed in. Due to this error and due to parts of the test being skipped, Intel was able to gain an unfair advantage in the test scores.

    So has this been fixed? Well, sort of. AnTuTu has released an update to fix the problem and Intel’s performance has since dropped dramatically. Check out the screen shot below for the new scores.

    Intel vs Arm

    Image Credit: Jim McGregor

    So all is right again in the universe and ARM is better than Intel?

    Well according to these test results, that would be the indication. However, ARM fans shouldn’t grab out their celebratory drinks yet. While McGregor was able to find the issue and AnTuTu fixed it, there is still more going on. AnTuTu is now going back and revising their standards. According to McGregor, there will be a new standard released sometime in August that should be more fair between the two warring behemoths. The keyword in that last sentence was “should”. Even though Intel can no longer bypass parts of the test, it is now possible that the new standard may further alter these test results.

    For now, we’ll count it like a game. Intel has one point and ARM has one point. The tie breaker will come in August with AnTuTu’s new standard. With the playing field evened out and one more proverbial inning left in the game, who comes out of 2013 with the stronger chipset? Intel or ARM? If you want to take a guess, feel free to leave a comment and let us know.

    77 32 9

    Comments

    • Madsen

      Where is SunSpider test? Intel still best in it?

    • Hoggles

      This is funny news. Shame on you Intel.

      • alexander

        This was an AnTuTu botch. I am not sure why Intel should be chastised. The compiler and compile options AnTuTu used for Atom caused the compiler to generate code for the bit operation loop that tests for a range of 32-bits of “do nothing” and then perform the do nothing in one pass. That might be good for an app you are running but it is pretty dumb for a benchmark.
        I would like to know how they figured out which compiler was used. I have not been able to see any discussion from AnTuTu.

        In some of the discussion, it was said that AnTuTu benchmark started out as the Byte nbench benchmark. I compiled the bit operation code with both the gcc and icc compilers using the -O3 option that came with the nbench source code. The icc “skip code” did not happen. I suspect AnTuTu had some other options on that caused icc to perform the match test. Value profiling is a feature of icc that might be responsible for the code.
        Interestingly, gcc 4.81 code ran faster than icc 13.1 code.
        The AnTuTu response to their botch.

        http://www.antutu.com/view.shtml?id=6797

        Now, AnTuTu has built fluent communication with ARM, Samsung, Qualcomm, and other device vendors. So, you all do not need worry any issues about the test scores/results. These professional device vendors surely supervise us for any changes on benchmark matrix, though we have non-open source codes in product. We shall do our best efforts to improve AnTuTu Benchamrk, better and better.

        As to this disputation between ARM and Intel, we have recieved notices from the both vendors. And we have already fixed this problem within the version 3.3.2 which was released on July 8th, 2013 in China (google search) , and updated product at Google Play was released on July 10th, also. We fixed the problem much earlier than the query came out.

        We are very sorry about this unhappy issue, which is caused by our aim of seeking ultimate hardware performance to mobile devices, in product version 3 which has exploited Intel complier in. We will not make any special changes to any device vendors. In the coming August 13th, we shall release our new V4 benchmark matrix, a return from ultimate hardware performance to user experience. We believe the new AnTuTu Benchamrk V4 can surpass any benchmark apps in the market, since it is the standard for users.

    • Gaja

      K900 is old Atom… What about Bay Trail?

      • Leonardo Rojas

        Is not old, it’s the lattest Atom for smartphones and it’s been only months in the market.

        • Gaja

          Yeah but it’s not K900 that crushed Snapdragon 800 just recently, it was Bay Trail, so this article is pointless, as it doesn’t say anything about Bay Trail..

          • Leonardo Rojas

            True! Good point.
            Looks like Jim McGregor couldn’t find any BayTrail powered device. Hahaha No, the article seems to be a bit confused. There did were some tests with the K900 showing it as the winner above the S4, and others both in power consuption as in overall/cpu power.

            The Lenovo K900 is just 3 or 4K points below the Snapdragon 800 score, as far as I could see. So, it shouldn’t bother McGregor so much the new generation of Atoms beat the latest from ARM. Let’s see if this keeps beig true when actual devices are released.

            • JosephHindy

              Yeah, kind of hard for regular people to test things on our own when there is no existing device that runs the hardware that is publicly available.

              That’s why there is no Bay Trail in this article, but there exists no public device that has it. You’ll have to be patient and wait for it in order to see it’s benchmarks (improved benchmarks because Bay Trail, being Intel, would have the same errors that the other intel processors had).

          • alxlr8

            I think the point of the original analyst piece, and the subsequent articles such as this one that summarise it, is that the use of optimising compilers that eradicate the intent of the benchmark are not actually doing an apples to apples comparison. ARM compilers could be set to high levels of optimisation as well, but for one reason or another, the binary built for AnTuTu on Intel DID use heavy optimisation, and the binary built for ARM did not. As such, the ARM version of the compiler was doing what the benchmrak ACTUALLY does.

            See this link for a disassembly of both benchmarks:

            http://forums.anandtech.com/showthread.php?t=2330027

    • Leonardo Rojas

      My Razr i can perform as good as Note 2, you can check it out in Youtube. Note 2 won the multitasking versus just because it has 2GB of RAM, but still the Razr i moved everything as fast and as good as the Note 2… having the Razr i “disapointing’ benchmarks scores.
      So, right now with the Razr i and the Lenovo K900, Intel is on par *the best* ARM processors in terms of excelent user experience.

      Intel should concentrate in making the devices powered by it more known. Intel has lost a good chance of showing off its power with the Razr i, many people don’t even know about it. If they had promoted more its capabilities these last 10 months right now there would be people eager for an updrade now with Intel rather than thinking in giving it a try. Silly Intel.

      Also, Lenovo is taking to long to release in more markets its K900, Intel is lossing many ustomers there too.

      • Grman Rodriguez

        Benchmarks are overrated

        • mrband

          Opinions about overrating too :p

          • lil bit

            Not yours.

      • Jerel Butler

        what if the moto X is running a dual core intel chip

    • Roberto Tomás

      ” Intel beat out the Snapdragon 800 by 11000 points and the Exynos 5 Octa by 16000 in the widely used AnTuTu benchmark” — hate to say it, but you are misinterpreting the article. That may be true, but that was the more recent Bay Trail benchmarks. The benchmark that he wrote about was the K900, which only scored 25k-27k. That’s less than either ARM chip (compared to the S4 Pro that was available at the time, it was still a pretty respectable score though).
      Assuming they continued to use benchmarks that are compiled with optimizations for intel but not for the arm chips, then yea the latest benchmark is also wrong. But that wasn’t what I read over at cnet.

      • JosephHindy

        Misinterpreting what article? I pulled those stats off a screen shot from an article one of my colleagues wrong two weeks ago (linked in the first paragraph). It’s subtracting two numbers, you can’t possibly misinterpret that. I also never specified which Intel chipset (which I probably should’ve done) so you can’t even tell which chipset I’m talking about now can you?

        • Roberto Tomás

          I meant that the benchmarks for the chip in the K900 were shown to be off, the others were not. The K900 (Clover Trail I think?) chip did not outscore current-gen ARM chips, in fact it only scored 25k the first time and ~27k later. At the time, that was better than any ARM chip, but that was right before the entire current gen started rolling out. The Bay Trail chip did outscore the ARM chips, but I didn’t see where he showed that actual benchmark results specific to the Bay Trail chip were wrong. Is that available in the reference? I’ll go check again.

    • Roberto Tomás

      “Check out the screen shot below for the new scores.” — No, these are the old scores. This was the graph they used to show how he first found that Intel had cheated: every other benchmark was favouring ARM. But AnTuTu was strangely favouring Intel.
      You can tell very clearly that this isn’t AnTuTu using a proper compiler, because the results there are all sorts of benchmarks. AnTuTu isn’t a meta-benchmark made of other benchmarks.

      • JosephHindy

        If AnTuTu wasn’t using the proper compiler, then why did they admit that they were and change it…then state that they were changing it again next month?

        • Roberto Tomás

          At this point I am wondering if you realise which image you posted to the article. Read the labels on the graph. The red line is the K900 scores.

          Also, reading the articles you cited, it is clear, they go through great pains to explain, that it was along with a change to the ICC compiler. They both changed the compiling settings, and apparently changed the code, to better conform to the ICC compiler. Some degree of that makes sense.

    • CpuKnight

      NOOOOO :( I was looking so forward to getting the uber powerful Atom Phones when i Recontract in a year :/ Darn but then again K900 isnt merrifield or baytrail, its saltwell so Im pretty sure it will still annhilate ARM soon enough. =

      • John

        Nah, get your head out of the clouds

    • Angel Angelov

      Why they used so Old version of Antutu? Antutu benchmark is now version 3.3.2!
      They have been at 3.x.x for nearly half an year!

    • Roberto Tomás

      Just some notes on the problems with AnTuTu:

      • Its benchmarks are in disagreement right now when compared over the ARM and Atom chips, versus most other major benchmarks. (see image in the article here)
      • It was skipping 2/3s of the work in the memory tests for the benchmarks Intel sited for Clover Trail. (see image in the article here)
      • It’s memory subsystem test is not necessarily a good test of the memory subsystem at all, and was designed to test architectures decades ago. (see http://www.bdti.com/InsideDSP/2013/07/11/JeffBierImpulseResponse )
      • There was also a CNET article that claimed (though appears to have been modified now) that the AnTuTu benchmark was built with compiler optimizations for Intel (along with the Intel-specific ICC compiler), while was run with the GCC compiler without any optimizations for ARM chips. It is unlikely that any gcc compiler setting is as optimized for a chip from Samsung, Apple, etc, as the Intel-specific ICC compiler can be for Intel Atom chips. If Intel doesn’t come to become the default platform or Android, most applications won’t benefit from the results reflected by the ICC compiler.

      • alxlr8

        It is true – see my link posted above that looks at the disass of the benchmark on ARM and Intel – large chunks of the RAM test are simply removed by ICC.

      • Roberto Tomás

        It makes sense to use a system-wide test like AnTuTu to test not CPUs, but real systems. “What are real CloverTrail+ mobile systems like?”, you might ask.

        Just in the past week, the ASUS MEMO PAD FHD 10 came out with it’s Atom Z2560. The 2580s are not for mobile devices. This tablet, with the Z2560, scores 17,659 in current AnTuTu (3.3.1 yea? this improves over the ram fiasco, at least) That is still using the ICC compiler instead of using the standard compiler — that change will come with AnTuTu 4. This is an appreciably lower score — the z2580 was touted as getting ~28k earlier, and the z2560 was touted as getting ~25k

    • Corey

      Detecting what app is running and deliberately skipping all loop iterations isn’t antutu’s fault that’s intel flat out cheating. Also the arm app was compiled with a completely transparent open source free compiler where as intels was compiled with a closed source off the shelf $700 compiler. I wouldn’t say its 1-1, I’d say disqualification for intel!

    • Jayant

      Seriously man! Trying to be an ambulance chasing lawyer or what? It was Bay Trail that demolished the Snapdragon chips

    Popular

    Latest