Qualcomm benchmarking controversy: What's happening?

Yesterday, Qualcomm officially unveiled the new Snapdragon X Plus chipset. Designed for Windows on Arm laptops, the Snapdragon X Plus (and its more powerful sibling, the Snapdragon X Elite) is designed to bring to Windows users what macOS users have enjoyed since the original launch of Apple’s M-series chips: powerful, fan-less laptops with crazy-long battery life.

If you are just catching up on this, I highly recommend watching the video embedded above. I give a lot of context to the Snapdragon X series and how it might change the landscape for Windows laptops.

Unfortunately, not everyone is as excited about these new Snapdragon X chips as Qualcomm. Charlie Demerjian at SemiAccurate recently penned a long opinion piece that claims Qualcomm is “cheating” when it comes to the promoted benchmarks of the X Elite and X Plus. This has started to spread on social media and sparked a broader conversation about Qualcomm, Windows on Arm, and the integrity of these chips.

Android Authority has attended multiple Qualcomm events that featured hands-on access to Snapdragon X machines, so we thought we’d jump into the conversation. Here, I want to discuss what’s happening and what we can expect to see when retail machines with the Snapdragon X chips become available.

Qualcomm benchmarking controversy: Some background

Most people reading this are familiar with benchmarking, which involves putting hardware through an objective series of tests to determine how it compares in various metrics to other hardware. If you know about benchmarking, you know that it can also be manipulated to either increase or decrease the score. For example, running a smartphone benchmark while the phone is on a hot table in direct sunlight will likely give you a poor result compared to running that same benchmark in a walk-in refrigerator.

In other words, just having a benchmark number isn’t the whole story, especially if the person conducting the benchmark left out crucial details about how it was obtained.

During all the Snapdragon X events Android Authority has attended since October 2023, Qualcomm has had plenty of benchmarks to show us. As one would expect, all of them show the Snapdragon X Elite and X Plus running circles around competitor chipsets, including Apple’s M3 and the latest Core Ultra chips from Intel. In all cases, Qualcomm claims the X series processors are more powerful, more battery efficient, and better at conducting AI-based tasks. In the case of Apple, in particular, this is pretty incredible when you consider Apple has already launched three generations of retail-ready Arm chips, and Qualcomm hasn’t launched any of the X series to retail yet.

The problem, though, is that Qualcomm has repeatedly dodged questions about how it obtained these benchmarks. At an event I attended before the launch of the Snapdragon X Plus, Qualcomm showed how much more powerful the chip is than the Apple M3 silicon. I raised my hand and asked the presenter what the chip thermal design power (TDP) was for both machines used for this benchmark since the information wasn’t on the slide. The presenter could not answer and said they would get back to me (which they didn’t).

To my knowledge, the chip TDP of the Snapdragon X Elite/Plus benchmarks has never been disclosed to anyone (remember that device TDP and chip TDP are two different things). This is a huge problem because a significant TDP disparity would dramatically alter the benchmark’s pedigree. In other words, if the Apple M3 laptop is running at its standard 20W TDP but the Snapdragon X processor is running at 80W, the benchmarks are no longer comparable.

When you combine this with the fact that Qualcomm hasn’t allowed any fully independent testing of the X series, you have a lot of skepticism about its claims. Even when we benchmarked the Snapdragon X Elite, it was a very hands-off affair. We could only watch Qualcomm’s chosen benchmarks happen and could not install other apps or check details about the benchmarking device.

Long story short, Qualcomm has a very “trust us, bro” attitude regarding its claims for the Snapdragon X series.

New claims of ‘cheating’

In the previously mentioned SemiAccurate article, Charlie Demerjian claims to have spoken to anonymous sources with direct intelligence on internal Qualcomm and partner OEM businesses. Among many things claimed in the article, here are some notable quotes:

• “SemiAccurate is 100% confident in saying that some of the numbers Qualcomm was showing off [to press] can not be reproduced with the settings they claim.”
• “After OEMs got initial samples and made something close to the final designs, SemiAccurate got reports of poor performance. By poor we mean far sub-50% of the numbers Qualcomm was telling them in the technical docs and presentations.”
• “Later, with more Snapdragon X Elite samples in the wild and many more revisions of [Windows on Arm], we got similar reports from OEMs and another Tier 1. Both reported numbers that were nowhere close to what Qualcomm promised. How not close? Above 50% this time but one used the term “Celeron” to describe performance.”

Essentially, Demerjian claims that Qualcomm is using souped-up benchmarks to sell the Snapdragon X series not only to journalists but also to the companies that will make the laptops in which these chips will actually appear.

Unfortunately, Demerjian has no hard evidence to show for these claims. The reason for this is that SemiAccurate “facilitated some questions between OEMs and Qualcomm engineers, which may point to a source if we aren’t careful.” Demerjian adds, “anonymity of our sources will always come first.” That’s all well and good, but SemiAccurate is basically doing the exact same thing it’s calling out Qualcomm for doing: “trust us, bro.”

Our take and Qualcomm’s statements

For this story, we reached out to Qualcomm. When asked about the SemiAccurate article, Qualcomm had this to say:

We stand behind our performance claims and are excited for consumers to get their hands on Snapdragon X Elite and X Plus devices soon.

When asked about its testing methodology, it said:

Our benchmarks are measured in AC mode (wall charger connected), and are in Balanced mode (default on retail devices).

Notably, when we asked about the chip TDP, it pointed us to a collection of press materials that do not contain this information. This appears to be the one thing the company won’t tell anyone for whatever reason. With this lack of official info, we can only assume the chip TDP must be high — at least higher than Apple’s 20W. If it was 20W or lower, after all, Qualcomm would likely openly tell anyone who wants that information.

Although we can’t answer the TDP question definitively at the moment, we can give you a sneak peek at what to expect from Qualcomm’s chips in this area. Thanks to frequent Android Authority collaborator Kamila Wojciechowska, we have some leaked benchmarks of various Snapdragon X machines, which you can read about at the link. The whole thing is really interesting, but the gist is that the top-of-the-line Snapdragon X Elite machine has a pretty high TDP as it pushes clocks higher. The other Elite and Plus models are reported to be closer to Apple’s M3 Pro and M2, though perhaps also still a bit higher in power draw.

Getting back to our Qualcomm questions, we also asked the company when we could test out retail-ready laptops with Snapdragon X processors, but it said it couldn’t share information about its OEM partners’ plans. It then said, “products powered by Snapdragon X Elite and our brand new Snapdragon X Plus will be announced in mid-2024,” which is info we’ve already had.

Personally, I am very excited about the Snapdragon X series. Even if Qualcomm’s benchmarking claims turn out to be fudged — or even outright lies — the possibility of using a powerful Windows laptop that sips on its battery is too enticing for me to reject outright. Even if a MacBook outperforms it in all metrics, if I can use a Snapdragon X laptop to write an article, edit a video, and play Baldur’s Gate 3 on an airplane ride while on battery the whole time, I’ll buy one instantly. Numbers on a page don’t matter to me as much as real-world experience.

We’ll find out the whole truth in just a few weeks, as we expect Microsoft to unveil a Surface laptop running a Snapdragon X chip in May for a June rollout. Once that laptop is out there, Qualcomm’s promoted early benchmarks will stand the toughest scrutiny. If the benchmarks match up, then Charlie Demerjian and others will need to eat their hats. If the benchmarks are lacking, well…Qualcomm will have a mountain of negative press to deal with, putting the future success of the Snapdragon X series (and, consequently, Windows on Arm) into serious jeopardy.