Google says it's 'very comfortable' with Tensor not winning benchmarks

Google’s Pixel 6 and Pixel 7 phones are powered by semi-custom Tensor processors, but these SoCs don’t really take the fight to Apple, Qualcomm, and other chipmakers when it comes to most benchmarks.

Now, senior director of product management for Google Silicon Monika Gupta has spoken out about the Tensor line’s benchmark performance in the official Made By Google Podcast (h/t: 9to5Google).

Gupta suggested that benchmarks are only part of the story when it comes to smartphone processors:

I think classical benchmarks served a purpose at some moment in time, but I think the industry has evolved since then. And if you look at what Google is trying to do by pushing AI innovations into a smartphone — because we feel like this is the approach that will deliver helpful experiences like some of the ones I just mentioned — classical benchmarks were authored in a time where AI and phones didn’t even exist. They may tell some story, but we don’t feel like they tell the complete story.

She added that Google benchmarks the “actual software workloads” that will run on its chipsets rather than relying on benchmark apps with synthetic workloads. 

“We know exactly what we’re building for, and if that means we’re not going to win on benchmarks or not look as great on benchmarks, we’re perfectly comfortable with that, because the end-result speaks for itself,” Gupta explained.

“Like on Pixel 6 and Pixel 7, you can see all the amazing innovations that we have landed, and a lot of them were the first on Pixel. So we’re very comfortable with that approach.”

The right approach for Tensor?

It’s definitely true that benchmarks aren’t entirely representative of real-world results and workloads. For one, many benchmark tests focus on peak performance that you’ll rarely see rather than sustained performance. In fact, our own Tensor G2 testing shows that the Pixel 7 chipset lagged behind rivals when it came to peak results but actually beat some Snapdragon 8 Gen 1 phones in sustained testing. And sustained performance is a must for tasks like games and photography.

Google is also pushing machine learning in a big way for tasks like computational photography, and benchmark apps like Geekbench and 3DMark don’t really take this hardware into account. There are some AI benchmarks out there, but the varying nature of AI hardware and workloads means that even these tests don’t tell the whole story. For what it’s worth, the AI-Benchmark ranking shows that the Pixel 7’s Tensor G2 SoC slots in behind Snapdragon 8 Gen 1 and Dimensity 9000 phones.

It must also be said that this isn’t a new approach for Google. The original Tensor chipset seen in the Pixel 6 uses two Cortex-X1 CPU cores, and the company noted at the time that this approach was meant to deliver maximum efficiency at “medium” workloads. The company asserted that one Cortex-X CPU core (as seen in rival SoC designs) was only useful for winning benchmarks.