What's driving the Snapdragon 865 camera? Two gigapixels.

Two gigapixels per second. This spec may seem like a relatively arbitrary improvement in Qualcomm’s Snapdragon 865 series, but it’s the biggest driving force behind most of the new Snapdragon 865 camera features.

I had a moment to sit down with Qualcomm’s Judd Heape, Senior Director of Product Management for Camera at Qualcomm, to talk about the new capabilities.

How is Qualcomm doing this?

Hitting two gigapixels per second is no small feat. The Spectra ISP in the Snapdragon 855 could achieve 1.4 gigapixels per second, so a 40% jump seems like a big deal, especially where thermals are involved. And while you normally need to crank clock speed to see better performance, Qualcomm decided to do the opposite.

“We knew we couldn’t just keep increasing frequency,” explained Heape. “You can’t go into the multi-universe of frequencies and still be thermally viable. So now instead of processing one pixel at a time, every clock cycle we’re processing four.”

Instead of pushing more power into the ISP to process more clocks, Qualcomm is slightly lowering the clock speed and processing four pixels per clock instead of one. This results in 16% better power efficiency and much more data per clock — vital for the development of higher-resolution cameras and better AI. Where pure clock speed used to reign supreme, Qualcomm has found a new home in multi-pixel processing.

This shift will speed up current camera sensors a lot. But what about future technologies? For that, Qualcomm is setting its sights on ultra-high resolution.

Huge 200 megapixel photos

With all that extra bandwidth, Qualcomm is looking at what’s next for imaging technology. While I would argue that photosite size and sensor size are much more important than strict megapixels, capacity for higher resolution sensors enables things like higher resolution photos and videos.

“200MP is the logical next step for quad-CFA,” said Heape. “It might even turn into 8×8 or 16×16 in the future. Right now we’re working with partners on a 200MP sensor.”

Quad-CFA is the technology currently used to bin pixels for better light gathering. The CFA or camera filter array takes groups of pixels to capture more light for better imaging, and a quad-CFA binned 200MP sensor would produce 50MP images. Assuming 8×8 CFA’s became popular in the future, the image would bin to 25MP but have a much larger effective photosite, resulting in better images, especially in non-ideal lighting conditions.

Without two gigapixels per second processing, a 200MP sensor would suffer from distracting shutter lag. The faster the ISP can process the data, the faster you can get back to taking photos. For ODMs that would prefer to stick to lower resolution sensors, shutter lag as a concept should become a thing of the past.

Something else this new speed enables is better autofocus. Now, autofocus points can cover the entire frame to cover an area nine times as large as before. This should help you photos lock on to your subject much faster, and it will make video performance much better as well.

Better video is coming

With two gigapixels per second processing, Qualcomm can record in several new resolutions and frame rates. You could do 8K at 30fps, 4K at 120fps, or even 720p at 960fps for sustained periods.

This opens up a world of new video capabilities in smartphones. 8K cameras and super slow-mo cameras are currently only available to those with deep pockets, and even though the quality of a smartphone won’t match that of a cinema camera any time soon, artistic creativity is arguably more important than raw quality.

Each of these resolution/frame rate combinations takes about the same data throughput to process. 8K at 30fps is roughly equivalent to 4k at 120fps, and you could even split the work into two 4K 60fps streams. Shooting in high quality with front and rear, or main and wide-angle cameras at 60fps, is now doable. It’s not out of the picture to assume Qualcomm may be able to split stream processing in the future to enable recording from four or more cameras at once.

Where slow-motion is concerned, 960fps video has been done before. The difference here is that in older applications, you could only take a short burst of 960fps video before you’d have to stop. With the Snapdragon 865, you can take sustained 720p 960fps video for as long as you like. The energy savings here are actually quite a big deal, and you’d normally need a huge specialized camera with crazy cooling to get sustained high frame rate video.

“Smartphones are starting to take the place of cameras,” continued Heape. “They already are for DSLRs for videos, and not just for snapshots.”

8K ready

One of the big problems Qualcomm wanted to tackle with Snapdragon 865 is 8K video. As 8K TVs become more popular, Qualcomm wanted to make sure that people could capture 8K footage on their smartphones for viewing on 8K TVs. For people who want to take slow-motion footage more easily, the new ISP is addressing this, too.

“8K seems like the next logical step,” said Heape. “We want customers to capture 8K content on their devices so they can view it on televisions.”

Don’t miss: Samsung Galaxy S11 to have 108MP camera with 8K video?

This also highlights Qualcomm’s improved power efficiency in the Snapdragon 865. While that 16% benefit doesn’t seem like much, it helps a lot when you’re pumping so much data through the processor.

What else uses a lot of data? Artificial intelligence.

The Snapdragon 865 camera accelerates AI processing

Qualcomm’s new Hexagon 698 DSP is rated to be two times the speed as the last generation. This is useful for advanced computational photography techniques like semantic segmentation. The camera can differentiate between parts of the scene like skin, clothes and more, and apply different filters to each material to make it look its best.

This kind of processing is intensive and requires a lot of tensor cores to work. With the new speed enabled by AI processing, images could start to look a lot better right out of the camera. No need for editing in post.

One thing this new spec pulls into question is the usefulness of Google’s Pixel Visual Core. That processor is specifically designed to handle AI tensor workflows, and with Qualcomm’s catching up in speed and capability, it makes us wonder if Google still needs its own tensor accelerator.

With all this in mind, it looks like multi-pixel processing is going to be the future of ISPs. Whether we see Qualcomm branching out even further to process eight or even more pixels per clock remains to be seen, but this process seems like it could speed up image processing considerably. Two gigapixels is already a huge leap, but next year may prove even more interesting.