Qualcomm explains how the Snapdragon 888 is changing the camera game (Video!)

In smartphones, an ISP, or an Image Signal Processor is responsible for basic image processing tasks, like white balance, autofocus, and HDR. Over the years, these processors have become more advanced and full-fledged, gaining the ability to process much more data at once. This is what has lead to many of the advancements we’ve seen in smartphone camera quality in recent years.

Last year, the Spectra 480 ISP in Qualcomm’s Snapdragon 865 broke the two-gigapixel barrier. That is, it had the ability to process two billion pixels in one second, unlocking some pretty incredible features in smartphone cameras.

Don’t miss: The top Qualcomm Snapdragon 888 features you need to know

Processing that much data at once allowed support for 200MP image capture, 8K video recording, and semantic segmentation for better AI photo enhancement, among other things. With a 40% jump in overall speed, it was a pretty massive upgrade over the Spectra 380 found in the Snapdragon 855.

This year, Qualcomm is looking to trump last year’s ISP in a big way, touting 35% faster performance with its new processor, the Qualcomm Snapdragon 888. But what is Qualcomm doing with 2.7 gigapixels of throughput? Android Authority had the opportunity to talk to Judd Heape, VP of Product Management for Camera at Qualcomm to find out.

Three cameras at once

One of the biggest changes coming to Qualcomm’s Snapdragon 888 is the addition of a third image signal processor. Since the Spectra ISP was first introduced, previous Snapdragon SoCs packed two of them. That meant smartphones using Qualcomm’s chips could record with two cameras at the same time, but it also meant Qualcomm could have much higher throughput to a single camera. The company’s switch to processing four pixels at once in the Snapdragon 865 is what allowed for the massive 40% performance gain in its last generation ISP. This year, however, Qualcomm is taking a simpler approach: volume.

Read more: Snapdragon SoC guide — all of Qualcomm’s smartphone processors explained

With the Snapdragon 888, the company is adding another image signal processor to its SoC. That brings the total to three for the first time in a Qualcomm chip. This unlocks a slew of new features, but one of the most interesting is support for recording with three cameras at once. Considering many flagship smartphones now have at least three rear cameras, the third ISP allows users to record three focal lengths at the same time, up to 4K 10-bit video at 30fps. If you’re recording a scene and want to jump between three different focal lengths while editing, the Snapdragon 888 has support for that. OEMs will have to implement it into their devices, however.

Using three ISP’s also unlocks improvements in smartphone camera zoom. Last year’s Snapdragon 865 enabled smooth zoom between sensors, giving the impression that you were using a single telephoto zoom and not three separate lenses. But with just two ISPs, it wasn’t so easy for the chipmaker to know which lens you were going to switch to at any given time.

“Before, we had to guess which lens you were going to switch to,” says Judd Heape, VP of Product Management for Camera at Qualcomm. “Now, we don’t have to anymore.”

While you would generally see a smooth zoom between lenses, there could often be a delay when switching one ISP to another sensor if you swapped focal lengths unexpectedly. With three ISP’s, Qualcomm can have all three sensors active at once, assuming your smartphone only has three rear cameras.

Super-fast capture and playback

While devices running the Snapdragon 888 can capture three 10-bit 4k HDR video streams at 30fps due to that third ISP, Qualcomm considered what would happen if it funneled all that throughput into one place. After all, many modern smartphones have 90Hz, 120Hz, and even 144Hz displays. Shouldn’t you be able to record video at the same rate?

With three Spectra 580 ISPs in the Snapdragon 888, the chip can achieve 2.7Gpix/sec (gigapixels per second) of throughput. With this much data, Qualcomm can capture and playback a single video stream at 4K 120fps. This means if you have a smartphone with a 120Hz display, you’ll be able to capture and view video at the frame rate of your display. If you thought 60fps video was smooth, this is the next step forward.

Of course, just like HDR 10 and Dolby HDR video, both the device you watch the video on and the viewing platform have to support 120fps playback. Currently, YouTube only supports playback of 4K video at up to 60fps, for example. If you share directly to another device, that device will also need to have a 120Hz display to experience the video how it was captured. Fortunately, the more popular the format gets, the more likely devices and hosing services will support it.

Computational HDR video

While smartphone cameras can now produce photos with an astounding dynamic range due to HDR processing, smartphone video generally hasn’t benefited from the same improvements. In fact, this is a big reason the Google Pixel 5’s video recording isn’t nearly as good as its photo capabilities. While computational photography has seen rapid development in the last few years, video has generally remained an afterthought.

Now, Qualcomm is hoping to change that with staggered HDR video recording. First, I should note that staggered HDR is much different from the likes of HDR 10 and Dolby HDR video. While those are standards defining the brightness of a display and how much color detail can be captured in highlight and shadow areas, staggered HDR video aims to achieve much the same goal as HDR photos. It uses multi-exposure processing to retain as much highlight and shadow detail as possible.

Staggered HDR video captures multiple exposures for each frame, each at a different shutter speed. Because of this, the sensor can retain detail in both the highlight and shadow areas, then fuse exposures to achieve a more balanced video stream. Heape tells me the Snapdragon 888 can theoretically do this at 60fps, capturing 120 frames in one second. However, he’s opting to recommend 30fps Computational HDR recording to most OEMs for now.

Currently, the Snapdragon 888 will fuse two exposures per frame, but Heape didn’t rule out three or more exposures per frame coming in future Snapdragon models for even better dynamic range.

“We could certainly see three or more exposures per frame in the future,” says Heape. “That’s definitely not out of the question.”

84MP images with zero shutter lag

On smartphones, capturing a video is nearly the same as displaying a constant stream from the camera to your display. The only difference is actually storing these images. Even so, most smartphones will still temporarily store frames in RAM to capture the moments before, during, and after you press the shutter button. This is the technology that enables things like HDR photos, live photos, and so on.

Last year, the Qualcomm Snapdragon 865 unlocked the ability to display and capture 64MP images with zero shutter lag. This meant a smartphone running the chip was able to display a 64MP image stream at 30fps and have no lag or blackout between tapping the shutter and storing the image in memory. Now you’re probably thinking, “Wait a minute, if it only had support for 64MP images, how did phones like the Samsung Galaxy Note 20 Ultra capture 108MP photos?”

Here’s what Heape had to say:

For phones running the Snapdragon 865, 108MP images were actually previewed at 1/4 resolution. When the user tapped the shutter button, the sensor would switch to 108MP mode for one frame, dump that frame into memory, and resume displaying the scene at 1/4 resolution. Because of this, Qualcomm does not consider 108MP images to have zero shutter lag.

“There isn’t any image sensor, MIPI PHY interface, or ISP that can truly run at 108MP at 30fps (3.2Gpix/sec) in ZSL mode today,” continued Heape. “But don’t rule that out for the future.”

That said, the 35% speed increase in the Snapdragon 888 ISP has unlocked 84MP images with zero shutter lag. This means users will see the full resolution image before they tap the shutter button, up to 84MP. While the Snapdragon 888 hasn’t quite hit the 3.2Gpix/sec quota it needs to display 108MP images in full resolution, improvements in the last two generations make 108MP ZSL images look promising for the near future.

Smarter autofocus

In the Snapdragon 888, Qualcomm set out to improve what it calls “the three A’s.” Those are auto exposure, auto white balance, and autofocus. To do this, the company used machine learning on an enormous set of images to train the Snapdragon 888 on what an image should look like in a given circumstance. Yet, in the case of autofocus, Qualcomm went even further. It trained the chip using real people.

We used AI and machine learning to train auto exposure, white balance and autofocus,” says Heape. “But in the case of autofocus, we used real people.”

See also: The best Android camera phones you can get

Qualcomm used VR headsets with eye-tracking technology to track where a human looked when presented with a given image. Even if an object is in the foreground, that doesn’t mean it’s the subject of the photo. Humans are quite good at distinguishing the true subject in a given scene, so training the chip using people’s eye movements allows the Snapdragon 888 to focus on the true subject with much higher precision. Qualcomm says they used hundreds of subjects and thousands of images to create this data set. It should hopefully improve even further in future Snapdragon models.

Better autofocus alongside multi-frame noise reduction and Qualcomm’s HDR engine also allows sensors to do something smartphones haven’t done before — focus at just .1 lux of brightness. While previous devices would get confused and couldn’t operate effectively at this extreme level of darkness, Qualcomm is confident its new low-light architecture can achieve this.

There are even more camera improvements coming to devices running the Snapdragon 888. However, many of those changes have to do with other components like the Hexagon AI engine. A third ISP alone has given Qualcomm a massive leap in processing capabilities in the Snapdragon 888, and we’re hopeful that we’ll see similar improvements in future Snapdragon generations.