Arm is leading the charge for mainstream AR and VR

The promises of what is possible with Virtual Reality (VR) and Augmented Reality (AR) are tantalizing, but the realization of those promises hasn’t happened as quick as hoped. The challenges to mainstream acceptance of VR and AR are not only technical, but also consumer related. A new technology needs a “killer application,” which everyone wants, to really propel it into the mainstream.

This is where productization and technology converge — sometimes the killer app can’t be built until the technology is ready. High performance, good power efficiency, and the right price are key to the success of VR and AR.

The good news is these challenges are being overcome right now. It is predicted by some analysts that consumers will have installed over 6 billion Augmented Reality mobile apps by 2022!

VR and AR products come in a huge variety of different forms. The same basic technology lies at the heart of both: displays and powerful processors to produce content for those displays.

VR and AR differ in how they use those displays and processors, but the demands are the same for both. Arm specializes in designing power-efficient processors including CPUs, GPUs, display processors, video processors, and microcontrollers. All of these processor components are used in conjunction to create VR and AR products. A typical System-on-a-Chip (SoC) like HUAWEI’s Kirin or Samsung’s Exynos use most, if not all, of these processors.

These SoCs, like the Kirin 970 or the Exynos 9810, are found in smartphones but can also be used in other embedded applications. The easiest platform for the productization of VR and AR is the smartphone. It has all the parts needed to make VR and AR successful. It’s got a high resolution display, high performance SoC, camera, GPS, and various accelerometers for detecting movement.

Providing enough CPU and GPU performance to make VR and AR useable has been a technical challenge. This is particularly important for VR, which needs high frame rates to avoid motion sickness. This has meant the leading VR and AR solutions tend to target flagship devices. But for VR and AR to become mainstream the list of target devices needs to grow.

Mobile tech on the march

Recently Arm announced two new GPUs, along with a new display processor and a new video processor. These new processors are important, as they help to deliver flagship experiences to mid-range devices. For maybe the first time, VR and AR developers have the technology to make their products for the mainstream.

The new Arm Mali-G52 offers 30 percent better performance density, 15 percent better energy efficiency and over three and a half times more machine learning improvements than its predecessor, the Mali-G51. With support for modern APIs like Vulkan and OpenGL ES 3.2, VR and AR solutions can start to target mid-range smartphones and offer usable performance, without killing battery life.

While VR applications are GPU heavy, AR solutions also rely on the display processor and the video processor. Since the “real world” input is coming from the camera as a video stream, with layers of augmentation added on top, AR uses a combination of the GPU, video processor, and display processor, along with a healthy dose of machine learning.

You might be tempted to think AR apps would really just use the GPU, but the combination of a display processor (for compositing and overlays) can result in up to 30 percent gains in efficiency compared to just using the GPU.

For ultra-efficient devices Arm also announced the Mali-G31. Compared to a dual-core Mali-G51 GPU, the Mali-G31 is 20 percent smaller and offers 20 percent better performance density, all while still supporting Vulkan! This is Arm’s smallest GPU to support Vulkan. The aim is to help developers create high quality content for millions more users than just flagship owners.

Virtual Arts

One company working closely with Arm to bring Augmented, Virtual, and Mixed Reality to the mainstream is Virtual Arts. The company is focused on developing the tools VR and AR developers need to create visually stunning experiences within a limited power budget.

The company includes many veterans from Arm and Sony Guerrilla Cambridge Audio Audio with experience not only in gaming and 3D demos, but also in performance analysis and GPU debugging.

I recently had the chance to ask Nizar Romdan, the CEO & co-founder of Virtual Arts, about the challenges faced by VR and AR companies. He told me that along with the need for high frame rates, there is also the need for high quality visuals, as the human eye can quickly spot images that look computer generated. He likened their vision for building amazing VR and AR solutions to that of building a high speed racing car that is as easy to drive as a family sedan, while using minimal amounts of fuel.

One early decision taken by the Virtual Arts team was to exclusively use Vulkan on Android rather than OpenGL ES. Vulkan has several major advantages over OpenGL ES including greater performance and better integration with multi-core CPUs.

The new Mali-G52 and Mali-G31 GPUs are enabling companies like Virtual Arts to lead the way in VR and AR by offering Vulkan support in mid-range devices.

Cutting the cord

Today’s technical challenges for untethered VR and AR are performance and energy efficiency. Arm is meeting those challenges head on with its continued investment in power-efficient GPUs. Thanks to its diverse developer ecosystem, these GPUs are being used by companies like Virtual Arts to deliver the next generation of Augmented, Virtual, and Mixed Reality apps on Android and beyond.

Next – What’s the difference between AR and VR?