I love the idea of Live Caption on Android, but don't use it

I strongly prefer to watch video content with subtitles, and it looks like I’m not alone. A recent YouGov survey found that 38% of Americans prefer to watch content with subtitles enabled, with a massive 63% of adults under 30 watching content with subtitles enabled.

So I was extremely happy when Google announced its Live Caption feature back in 2019, initially as a Pixel-exclusive feature before landing on other top Android phones. Unfortunately, as the years have gone by, I’ve realized that I love the idea of the feature but don’t actually use it.

Live Caption was groundbreaking but I stopped using it

Simply put, Live Caption is a system-level captioning system for audio-related content on Android devices. This accessibility feature automatically generates subtitles for almost any type of content with audio in it, such as videos, voice notes, podcasts, and social media clips.

Perhaps the best thing about Live Caption is that it runs locally and offline, which means you don’t need an internet connection of any kind to use it. This also has some privacy benefits, as Google isn’t sending your data to servers to get the captions. It all sounds great on paper, but practice is another story.

My issue with Live Caption boils down to the fact that it’s simply too slow, reminiscent of live closed captioning (CC) you’d see on US TV channels. I’ve generally found that the captions are about two to three seconds behind the audio when using the tool for voice notes or video content. This isn’t the worst delay in the world and is definitely essential for accessibility purposes, but as someone with no hearing issues, I find it annoying when watching content.

It’s also worth noting that Live Caption’s accuracy leaves me wanting as well. I could overlook some of the accuracy issues if the speed were there, but alas, here we are. There’s an argument to be made that Google wants the feature to be more in line with traditional closed captioning rather than a real-time subtitle engine. But then again, the term “Live Caption” has “Live” in it.

My short Live Caption wishlist

The ideal solution would be for Google to simply speed things up, but it almost goes without saying that this is much easier said than done. One possible approach would be for Google to harness its semi-custom Tensor processors seen in its Pixel line-up to give Live Caption a boost.

We’ve seen Google bring offline voice typing to its recent Pixel phones, claiming that this feature was enabled by the TPU silicon in Tensor. So it stands to reason that the company could offer a similar speed and accuracy boost to Live Caption by taking advantage of its hardware.

It’s also unclear if Google has actually refined the Live Caption training model in a significant way since its launch four years ago. If not, the company could potentially bring a major speed and/or accuracy upgrade with a newer model for non-Pixel devices.

Another potential workaround for speed would be the ability to feed audio and/or video files into Live Caption. From here, Live Caption would generate subtitle data and use this data for subsequent playback of these files. This way, the company would be able to give the illusion of real-time subtitling when it’s simply referring to existing captioning data and syncing it correctly.

In fact, I also want the option to feed files into Live Caption and have it spit out a subtitle file that can be used elsewhere. This would be great if you’d like to play your recorded clips or downloaded videos on a different device.

It’s clear that Live Caption is a great tool for the hearing-impaired. The fact that Apple and Microsoft have copied this feature shows that Google came up with a fantastic idea.

However, it’s increasingly evident that, while Android 14 brings some much-needed accessibility additions, Live Caption needs significant speed and accuracy improvements. So we hope Google brings these upgrades to the table, especially now that a growing number of people are using subtitles.