Gboard could soon be able to understand sign language with your phone's camera

Google’s Gboard keyboard wants nothing more than to help you get the thoughts in your head turned into words on your screen, as easily as possible. We already have so many different ways to accomplish that, from good old-fashioned tapping away at letters, to the oh-so-slick “Glide” swipe input, to the modern convenience of voice typing. But now Google could be about to seriously level up its accessibility game, as Gboard gets ready to understand sign language.

Being able to automatically convert sign language gestures into written text sounds almost fantastical, but with improvements in machine vision and AI processing, the stars are finally aligning to make this feat not just possible, but practical. Last year, Google Deep Mind teased its advanced SignGemma model for interpreting sign language input, and now we may finally be getting a look at one of its first major uses.

Checking out the changes present in version 17.8.3.939743344-beta-arm64-v8a of the Gboard app, we’re able to get a very early look at a new Sign-to-Text input option:

While we’re not yet able to get the tool actually working, this introductory pop-up gives us an early overview of what to expect.

As you can see, Google’s building this with a hybrid local/cloud model, processing your actual video input on-device in order to extract the raw gestures, and then sending those to Google for analysis and extraction of the words you’re trying to communicate. While that sounds a bit cumbersome, there are clear privacy benefits to not sharing the actual video of you signing, if that’s not totally necessary.

Despite Google doing that last leg of actual processing in the cloud, we are very curious if there will be any device restrictions on precisely which phones will be able to take advantage of Sign-to-Text.

Digging around in the app for text strings related to this capability, we find messages meant to help users improve their visibility to the camera:

Even with a strong general sense for how Sign-to-Text will likely operate, we still have plenty of questions about the all-important details here. For instance, it feels probable that Google would support American Sign Language right out of the gate, but what about British Sign Language, or any of the dozens of other regional variants around the world? Right now, we don’t yet have any real insight into how those options could line up.

While there’s clearly still more to learn about this tool, it’s already one that’s easy to get excited about. This sort of input solution is exactly the sort of thing smartphones are uniquely equipped to deliver, and the fact that this one could open up new communication options for users who need them most is just really cool to see Google working on. Hopefully we’re able to give you a preview of it in operation soon.