Google turns to machine learning to advance translation of text out in the real world

Google is giving its translation service an upgrade with a new machine learning-powered addition that will allow users to more easily translate text that appears in the real world, like on storefronts, menus, documents, business cards and other items. Instead of covering up the original text with the translation — the new feature will instead smartly overly the translated text on top of the image, while also rebuilding the pixels underneath with an A.I.-generated background to make the process of reading the translation feel more natural.

“Often it’s that combination of the word plus the context — like the background image — that really brings meaning to what you’re seeing,” explained Cathy Edwards, VP and GM of Google Search, in a briefing ahead of today’s announcement. “You don’t want to translate a text to cover up that important context that can come through in the images,” she said.

Image Credits: Google

To make this process work, Google is using a machine learning technology known as generative adversarial networks, otherwise known as GAN models — the same technology that powers the Magic Eraser feature to remove objects from photos taken on the Google Pixel smartphones. This advancement will allow Google to now blend the translated text into even very complex images, making the translation feel natural and seamless, the company says. It should seem as if you’re looking at the item or object itself with translated text, not an overlay obscuring the image.

The feature is another development that seems to point to Google’s plans to further invest in the creation of new AR glasses, as an ability to translate text in the real world could be a key selling point for such a device. The company noted that every month, people use Google to translate text and images over a billion times in more than 100 languages. It also this year began testing AR prototypes in public settings with a handful of employees and trusted testers, it said.

While there’s obvious demand for better translation, it’s not clear if users will prefer to use their smartphone for translations rather than special eyewear. After all, Google’s first entry into the smartglasses space, Google Glass, ultimately failed as a consumer product.

Google didn’t speak to its long-term plans for the translation feature today, noting only that it would arrive sometime later this year.

Google turns to machine learning to advance translation of text out in the real world by Sarah Perez originally published on TechCrunch