Mark Zuckerberg demos a tool for building virtual worlds using voice commands

Meta, formerly known as Facebook, today showed off a prototype of an AI system that enables people to generate or import things into a virtual world just by using voice commands. The company sees the tool, which is called “Builder Bot,” as an “exploratory concept” that shows AI’s potential for creating new worlds in the metaverse. Meta CEO Mark Zuckerberg showed off the prototype at the Meta AI: Inside the Lab event on Wednesday in a pre-recorded demo video.

In the video, Zuckerberg explained the process of building parts of a virtual world by describing them. He begins with the prompt, “let’s go to a park.” The bot then creates a 3D landscape of a park with green grass and trees. Zuckerberg then says “actually, let’s go to the beach,” after which the bot replaces the current landscape with a new one of sand and water. He then says he wants to add clouds and notes that everything is AI-generated. Zuckerberg then changes up the landscape by saying he’d rather have altocumulus clouds, which is meant to demonstrate how specific the voice commands can be.

He then points to a specific area of the water and says “let’s add an island over there,” and then the bot creates one. Zuckerberg then issues several other voice commands, such as adding trees and a picnic blanket. He also adds the sound of seagulls and whales. At one point, he even adds a hydrofoil — a nod to one of his favorite hobbies, which later turned into a meme. 

Throughout the video, the Builder Bot appears to be using voice commands to create 3D objects and placing them onto the landscape. In the blog post that announced the prototype, Meta said the tool will “fuel creativity in the metaverse,” but didn’t offer technical details.

Image Credits: Meta

The technology, if successful, could have implications for other VR worlds and platforms. For instance, game platform Roblox has begun testing voice features in recent months and it offers its own developer platform. It’s interesting to imagine how a company like this could one day adopt the type of technology displayed in Meta’s prototype for a similar world-building experience.

However, in its present form, the world that Builder Bot created is fairly simple in terms of its looks and functionality. And while it may be fun at first to speak commands to have objects appear, it’s not a scalable way to build more complex 3D environments. If anything, it could be a fun kids’ playground for an entry-level experience in virtual world creation. (Unfortunately, though, Meta is already proving that its virtual environment may not be a safe place for children.)

Meta’s unveiling of its prototype comes as the company is spending billions on the metaverse. Earlier this month, Meta released financials for its Reality Labs division for the first time and revealed that it lost more than $10 billion last year. The company said it expects the losses will only get bigger this year, which indicates that Meta has seemingly endless money to spend on building out the metaverse and likely has quite a bit of time to pull it off ahead of other smaller companies. The company’s deep investments into the metaverse also suggest we’ll see more prototypes that are designed to advance the metaverse.

Although a true “metaverse” may not yet exist, the buzzword is being used by Zuckerberg and Meta a lot over the past year and even fueled its recent corporate branding change. Zuckerberg previously described the metaverse to investors as a “virtual environment where you can be present with people in digital spaces. You can kind of think of this as an embodied internet that you’re inside of rather than just looking at.”

Meta made a few other announcements at its event today, including its plans for AI-powered chatbots, an AI system card tool, and a Universal Speech Translator. The company says the latter could provide instantaneous speech-to-speech translation across all languages, including those that are mostly spoken, which would be a leap over existing translation systems. Meta noted that 20% of the world’s population does not speak languages covered by current translation tools and that it plans to overcome this by deploying new machine learning techniques.