Microsoft’s new Power Apps feature turns sketches into apps

Showcasing its investments in AI, Microsoft today announced Power Apps Express Design, a new feature in the company’s no-code/low-code Power Apps service that can convert an uploaded PDF, PowerPoint, Figma design files, screenshots, or hand-drawn sketch into a working app (after some fiddling). Unveiled at Build 2022, Microsoft said that Express Design employs “industry-leading” AI to scan inputs and produce working app controls backed with data storage.

Express Design is the first major AI-powered capability to arrive in Power Apps following the launch of Power Fx, which leverages OpenAI’s GPT-3 to translate instructions like “find products where the name starts with ‘kids'” into formulas — and it taps into the growing no-code/low-code trend. According to Gartner, as much as 70% of applications will be developed using no-code/low-code technology, including AI, by 2025. The firm further predicts that “citizen developers” (i.e., those without formal computer science backgrounds) will begin to far outnumber traditional developers at large organizations within the year.

Microsoft explained how Express Design works in a series of blog posts. Object detection models from Microsoft’s Azure Cognitive Services platform perform image recognition, mapping forms and figures in documents (including sketches) to the constructs that exist within an app. Express Design can understand what might be a button, grouping, or a text box and generate an app automatically based on those drawings — or at least an approximation of an app.

Image Credits: Microsoft

Express Design doesn’t do all the hard work for you. Once it scans a file, app components such as fields, text labels, date pickers, and buttons have to be manually tagged and connected to a database in Microsoft’s Dataverse service. As with apps created the usual way within Power Apps, users can add additional screens (either manually or by converting new sketches) and upload images to customize the background, buttons, and other components before publishing.

Interestingly, Microsoft first experimented with transforming sketches to code back in 2018 with Sketch2Code, a research project that used AI to convert hand-written drawings to working HTML prototypes. Many others have investigated the idea, including Airbnb and Alibaba, which developed their own AI systems to translate interface sketches into product mockups and code.

On the commercial side, startup Zecoda uses AI to turn design files into front-end code automatically. But Express Design has the advantage of Microsoft scale.

“One of the big pushes we’ve been doing is to go to the next level, to go from hundreds of millions of people that can use these tools to billions of people that can use these tools,” Charles Lamanna, Microsoft corporate vice president of business apps and platform, was quoted as saying in a blog post. “If we want everybody to be a developer, we can’t plan on teaching everyone how to write Python code or JavaScript. That’s not possible. But it is possible if we create the right experiences and get them in front of enough people who can click and drag and drop and use concepts that are familiar to create amazing solutions.”

Express Design is available now to all Power Apps customers.