Microsoft’s PeopleLens project helps blind kids learn social cues in conversation

Among the challenges of growing up with a visual impairment is learning and participating in the social and conversational body language used by sighted people. PeopleLens is a research project at Microsoft that helps the user stay aware of the locations and identities of the people around them, promoting richer and more spontaneous interactions.

A sighted person looking around a room can quickly tell who is where, who’s talking to whom and other basic information useful for lots of social cues and behaviors. A blind person, however, may not know who has just entered a room, or whether someone has just looked at them to prompt them to speak. This can lead to isolation and antisocial behaviors, like avoiding groups.

Researchers at Microsoft wanted to look into how technology could help a child blind since birth access that information and use it in a way that made sense for them. What they built was PeopleLens, a clever set of software tools that run on a set of AR glasses.

Using the glasses’ built-in sensors, the software can identify known faces and indicate their distance and position by providing audio cues like clicks, chimes and spoken names. For instance, a small bump noise will sound whenever the user’s head points in anyone’s direction, and if that person is within 10 feet or so, it will be followed by their name. Then a set of ascending tones helps the user direct their attention toward the person’s face. Another notification will sound if someone nearby looks at the user, and so on.

The PeopleLens software and its 3D view of the environment. Image Credits: Microsoft Research

The idea isn’t that someone would wear a device like this for life, but use it as a learning aid to improve their awareness of other cues and how to respond to them in a prosocial way. This helps a kid build the same kinds of non-verbal skills that others learn with the benefit of sight.

Right now PeopleLens is very much an experiment, though the team has been working on it for quite a while. The next step is to assemble a cohort of learners in the U.K. between the ages of 5 and 11 who can test out the device over a longer period. If you think your kid might be a good match, consider signing up at Microsoft Partner the University of Bristol’s study page.