Even if the results are somewhere between moderately unpleasant and horrific, Facebook Reality Labs aims to help people see your eyes when you’re in virtual reality.
FRL published a report earlier this week on “reverse passthrough VR,” a method for making virtual reality headsets less physically isolating.
Researchers have developed a method for projecting your face onto the front of a headset, albeit they stress that it is still in its early stages.
Passthrough VR is a function that shows a live video stream from a headset’s cameras while the user is still wearing the device, allowing them to see the actual world.
When users leave the confines of their virtual reality realm, Facebook’s Oculus Quest platform, for example, displays a passthrough feed.
It can be used to swiftly exit VR and to enable augmented reality by adding virtual items to the camera stream.
However, as FRL points out, even though the wearer can see them precisely, the people surrounding them are unable to make eye contact.
Bystanders who are accustomed to seeing their friend or coworker’s bare face will find this awkward.
Nathan Matsuda, a scientist at FRL, sought to fix that.
According to a blog post, Matsuda began in 2019 by mounting a 3D display onto an Oculus Rift S headset.
The screen showed a virtual representation of Matsuda’s upper face, with custom-rigged eye-tracking cameras capturing where Matsuda was looking so that his avatar’s eyes could gaze in the same direction.
Matsuda wore a telepresence tablet that displayed a duplicate of his own face as a result, which is probably just as embarrassing but with a more intriguingly postmodern twist.
According to the blog article, FRL chief scientist Michael Abrash didn’t think the proposal was very feasible, which is understandable.
He says, “My first thought was that it was kind of a ridiculous idea, at best a novelty.” “However, I don’t tell researchers what to do since creativity requires the opportunity to try new things.”
Matsuda took the idea and led a team in designing a sleek design over the next two years. The team’s prototype headset adds a stack of lenses and cameras to a regular VR headset display, which it presented ahead of next week’s SIGGRAPH conference.
Inside the headset, the stereo cameras record an image of the face and eyes, and their movement is mapped into a digital model of the face. The image is then projected onto a light field display that faces outward.
That display gives the impression of gazing through thick goggles and seeing a pair of eyes, but you’re actually looking at a real-time animated duplicate. The display can go blank if the wearer returns to full VR, indicating that they are no longer engaged with the outside world.
The end result is a pair of octagonal goggles that belong in a Terry Gilliam film. FRL demonstrated the technology with its more realistic Codec Avatars, as seen below, as well as a simple portrayal of a virtual face.
Individual components of the system aren’t all revolutionary, according to FRL. HTC already offers a face tracking add-on for its Vive Pro headsets; it maps movement to an avatar inside VR rather than an outward-facing screen, but the concept is the same.
The focus of this week’s research is on the possibilities of light field displays and how the system might improve in-person social interactions.
Although many of those glasses have darkened lenses, and as Road to VR points out, projected light on clear lenses can potentially block your gaze, HoloLens-style projection glasses theoretically leave your face considerably clearer than passthrough displays.
However, while businesses such as Apple are said to be experimenting with passthrough designs, Facebook’s recent research demonstrates that a solid screen isn’t always a barrier to eye contact… sort of.
