Google has taken up the challenge of turning the virtual reality experience into a shared experience by making the headset disappear – Google headset removal – allowing others to see the person behind the mask while they enjoy immersive worlds.
The project involves researchers from Google’s Machine Perception team, Daydream Labs team and the YouTube Spaces recording studio. They film a participant standing in front of a green screen to generate a background for the mixed reality output. The subjects face is filmed using ‘dynamic face model capture’, which records images that are then combined to build a dynamic model of the human face.
Seated in front of a camera, the subject is made to look in all directions while their gaze is filmed. The collection of eye movement and blink recordings are stored in a gaze database.
SMI’s Eye Tracking HMD based on HTC Vive is needed when the experiment finally comes together in the studio. As this video shows, the subjects paint and play games in 3D, while their faces are seen through a semi-transparent headset which Google calls the “scuba mask effect”. For the first time someone wearing a VR headset and all their actions are visible for others to see.The reason we can see their eyes move is because gaze direction is measured by SMI Eye Tracking and in turn, that eye tracking data triggers the appropriate stored imagery from the gaze database; making for a realistic depiction of the person behind the mask.
In the final scene of this video, the player removes the headset to reveal his face which is uncannily like that which has just been created for this project.
So why have they done it? After all, this isn’t a product or app you can download and use yourself. Tom’s Hardware believes such mixed reality experiences have value in that they could help to “catapult VR into the mainstream”. Eye tracking – and human faces with realistic eye movements – help make VR real as a shared, multi-media experience.