Sony Patents Face Recognition System for VR Headset

Summary

• For the next generation of helmets or the one beyond?

The American body that studies patents has just granted one to Sony. It is called "integration of a face tracking option for virtual reality users in a VR environment". Thus, via cameras arranged on its underside, the headset detects mouth movements and facial expressions. This should make it possible to reproduce them on the avatars of a virtual social space. This avatar would then correspond to a faithful representation of the person. Thus, when a group of players or friends meet in virtual reality, they can see each other talking, laughing, smiling, surprised or angry.

The patent also mentions a gaze tracking function. Again, it will reproduce eye movement in VR. All these aspects should bring a heightened sense of realism during interactions between people in virtual reality. However, these should become more common in a post-coronavirus world.

For the next generation of helmets or the one beyond?

The system developed by Sony reproduces the entire face while only the eyes and mouth are captured. Indeed, an important part including the nose, the cheekbones, and the browbones is masked by the helmet. The Japanese manufacturer thus mentions several techniques in its patent to circumvent this problem. Overall, they try to identify the position of the nose using proximity sensors. But also its size by determining the space it occupies in the part of the helmet provided for this purpose.

Sony engineers are not the first to explore facial recognition to recreate photorealistic avatars. Last year, Facebook had already published a few videos showing the possibilities of such a system. The result was quite impressive. Nothing says however that this technology will be present from the next generation of headsets like the PSVR 2. This patent may never even be used in a product for the general public. If this were nevertheless the case, it is certain that the social experience in virtual reality would greatly benefit.