VR Controller - What will the controllers of the future look like for VR?

Who I am
Tom Chatfield


  • Hand-Tracking: the VR controller of the future will be your hand
  • Full body-tracking: it will soon be possible to interact in VR with all parts of the body
  • VIVE Tracker: Any object can act as a VR controller
  • Brain-machine interface: the human brain as a VR controller

Right now, the VIVE Wand and the Oculus Touch are hands down the most advanced VR controllers. Thanks to motion tracking technologies, these accessories are detected by motion sensors and therefore allow users' hand movements to be taken into account within virtual reality. Until now, such controllers have been the preserve of high-end PC VR headsets. As of this week, however, a year after the release of the Rift and the VIVE, the Samsung Gear VR mobile headset also benefits from a wireless controller with motion detection. This is proof that VR technologies are advancing at breakneck speed.

So within a few years, if not just a few months, it is highly likely that an even more impressive VR controller will be released. In reality, many projects are under development, and some are already at the functional prototype stage. Here are some examples of what the VR controller of tomorrow could look like.

Hand-Tracking: the VR controller of the future will be your hand

The VIVE Wand, and the Oculus Touch even better, manage to simulate the presence of hands in virtual reality. The movements of the controllers are transcribed in real time in the VR, and the Touch goes so far as to take into account the position of the fingers on the keys. It is thus possible to close the fist, open the palm or point the finger to interact in a natural way within the VR.

However, the next step will be to be able to use your hands directly in the VR, without having to resort to an intermediate accessory. Several companies are already working in this direction. The best known example is of course the Leap Motion, already commercially available. Similar to Microsoft Kinect in its operation, this universal VR controller is able to detect the position of hands and fingers in three dimensions by infrared cameras. The hands and their movements are then reproduced in virtual reality. The system is surprisingly precise, but unfortunately is not yet foolproof.

Since 2014, a group of Microsoft researchers is also developing a hand-tracking technology called Handpose. This technology aims not only to allow interaction in VR and AR, but also with computers and other connected devices. A Kinect sensor captures depth information, which is then transformed into data points by Handpose's software to model hands in VR. The movements are interpreted using Machine Learning. Again, this system is not yet fully functional, but is already proving convincing and intuitive. It is not yet known when this system will be marketed.

Apart from these hand-tracking systems, there are also several models of VR gloves: Gloveone, Manus VR, Noitom Hi5, Virtalis ... these gloves make it possible to detect the hands with precision to model them in the VR and to transcribe all their movements with precision. Some even allow you to reproduce the sensations of touch and the different textures thanks to electrodes. Last February, Mark Zuckerberg himself presented a prototype VR gloves for the Oculus Rift.

Well-known props maker Razer, specializing in control peripherals for virtual reality, is also looking to develop an industry-standard VR controller. In the eyes of Chris Mitchell, Product Manager of Razer OSVR, the success or failure of virtual reality will depend on the methods of interaction that will be developed. It is therefore necessary to create a standard VR controller, so that game developers can create their games knowing that all users will have the same equipment. In addition, Mitchell believes that we should not be limited to hand-tracking, but also develop systems to track the movements of other parts of the body.

Full body-tracking: it will soon be possible to interact in VR with all parts of the body

Hand-tracking greatly increases the feeling of immersion in virtual reality, but for the illusion to be complete, it is essential that all parts of the body can be detected and allow interaction in VR. However, such a system is very difficult to set up and should not see the light of day for a few years.

Le developer Cloudgate Studio, however, managed to create an outline of "full-body tracking" by attaching two VIVE Wand controllers to the feet of a user. It is thus possible to move your feet in the RV, and to kick. On the other hand, this device does not yet make it possible to move around by walking in a natural way.

More recently, last March, HTC has provided developers with reference source code to materialize the whole body in VR. By attaching a VIVE Tracker to each foot, and another to the belt, it is possible to detect all parts of the body and their movements. By the end of the year, video games using this system should appear on Steam.

The developers of the game Tornuffalo recently presented a demonstration of their game, adapted for full-body tracking. Thanks to this advanced motion detection system, the player can dodge projectiles with precision, and even jump to rise into the air. It will therefore very soon be possible to use all parts of the body as a VR controller.

To really be able to move naturally (by walking) in virtual reality, it will probably take a few more years. Several concerns arise with the development of such a system. First of all, rapid trips in the RV can easily cause nausea. It is the reason why most developers offer a teleportation-based locomotion system in their video games.

Moreover, in order to move in the real world and in VR simultaneously, it is necessary to have a large play area. Several solutions already exist to remedy this second problem. We can cite VR treadmills, like the Virtuix Omni, a mat that allows you to move around at will while staying in the same place. Other solutions include the RotoVR chair, or the 3DRudder. This movement controller is used with the feet, while remaining seated, and allows you to move freely without getting up from your chair.

VIVE Tracker: Any object can act as a VR controller

The VIVE Tracker truly has the potential to revolutionize VR. This small sensor does not only allow full body-tracking, as mentioned above. It also allows to use any real world object as a VR controller. By attaching the accessory to an object, it is possible to directly materialize the object in virtual reality and use it.

The possibilities are therefore limitless. It is possible to take a saber, a gun, a baseball bat, a knife or any other object to use it as a VR controller. Some have even managed to model a cat in VR by hanging a VIVE Tracker on its back, but there is no question of using it as a control device.

However, to use an item as a controller, a game must first be developed. For example, it is quite possible to use a fishing rod in the RV, but first a fishing game designed for this purpose must be created. Thus, a first wave of trackers was sent to developers to allow them to bring their ideas to life. The first games based on this accessory should see the light of day at the end of the year.

Brain-machine interface: the human brain as a VR controller

As part of the Facebook F8 conference, Mark Zuckerberg recently unveiled his brain-machine interface project. This revolutionary technology would allow scan brain activity hundreds of times per second and interpret thoughts as commands for machines. At first, this interface should be used for writing. The user will be able to write what he thinks without having to use a keyboard.

In the long term, Facebook also plans to use this interface as a VR controller. The user could directly interact in virtual reality by thinking. There would therefore no longer be any need to model the different parts of the body in VR, or to find a suitable means of locomotion: the player would enjoy total freedom of movement and interaction. It would suffice to think of moving forward to move forward, to think of taking off to rise into the air at high speed ... which is to say that such technology will bring us closer and closer to the virtual reality depicted in the Matrix movie. This interface is both fascinating and terrifying.

add a comment of VR Controller - What will the controllers of the future look like for VR?
Comment sent successfully! We will review it in the next few hours.