How does virtual reality work? (step by step explanation)

Summary

• The importance of immersion
• How the virtual environment works
• Interaction at the service of the user
• Video games: a perfect teacher
• Let's talk about hardware
• The challenges of VR

To start this article, nothing better than to start with a definition of the concept behind the technology. The principle is simple, a computer is used to produce a three-dimensional simulation of a world that the user can browse and manipulate and which will give him the feeling to be immersed in this world. Scientists, engineers and theorists have created hundreds of applications and techniques to make virtual reality work today.

If we had to name a few fundamental elements that define a virtual reality experience, we would not fail to mention the following two:

• Three-dimensional images that are thought through from the user's perspective.
• The ability to follow certain movements of the user. In particular his head and eyes, and make the 3D environment adapt to his perspective and movements.

The importance of immersion

There is no virtual reality without this feeling of immersion so present when you equip a virtual reality headset. Engineer Jonathan Steuer evokes the presence of two components that serve this immersion: “And respectively the depth and breadth of the information. The first, depth, refers to the quantity and quality of data the user receives when it interacts with its virtual environment. This can range from the resolution to the complexity of the graphics environment to the sophistication of the audio system. The extent of the information refers to to the number of senses present in the virtual universe and which act jointly. Most VR experiences use visual and audio sensors that work in tandem. For example, visual information such as the presence of a character on the screen who is running calls upon the visual sensors, while the running noise which is heard in parallel calls for the audio sensor. The operation of the two devices together is necessary to give scale to the scene.

Current research by engineers and scientists tends to show that the next step would be the incorporation of touch as a third sensory device. Device that would transcribe the user's pressure and the general feeling of touch (more information on some devices of this type here).

Another important criterion for the operation of the immersion is the response time. System stability relies on the illusion that what is seen is real and any change of angle on the part of the user must be immediately reflected on the two screens placed over the eyes. According to Dr. Frederick Brooks, a pioneer in virtual reality, the device must have a minimum of 20 to 30 images per second to create the necessary illusion. But be careful because if this is the minimum, we are closer to video game standards which tend to show that if 30 FPS is the basis for correct playability, 60 FPS represents the “real” fluidity to achieve. For virtual reality, this seems even more true because many people believe that 90 to 120 FPS is not too much to obtain a sufficient response rate to the user's head movements.

How the virtual environment works

The third criterion essential to the functioning of the immersion that we have just seen is also the entry point of this part devoted to the credibility of the virtual environment. We have just mentioned the number of senses present in a virtual environment (sight, hearing, touch…), but we have not mentioned the number of uses of these senses within the experience itself. In other words, microphone management of the senses in real time. For example, if the environment has a set of elements with 3D sound feedback, these different elements should react according to the position and orientation of the player. The most distant elements will thus be less perceptible than those which are a few centimeters from the user and will resonate more in the right or left ear depending on the orientation of the latter.

Of course, the speed with which these events will appear will create a credible illusion. We then speak of latency, either the response time between the system and the user's action. To better understand how latency works, let's make a parallel with video games. In creating a game, one of the most important things are signs and feedbacks. The signs represent everything a bunch of visual, sound or haptic elements (linked to touch, often the vibrations of a controller), which are used to indicate information to the player, to guide him in short. For example his life bar gives him information on the state of his character, the way an enemy moves can give him information on how to fight him, etc ... The signs are legion and essential to the functioning of a game. It's the same in real life, when you see a red light you know you have to stop, that's a sign.

Feedbacks correspond to feedback following a user action. For example, if the player decides to jump, he will press a key, at the very moment he performs this action, a return must be made: Launch of the jump animation, sound effects of the character, decrease in the bar endurance, etc ... So much feedback that allows the user to understand that his action has worked well. Each action, regardless of its importance must be associated several feedbacks from different categories. When you tap on your cell phone, when you withdraw money from the ATM, many feedbacks are present to make you understand that you have just pressed a key, that you have inserted your credit card, etc ... a simple and implicit operation which is necessary for Man.

In a virtual environment, it is the same. The signs must be permanently present, visual, sound, etc. It is their concordance that will make the player understand what is happening in front of him. If he walks, jumps, moves, turns his head, the feedback must be immediate. Hence the importance of having a very low latency so that this return is immediate. Humans can perceive a latency greater than 50 milliseconds. If this time is exceeded, the immersion is broken and the artificial environment is destroyed. The brain then does not understand why after moving the head nothing is immediately produced.

A true virtual reality experience makes the user forget that they are in a fictitious world.

Interaction at the service of the user

There are several types of virtual reality content, some offer more possibilities to the user than others, between playing a video game in VR and watching a movie in VR as well, there is a chasm in terms of interactivity. In the first case, the user is a player capable of interacting with the environment and of seeing the consequences of these acts on it, in the second case, the user is only a spectator to whom 'we give the possibility to choose which part of the scene to watch, Although the two uses are not really comparable it is a certain fact: Video games will offer a higher degree of immersion, because if in the end the technical degree will be less impressive than a film capable of creating in a photo-realistic environment, the possibility of being active and offering a list of feasible actions will be more successful in anchoring the player in the virtual world they are traversing.

Jonathan Steuer explains that there are three factors that help explain the degree of interaction offered by an application. The first is speed, it defines the speed at which the player's actions are incorporated by the software and transcribed on the screen in a way that the user can perceive (cf: feedbacks). The second is the door, it defines the number of consequences that can result from an action. The last one is mapping, it corresponds to the software's ability to define natural results in response to user action.

One of the most common interactions in virtual reality applications and particularly in games, results in the ability to move. The operation of navigation is an integral part of the possibilities for interaction and is often the intermediary to perform other actions. We also understand certain derivatives in the operation of movement such as jumping, climbing and running which also contribute to this degree of immersion while offering a larger range of movements. which gives the user the feeling of having greater control over the environment. The immersion is only reinforced.

Unfortunately the displacement is not enough to create sufficient interaction because it is only there to allow the user to visit the virtual environment, it has no real impact on his world. Scientist Mary Whitton explains that after just a few minutes spent in an environment that offers no more interactivity than that, the user will be too bored to be successful in investing and the environment will immediately appear as dummy, breaking user engagement.

So how do you make the user feel invested? He must be able to modify the virtual environment. The use of objectives and interactive objects can serve the user and make his trip captivating.

Video games: a perfect teacher

If there is one medium to take as a model, it is video games. Since it exists he never stopped pushing the limits of interactivity, to attempt, to fail, to undertake, to learn and to draw lessons on the functioning of the interaction between the player and the machine. It is a young medium but which evolves extremely quickly, there are today many specialists and all these people today know a lot of things about what to do and what not to do in a video game. Bases in short, like cinema, literature or music have theirs. Codes, imperishable rules. But what makes video games so special, is that it is the only medium in the world that does not make the user passive. We are active when we play, to a varying degree, and it is all this that makes virtual reality necessary to take a model from this medium in order to succeed in creating its own rules in the future. With video gaming being a part of the VR landscape, that doesn't mean that other apps aren't trying to create that interactivity, take advantage of the possibilities of virtual reality, and make sure that whatever the type of application, the user is no longer passive.

Let's talk about hardware

Now that we are done with the immersion techniques, let's take a look at the hardware behind the applications. Today most virtual reality headsets run on personal computers. The power of PCs today makes it easy to create and run applications. Be careful though, this still requires advanced components to enable efficient use and in good conditions for the most demanding applications. Headsets like the Oculus Rift or the HTC Vive require desktop PCs with high quality components. We will then turn to gaming components with latest generation graphics cards such as the GTX 900 series and with the latest, the GTX 1080. A rather substantial budget, therefore, reserved for lovers of high quality experiences.

Most helmets use an HMD system based on two screens, one for each eye, in order to reflect the depth of field in the user. The first HMD systems used “Cathode Ray Tube” monitors (CRT). Today it is the “Liquid Crystal Display” monitors (LCD) which dominate the market and which offer a resolution and a more interesting color panel.

Next to that, comes the motion tracking system which allows users to perform movements with their heads which are transcribed on the screen. The system detects the user's orientation and sends the corresponding image to the screens. 

Finally comes the accessories that can be associated with the use of helmets. There are controllers, often equipped with motion detection, gloves, voice recognition and environmental sensors that identify elements in the room (like the HTC Vive) and many others .

The challenges of VR

We are nearing the end of this dossier devoted to the functioning of virtual reality, but there is still one last point to address, the future challenges of VR:

• Succeed in creating systems that are more implicit for the user, more aforders and which therefore require less learning to open it up more to the general public.
• Work on the ergonomics of the accessories to offer a practical use in combination with the helmet.
• Create virtual environments that only make sense in virtual reality and that can therefore be used to promote technology
• Work on optimization and technology to make it more accessible so that it is not just reserved for an elite.

The file ends here, if nevertheless you are not yet satisfied with knowledge you can go further with this other article explaining the creation of applications dedicated to virtual reality.