Notes on Blindness Dev

Dev Diary of the VR experience

[notesonblindnessdev-blog]

5. Building a scene in Notes On Blindness VR

Hello fellows,

After talking about visuals and sounds, let’s see how we merge these when building a scene.

The experience of Notes On Blindness is composed of several scenes. Each scene is composed by what John hears and feels and is guided by a general script. The scene is also characterised by one or several interactions. The interactions have been thought to be meaningful and to help people immerse more into the universe and enhance the emotions we want to share.

We start building a scene by defining the space, generally helped by what John described in his recordings. This helps us get the general feel for the scene, we start by placing some strong elements on our 3D space like the floor, some trees or some walls which don’t emit sounds especially but which are part of the consciousness of John. We then create/record the first sounds for the scene like for the multi-point ambiances (wind, rain) with trees creaking and foliage.

The second step is programming. With placeholders for some of the sounds and most of the graphics we code the whole structure of the scene following the script. It’s a gradual building. The scene, by the end of this step, will still be sparse but fully functional.

Then we give a sound and a visual shape made with particles or shaders to each objects. The visuals are linked to and triggered by the sounds they make as we explained in previous articles.

Finally, we add liveliness and randomization to some of the objects to make each experience a bit different. For example, the paths people take, the thunder rolling, people coughing, a dog barking… All of these objects, will have a “living” behavior mimicking their real-life behavior. It is essential to get a very realistic and immersive sound space all around you.

[notesonblindnessdev-blog]

4. Binaural audio synthesis in Notes On Blindness VR

Hello everyone,

Today we would like to explain to you the main sound features of Notes On Blindness : the Binaural synthesis.

In most of audiovisual productions, the soundtrack is mixed in stereo. The spatialization of an audio source is defined by its volume difference between right and left ears. This allows only an horizontal spatialization (left/right) of sound and the listener can’t define a precise location angle.

The emergence of VR (virtual reality) came with a surge of interest in an old technology : Binaural audio. Binaural audio offers a much better location rendering and adds verticality to the audio source. It allows a three dimensional spatialization. There is only one constraint, you need headphones (any kind) for it to work. Which is also why it’s great with VR. We could call this “virtual reality for ears”.

Binaural synthesis is the sound spatialization method which is the closest to natural hearing. It is based on the psychophysiological characteristics of human hearing to locate the sounds origin. We can notice amongst these main characteristics :

Head-related transfer function (HRTF) which are used to define the transformations made by the body of the listener on the soundwaves, mostly the head, the ear and the auditory conduct as they allow human beings to determine the horizontal (azimuth) and vertical (elevation) origin of a sound.

Interaural Time Difference is the difference in arrival time of a sound between two ears.

Interaural Level Difference is the volume difference of the soundwave between the two ears.

Binaural audio recordings is possible thanks to “artificial heads” which dimensions match the average human heads.

Thanks to new real time technologies, we can use those parameters on monophonic audio sources in a 3D scene to simulate a natural listening in real time. This is real-time binaural synthesis.

In Notes on Blindness, all the sounds you hear are coming from mono sources, spatialized in real time through binaural synthesis. In the first scene for example, near the ending you have more than 40 sounds playing at the same time. Because binaural spatialization in real time takes a lot of computing power, we’ve had to use different kind of binaural renderings to avoid hitting too hard on the devices’ CPU. Some close sounds are rendered with great precision including early reflections also in Binaural, whereas more distant sounds are going through simple HRTF approximation filters.

Each scene has been built with the same principle, where all the soundtrack is composed of individual sounds with their own properties, shaping the global soundscape of the experience.

[notesonblindnessdev-blog]

3. Every sound is a point of activity

Hi everyone,

During your experience, you will hear sounds spatialized in the 3D space. For John Hull, “every sound is a point of activity”. He can only see what he’s able to hear. So the sounds are visually represented by a shader that gives a “ghost effect” to the objects which emit it. A “light” is attached to those objects. The intensity of the light is linked to the volume of the sounds produced by the object. Thus, the sounds create a field of activity John’s brain can interpret and from which he can extract a mental representation of what’s happening around him.

As we mentionned in our last posts, this week we will reveal you some tricks about how visuals are technically created.

In a game most of what you see is at first what we call a “mesh”, these meshes can be simple shapes like a cube or more complex ones such as 3D models of human body or a house. This is what is attached to them that will dictate the way they actually look in the game. In Notes On Blindness we use two kind of tools for our meshes which are shaders and particles.

Shaders are code written in a specific language which are used to describe how the mesh will behave with light as well as more advanced behaviours. In a nutshell a shader will make your mesh look like wood or gold or, in Notes On Blindness, blue ghostly shapes.

Particles, on the other side, tend to represent what John is conscious about : the floor under his feet, the floor in general, the sky, or entities like pools, walls etc. They are usually used in video games to create effects that cannot be done with a mesh such as fire, smoke, a magic fireball or anything like that.

In Notes On Blindness, we mostly use them on meshes. It means that the particles are linked to a shape such as a house, a lake, a tree and are not used to create effects that can’t be done with a mesh but are rather used to create a bit of constant life on the objects since they allow us a lot of control for a low performance cost. They also fit more with the artistic direction of Notes On Blindness.

For example, we could have used a shader on our lake mesh to make it look like water. But it was not only more effective to use small slowly moving particles but also more relevant from an artistic point of vue since John Hull cannot clearly see water. However, because of water lapping sounds, ducks, etc. he can get a sense of the lake presence, and that’s what particles are showing.

[notesonblindnessdev-blog]

2. Work approach and vision for Notes on Blindness VR

Hello everyone!

Let’s talk about the initial intentions we had when we started to work on Notes on Blindness.

It quickly became obvious that the recordings of John Hull would be the main anchor of the whole project. His voice, his pace, and his thoughts are very moving and convey a lot of emotions. Going through all the transcriptions and recordings, we tried to isolate the passages that could be interesting in terms of interactive narration and Audio gameplay.

Moreover, the combination of the narration and the Audio-interactive experience was a very stimulating challenge. Indeed, Audiogaming has used all of its audio and technical skills to provide the best possible experience.

Therefore, at the very beginning, we thought of creating an audio-only experience. Neither visual was planned at this stage, nor VR. The purpose was to be fully immersed in John’s story through carefully crafted audio narration and interactions using real-time binaural rendering. The user would have been blind himself, guided by John’s original recordings and helped by audio technologies, which will be detailed in further posts.

After initial tests, we quickly realised that, for a story like Notes On Blindness, the experience would benefit a lot from adding visuals, but not regular 3D interactive visuals. We started to turn around this idea of a “perceptive world”, a “world beyond sight” that John is describing so well. And we wondered how we could create that world. So the first prototype (a proof of concept we should say) became composed of real-time interactive binaural sound scenes and visual placeholders trying to mimic John’s description.

Visuals suddenly brought more versatility to gameplay mechanics and a new dimension to immersion. But then we had to think of an art direction for visuals linked to sounds.

[notesonblindnessdev-blog]

1. Crafting Notes On Blindness VR

Hey there !

We are the Dev Team of the Notes on Blindness immersive experience, which will be available on Android, iOS and Oculus. The experience is mostly targeting the fresh VR community and accompanies the feature-film “Notes on Blindness”, which is currently in production. It is co-produced with AgatFilms-ExNihilo, Archer’s Mark and with the French/German TV channel ARTE.

Award winner of the StoryScapes of the Tribeca Film Festival and the Kaleidoscope Festival, but also part of the official selection of the Sundance New Frontier, we are also blessed to be nominated in many VR and film festivals.

We are located in France and we work on creating a new form of storytelling using interactive technologies based on the emotional story of John Hull. If you haven’t yet watched the “Notes on Blindness” short movie, we encourage you to do so. It is only 10 minutes long, beautifully realised and rich in meaning. This will give you a great overview of John’s story.

Video of the movie: https://vimeo.com/84336261

The immersive experience will be available in a couple of month.

So why are we started this blog now? Crafting this experience was a lot about experimenting new forms of narratives and technologies. It also involves a lot of sound design and sound technologies, which are things we’ve always been into. We thought it would be cool to share with you the challenges we faced, the experience and interactions we imagined and realised. It would be great also to get your thoughts and feedback.

In the next weeks we will be blogging about the creation of the experience and the story behind Notes on Blindness. You’ll get some gory nerd details about the technologies, the design, the crafting of what we hope to be a very new and immersive storytelling.

We are looking forward to your questions and comments and you can make suggestions for future topics. Just shoot them at us and we will respond as soon as possible!

You can leave us a comment or write to us at [email protected].