Neurological research in human senses and, the understanding of how a human brain uses a collaboration of senses to create a perception of the environment around it has also suggested that the qualities associated with one sense may be perceived in a different way by another sense.
Flying Object Ltd won the 2015 Tate IK Prize for their installation concept, The Tate Sensorium. The Sensorium is designed to stimulate non visual senses to combine with the viewing of 4 individual pieces of Artwork. The trailer video can be viewed below.
Technology used in the installation seems to be that of Binaural audio, Touchless haptics and location specific smells.
Great examples of Binaural audio and touchless-haptics using ultrasound can be seen below respectively.
So, firstly i’ll discus the idea that specifically binaural sound could be used in games. I think we need to consider why binaural sound feels so immersive:
The way in which the microphone captures the sound. Using a stereo pair of microphones embedded in a head simulation (in particular the simulated construction of the outer ear) the sound is captured. Whilst gaming, sounds are generated, not captured. By this I mean sounds are triggered when a subject interacts with an environment. So all the nuances of acoustic and psycoacoustical phenomena have to be programmed. Therefor, you are left with a replication of an environment arguably not as ‘true’ as that of a recorded sound. All the sonic modelling done by the fake ‘pinna’ to create acoustic shadowing relative to the sound receptor (Mic/ear) is relatively straight forward in comparison. A further consideration is that of sound source. Binaural sound needs stereo headphones or a decent pair of stereo speakers located a discernibly wide distance apart relative to the subject. Typically, TV speakers are arranged in a narrow placement. Further, ‘Sound bars’ utilise arguably, counter effective output systems to the reproduction of binaural audio.
So binaural reproduction does not seem to be currently, a viable option. The processing involved in modelling an environment relative to the subjects movements then, maintaining the correct listening environment seems almost wasteful in processing, memory and storage.
Onto the idea of brining interactive ultrasound into gaming. Research, led by Dr Ben Long and colleagues Professor Sriram Subramanian, Sue Ann Seah and Tom Carter from the University of Bristol’s Department of Computer Science, showed the way 3 Dimensional shapes can be used. Ultrasound could enable surgeons to explore an object by using haptic feedback.
The method uses ultrasound, which is focussed onto hands above the device and that can be felt. By focussing complex patterns of ultrasound, the air disturbances can be seen as floating 3D shapes. Visually, the researchers demonstrated the ultrasound patterns by directing the device at a thin layer of oil so that the depressions in the surface can be seen as spots when lit by a lamp.
The system generates a virtual 3D shape that can be added to 3D displays to create something that can be seen and felt. The research team have also shown that users can match a picture of a 3D shape to the shape created by the system.
The direct correlation of the technology to gaming seems to carry a more practicable implementation. Now, obviously the technology in nowhere near a commercial scale. But, we would be daft to assume this will be the case forever. It could arguably open the door to a genuinely new genre of game. That of object exploration. Which, personally would be an exciting thing. An extrapolation of the haptic ultra sound technology for use in gaming is that of being able to create complimentary artefacts to that of the traditional on screen imagery and sound.
Summing up. New technology, like human senses, are now more than ever a seemingly collaborative thing. Sound, touch, vision, smell. There is all the technology there to implement these sensory interactive elements into games. The Billion dollar question is how to implement it in a way that is cost-effective and neither just a gimmick or just plain frustrating/distracting. I can’t answer that one.
This is why I don’t have a lovely super-car outside and a house with ‘wings’!