Video game technology. Part 3 of 3: Immersion through sound.

Can audio really make movies and games more immersive and fun?

What is the best video game you have ever played? Don’t answer yet. What the first thing that came to your mind when you read that question? Was it a specific level in a first person shooter? A character in a role playing game? In any case you first thought was probably associated with visual images. A gun fight from a first person perspective, a character jumping between platforms or a flash of light after an explosion. Every time we think about video games we tend to think in terms of visuals. Nevertheless, sound also plays an important role in our gaming experiences. Now, you are probably thinking about the rewarding sound you heard after picking up an item or finishing an enemy. Maybe you are just remembering how emotionally moving was the soundtrack of the last game you played. And while sounds and music are not the first thing we associated with movies or video games, it is still a very important part of the experience. They can tell if there are enemies around us and even communicate the emotional tone of a scene. But can sound really affect how immersed we feel or how much we enjoy a movie or video game? Is it really that important?

Silent hill reduced visibility and forced the player to rely on sound effects.

Sounds are an essential feature of our environment. They gives us information about the objects and events around us. They can even tell us about what is happening out of our field of view. It’s no surprise then that the presence of sound has been shown to increase the feeling of being inside an interactive virtual environment (presence) (Larsson, Västfjäll, Olsson & Kleiner, 2007). It has also been associated with higher levels of involvement towards the sensory stimulus, story and characters (immersion), the positive feeling of focusing on a cognitively demanding task (flow) and reduced tension while playing a first person shooter (Nacke, Grimshaw & Lindley, 2010). In this last experiment, however, sound didn’t provoke any significant changes in the physiological indicators of emotional valence (facial muscles activity) and arousal (electro-dermal activity) (Nacke, et al. 2010). This means that either the change in emotions wasn’t strong enough to affect the physiology of players, or that the presence of sound increased their expectations, provoking them to overrate how immersed and less tense they felt. This makes the results of the second study somehow inconclusive, but there’s still an important amount of evidence of how audio can affect the experience of media.

Audio can also provide a considerable amount of information about the things and events that provoked them. The sound an object makes when falling to the ground can give us clues about what it’s made of, how heavy it is or how fast it was going. Placing speakers in different positions around the user allows him or her to use the sound as a source of spatial information. For example: If a sound comes from a speaker placed to the left of the user, he or she will feel like it’s coming from this side of the virtual environment. Sounds can also provide information about the position of its source and the acoustics of the environment, something that tends to make them easier to identify as well as feel more realistic (Larsson, Västfjäll, Olsson & Kleiner, 2007). This has also been shown to increase the feeling of being inside the content (Larsson, et al. 2007). After all, people tend to use sounds to discover the position of objects as well as the shape of their surroundings in real life.

Outdoors and enclosed spaces sound different, even in video games.

If increasing the number of audio channels can help the user better identify the position of sound sources, then audio systems that use various speakers in different positions should be able to provoke a higher sense of presence and immersion, right? According to research, using five audio channels instead of two didn’t increase the feeling of being inside the content or how realistic the events felt in the case of a movie (Freeman & Lessiter, 2001), but it did when the media was a first person shooter (Skalski, Whitbred & Lindmark, 2009). The additional number of channels also made both, the experience of the movie and the video game, more enjoyable (Freeman & Lessiter, 2001 and Skalski, et al. 2009, respectively). This suggest that additional audio channels only increase presence and realism when the user is able to interact with the content, but that this isn’t necessary in the case of enjoyment. Maybe the additional audio channels are able to make the movie experience more interesting. But in video games, where the user’s actions can affect what happens next, the information coming from the additional audio channels becomes more important. The sounds now reveal the position of potential threats and rewards, increasing levels of attention towards this type of stimuli. In this circumstances the player is able to better appreciate the details and experience more intense emotions, perceiving the content as more realistic. Also, the additional levels of engagement will eventually make him or her forget about the technology providing the experience, producing a feeling of being inside the content.

Sound direction is more relevant in games, where it can reveal the position of an enemy.

Adding bass to the audio channels while watching a movie seems to increase how clear and exciting the audio seems, but also how uncomfortably loud it may feel (Freeman & Lessiter, 2001). This might have happened because people tend to associated strong vibrations with loud noises, like a sound system turned up to its maximum volume. Interestingly, the additional bass channel also made the visual stimuli more exciting (Freeman & Lessiter, 2001). This could be attributed to a halo effect, where the additional audio quality made the subjects overstate how well other stimuli felt.

The bass channel was also found to increase the feeling of being inside the movie as well as involvement and the believability of the events depicted on it (Freeman & Lessiter, 2001). It’s possible that the viewers not only heard the bass sounds but also felt the strong vibrations associated with them. The additional tactile information could have helped the viewers experience how it would have felt to be a part of the portrayed events, increasing both presence and perceived realism.

Strong bass sounds can make the viewer feel like the floor is moving,
increasing realism and presence.

Regardless of the apparently obvious benefits of using a speaker-based surround system, some people still prefer to wear headphones. One reason to choose them could be that they help isolate the user from external noise. This would make it easier for him or her to focus on the content, increasing engagement and maybe even involvement. However, research on the effects of using headphones or speakers in the experience of a movie is still inconclusive; with one study showing an increase in engagement and involvement when using headphones (Campanella, Pettey, Guha, & Rubenking, 2010) and another one finding no significant differences between each condition (Pettey, Campanella, Rubenking, Buncher & Gress, 2010). Nevertheless, both studies found that using headphones or speakers didn’t produce any significant differences in the feeling of being inside the content or how realistic the depicted events seemed (Campanella, et al. 2010; Pettey, et al. 2010). So maybe, in the end, it’s all about comfort.

What is better for playing video games then: Headphones or speakers? The previous study indicates that there seems to be no significant differences in the case of movies. Still, we should remember that, as previous research has demonstrated, the fact that an audio variable doesn’t affect the experience of watching a movie doesn’t necessarily means that the same will happen with video games (e. g. Freeman & Lessiter, 2001 and Skalski, Whitbread & Lindmark 2009). Maybe the relationship between the isolation provided by headphones and immersion becomes stronger when the user is able to interact with the media. Still, additional studies are necessary to clarify this last question.

While improvements in the field of images and animation has been more obvious, significant advances have happened too in respect to audio technology. Built-in speakers are still available, but users now have the option to invest several hundreds of dollars in complex surround systems and highly advanced headphones. However, additional studies are required before we can assure consumers that a specific piece of hardware will make them feel more immersed, at least in the case of video games. In the meantime just ask yourself this question: Does a device helps you focus on the game? Does it make you more involved in the story and characters? Did you found yourself suddenly feeling like if you were inside the virtual environment? If the answer is yes, then you probably made the right choice.

References:

Campanella, C., Pettey, G., Guha, T., & Rubenking, B. E. (2010). Sound out small screens and telepresence. The impact of audio, screen size and pace. Journal of Media Psychology, 22(3), 125-137. doi: 10.1027/1864-1105/a000017

Freeman, J. & Lessiter, J. (2001, August). Here, there and everywhere: The effects of multichannel audio on presence. Proceedings of the 1001 International Conference on Auditory Display, Espoo, Finland.

Larsson, P., Västfjäll, D. Olsson, P. & Kleiner, M. (2007, October). When what you hear is what you see: Presence and auditory-visual integration in virtual environments. Proceedings of the 10th Annual International Workshop on Presence, Barcelona, Spain.

Nacke, L. E., Grimshaw, M. N. & Lindley, C. A. (2010). More than a feeling: Measurement of sonic user experience and psychophysiology in a first person shooter game. Interacting with Computers, 22(5), 336-343. doi: 10.1016/j.intcom.2010.04.005

Pettey, G., Campanella, C., Rubenking, B., Buncher, M., & Gress, E. (2010). Telepresence, soundscapes and technological expectation: Putting the observer into the equation. Virtual Reality, 14(1), 15-24. doi: 10.1007/s10055-009-0148-8

Skalski, P., Whitbred, R., & Lindmark, P. (2009, November). Image vs. sound: a comparison of formal feature effects on presence, video game enjoyment, and player performance. Paper presented at the 12th annual international workshop on presence, Los Angeles, CA.