Video game technology. Part 3 of 3: Immersion through sound.

Can audio really make movies and games more immersive and fun?

What is the best video game you have ever played? Don’t answer yet. What the first thing that came to your mind when you read that question? Was it a specific level in a first person shooter? A character in a role playing game? In any case you first thought was probably associated with visual images. A gun fight from a first person perspective, a character jumping between platforms or a flash of light after an explosion. Every time we think about video games we tend to think in terms of visuals. Nevertheless, sound also plays an important role in our gaming experiences. Now, you are probably thinking about the rewarding sound you heard after picking up an item or finishing an enemy. Maybe you are just remembering how emotionally moving was the soundtrack of the last game you played. And while sounds and music are not the first thing we associated with movies or video games, it is still a very important part of the experience. They can tell if there are enemies around us and even communicate the emotional tone of a scene. But can sound really affect how immersed we feel or how much we enjoy a movie or video game? Is it really that important?

Silent hill reduced visibility and forced the player to rely on sound effects.

Sounds are an essential feature of our environment. They gives us information about the objects and events around us. They can even tell us about what is happening out of our field of view. It’s no surprise then that the presence of sound has been shown to increase the feeling of being inside an interactive virtual environment (presence) (Larsson, Västfjäll, Olsson & Kleiner, 2007). It has also been associated with higher levels of involvement towards the sensory stimulus, story and characters (immersion), the positive feeling of focusing on a cognitively demanding task (flow) and reduced tension while playing a first person shooter (Nacke, Grimshaw & Lindley, 2010). In this last experiment, however, sound didn’t provoke any significant changes in the physiological indicators of emotional valence (facial muscles activity) and arousal (electro-dermal activity) (Nacke, et al. 2010). This means that either the change in emotions wasn’t strong enough to affect the physiology of players, or that the presence of sound increased their expectations, provoking them to overrate how immersed and less tense they felt. This makes the results of the second study somehow inconclusive, but there’s still an important amount of evidence of how audio can affect the experience of media.

Audio can also provide a considerable amount of information about the things and events that provoked them. The sound an object makes when falling to the ground can give us clues about what it’s made of, how heavy it is or how fast it was going. Placing speakers in different positions around the user allows him or her to use the sound as a source of spatial information. For example: If a sound comes from a speaker placed to the left of the user, he or she will feel like it’s coming from this side of the virtual environment. Sounds can also provide information about the position of its source and the acoustics of the environment, something that tends to make them easier to identify as well as feel more realistic (Larsson, Västfjäll, Olsson & Kleiner, 2007). This has also been shown to increase the feeling of being inside the content (Larsson, et al. 2007). After all, people tend to use sounds to discover the position of objects as well as the shape of their surroundings in real life.

Outdoors and enclosed spaces sound different, even in video games.

If increasing the number of audio channels can help the user better identify the position of sound sources, then audio systems that use various speakers in different positions should be able to provoke a higher sense of presence and immersion, right? According to research, using five audio channels instead of two didn’t increase the feeling of being inside the content or how realistic the events felt in the case of a movie (Freeman & Lessiter, 2001), but it did when the media was a first person shooter (Skalski, Whitbred & Lindmark, 2009). The additional number of channels also made both, the experience of the movie and the video game, more enjoyable (Freeman & Lessiter, 2001 and Skalski, et al. 2009, respectively). This suggest that additional audio channels only increase presence and realism when the user is able to interact with the content, but that this isn’t necessary in the case of enjoyment. Maybe the additional audio channels are able to make the movie experience more interesting. But in video games, where the user’s actions can affect what happens next, the information coming from the additional audio channels becomes more important. The sounds now reveal the position of potential threats and rewards, increasing levels of attention towards this type of stimuli. In this circumstances the player is able to better appreciate the details and experience more intense emotions, perceiving the content as more realistic. Also, the additional levels of engagement will eventually make him or her forget about the technology providing the experience, producing a feeling of being inside the content.

Sound direction is more relevant in games, where it can reveal the position of an enemy.

Adding bass to the audio channels while watching a movie seems to increase how clear and exciting the audio seems, but also how uncomfortably loud it may feel (Freeman & Lessiter, 2001). This might have happened because people tend to associated strong vibrations with loud noises, like a sound system turned up to its maximum volume. Interestingly, the additional bass channel also made the visual stimuli more exciting (Freeman & Lessiter, 2001). This could be attributed to a halo effect, where the additional audio quality made the subjects overstate how well other stimuli felt.

The bass channel was also found to increase the feeling of being inside the movie as well as involvement and the believability of the events depicted on it (Freeman & Lessiter, 2001). It’s possible that the viewers not only heard the bass sounds but also felt the strong vibrations associated with them. The additional tactile information could have helped the viewers experience how it would have felt to be a part of the portrayed events, increasing both presence and perceived realism.

Strong bass sounds can make the viewer feel like the floor is moving,
increasing realism and presence.

Regardless of the apparently obvious benefits of using a speaker-based surround system, some people still prefer to wear headphones. One reason to choose them could be that they help isolate the user from external noise. This would make it easier for him or her to focus on the content, increasing engagement and maybe even involvement. However, research on the effects of using headphones or speakers in the experience of a movie is still inconclusive; with one study showing an increase in engagement and involvement when using headphones (Campanella, Pettey, Guha, & Rubenking, 2010) and another one finding no significant differences between each condition (Pettey, Campanella, Rubenking, Buncher & Gress, 2010). Nevertheless, both studies found that using headphones or speakers didn’t produce any significant differences in the feeling of being inside the content or how realistic the depicted events seemed (Campanella, et al. 2010; Pettey, et al. 2010). So maybe, in the end, it’s all about comfort.

What is better for playing video games then: Headphones or speakers? The previous study indicates that there seems to be no significant differences in the case of movies. Still, we should remember that, as previous research has demonstrated, the fact that an audio variable doesn’t affect the experience of watching a movie doesn’t necessarily means that the same will happen with video games (e. g. Freeman & Lessiter, 2001 and Skalski, Whitbread & Lindmark 2009). Maybe the relationship between the isolation provided by headphones and immersion becomes stronger when the user is able to interact with the media. Still, additional studies are necessary to clarify this last question.

While improvements in the field of images and animation has been more obvious, significant advances have happened too in respect to audio technology. Built-in speakers are still available, but users now have the option to invest several hundreds of dollars in complex surround systems and highly advanced headphones. However, additional studies are required before we can assure consumers that a specific piece of hardware will make them feel more immersed, at least in the case of video games. In the meantime just ask yourself this question: Does a device helps you focus on the game? Does it make you more involved in the story and characters? Did you found yourself suddenly feeling like if you were inside the virtual environment? If the answer is yes, then you probably made the right choice.

References:

Campanella, C., Pettey, G., Guha, T., & Rubenking, B. E. (2010). Sound out small screens and telepresence. The impact of audio, screen size and pace. Journal of Media Psychology, 22(3), 125-137. doi: 10.1027/1864-1105/a000017

Freeman, J. & Lessiter, J. (2001, August). Here, there and everywhere: The effects of multichannel audio on presence. Proceedings of the 1001 International Conference on Auditory Display, Espoo, Finland.

Larsson, P., Västfjäll, D. Olsson, P. & Kleiner, M. (2007, October). When what you hear is what you see: Presence and auditory-visual integration in virtual environments. Proceedings of the 10th Annual International Workshop on Presence, Barcelona, Spain.

Nacke, L. E., Grimshaw, M. N. & Lindley, C. A. (2010). More than a feeling: Measurement of sonic user experience and psychophysiology in a first person shooter game. Interacting with Computers, 22(5), 336-343. doi: 10.1016/j.intcom.2010.04.005

Pettey, G., Campanella, C., Rubenking, B., Buncher, M., & Gress, E. (2010). Telepresence, soundscapes and technological expectation: Putting the observer into the equation. Virtual Reality, 14(1), 15-24. doi: 10.1007/s10055-009-0148-8

Skalski, P., Whitbred, R., & Lindmark, P. (2009, November). Image vs. sound: a comparison of formal feature effects on presence, video game enjoyment, and player performance. Paper presented at the 12th annual international workshop on presence, Los Angeles, CA.

Video game technology. Part 2 of 3: The HD Experience.

 
claimtoken-534595084c47d
One of the most recent advances in entertainment technology has been the switch from standard resolution to High Definition. The change has affected not only displays but also the complexity of the media produced. Most TV shows, movies and video games are now made in this format. The capacity to offer higher resolution images even became a powerful weapon in the last version of the console wars. The technology has been so well received by the public that Ultra HD displays are already being sold and media producers are spending time and money to make more content available in this resolution. But putting the public demands aside, is High Definition really capable of improving the user experience? Does it have more complex effects than just making things looks prettier?

First let’s analyze what High Definition is and what it can (probably) do. Image resolution indicates the number of pixels (or squares) available to build an image. The higher the resolution, the more details an image can have. This could make the content easier to identify and more interesting, increasing the attention of the user. The type of attribute being noticed could also affect different aspects of the experience. More defined borders, better lighting and smoother movements, for example, could make the objects, characters and environments look more realistic. On the other hand, if the content affected is emotionally relevant (like the facial expressions of a character in a movie or the position of the enemies in a first person shooter), it could make the user more involved.

The reason we focus on this variables is because they seem to be strongly related with spatial presence (the feeling of being inside the content) and immersion. A higher level of attention towards the content usually implies a reduction in the importance given to external stimuli. This can gradually make the user less aware of the technology providing the experience and, eventually, make him or her feel like they are part of the content. Additionally, engagement, perceived realism and involvement are frequently included in questionnaires designed to measure immersion. Also, both spatial presence and immersion play a central role in the studies about High Definition we were able to find.

"Papers, Please" was able to provoke strong emotional responses using only a couple of pixels.

Let’s begin with two experiments about the effects of resolution on viewer’s experience. In 2005, a group of participants watching a short documentary in HD (1080 lines of resolution) perceived the sensorial stimuli as more believable and identified non-verbal behavior more easily than those who saw it on NTSC (with only 480 lines) (Bracken, 2005). The next year researchers asked the same thing to another group of individuals, but they showed a newscast instead of a documentary. Interestingly, this time HD didn’t make the content look more realistic (Bracken, 2006). However, it’s effect on perception of non-verbal behavior remained (Bracken, 2006). One explanation could be that documentaries tend to show slow paced scenes and smooth transitions, leaving the viewer enough time to appreciate the details designed to increase realism. Newscast, on the other hand, tend to use shorter clips and fast paced transitions with the intention to capture the viewer’s attention as soon as possible, reducing their chances to notice some of this features. High Definition also made the news anchors look more dynamic, qualified and trustworthy (Bracken, 2006). As we previously stated, it’s possible that improved image quality made specific details more noticeable for the audience. In this case, relatively stable attributes intended to increase the anchor’s believability (like their clothing, facial expressions and hand gestures). In any case, results suggest that increased image resolution can make details intended to promote realism and communicate non-verbal behavior easier to identify.

Participants who saw the documentary on HD showed higher levels of self-reported spatial presence (Bracken, 2005). This is consistent with the idea that improved image quality can increase the user’s levels of attention, which eventually can lead him or her to forget about the mediating technology. However, watching the documentary in HD didn’t make the viewers feel more immersed (Bracken, 2005). In order to explain this we should point out that the mentioned study measured immersion by asking participants how engaged and involved they felt, as well as how realistic the content seemed to them. As we have seen, image quality was able to increase both realism and (probably) engagement. However, involvement is a different issue. For a user to feel involved, he or she must find the content important. One way to achieve this is through emotionally relevant stimuli, like a sad story on the news, seeing a character we care about suffer in a movie or playing the last level of video game with only one life left. While increased image quality could have made this type of content more noticeable, chances are it wasn’t present in the video used by the experimenters: A short, dialogue-free documentary showing scenes of everyday life in japan. So, while the enhanced images could have increased both engagement and realism, there was no reason for involvement to do so; reducing the probability of self-reported immersion to experience a significant change. Nevertheless, when the visual stimuli was a newscast (a type of content that tends to be designed to provoke an emotional response), watching it on HD did manage to increase the level of immersion experienced by the viewers (Bracken, 2006). So, maybe high definition can make emotionally relevant content (if available) easier to identify, increasing involvement and, therefore, immersion.

Is high definition making emotional content like facial expressions easier to identify?

The previous studies suggest that increasing resolution in a video can provoke a significant effect on the viewer’s experience. But what would happen in the case of video games, were the interaction between player and virtual environment opens the possibility for much stronger psychological effects?

To answer this question Skalski, Whitbred and Lindmark (2009) asked a group of undergraduate students to play Tom Clancy’s Ghost Recon: Advanced Warfighter for the Xbox 360 on HD (1080 lines) or standard definition (480 lines). Results showed that improved resolution didn’t make sensorial stimuli feel more realistic or in-game events seem more possible to occur in real life (Skalski, et al. 2009). It didn’t increased enjoyment either (Skalski, et al.). One explanation could be that increasing resolution wasn’t enough to make the features designed to increase realism (like particles flying around or precise animations showing enemies falling to the ground) more noticeable. Additionally, it’s possible that perceived realism plays a more important role in the experience of games that try to depict real-life events (in this case, a soldier in the battlefield). So, maybe the inability for resolution to increase realism was in part what prevented enjoyment to improve.

Participants playing Tom Clancy’s Ghost Recon: Advanced Warfighter for the Xbox 360 on HD didn’t show higher self-reported engagement or spatial presence than those who played it on NTSC (Skalski, Whitbred and Lindmark, 2009). This suggest that the increase in resolution didn’t make the content significantly more interesting. The result is also congruent with the idea that increased levels of attention are required for the experience of non-mediation (forgetting that what we are perceiving is coming from technology and not reality) to occur. In a similar experiment, subjects playing Perfect Dark Zero for the Xbox on HD didn’t experienced more spatial presence either (Bracken & Skalski, 2009). Evidence from both studies seem to indicate that the relationship between image resolution and the feeling of being inside a first person shooter is surprisingly weak. Still, experiments with other types of games could show different results.

HD can surely increase the sense of presence in first person shooters, right? Think again.

The study using Perfect Dark Zero had the advantage of measuring self-reported immersion, finding higher levels for the group that played the game in HD (Bracken & Skalski, 2009). Maybe increased resolution made easier for the HD group to identify in-game objects like pick-ups or enemies. This would have made the players more worried about both rewards and mistakes, increasing their involvement and, eventually, making them feel more immersed.

Resolution has been an important aspect of technological purchases for fans of TV shows, movies and video games. They want to see their media with the best image quality available. And while most of them are convinced that their experience as users has improved, it’s most likely that the effects of image resolution go far beyond just making things looks pretty. They involve complex and strong psychological effects. Still, additional studies are necessary to better understand this relationship. And while there aren’t much papers on the effects of High Definition, the development of new visual technologies like Ultra High Definition and Virtual Reality will probably recapture the attention of scientists interested in how to improve the user experience.

Thanks for reading! Visit us in two weeks for the final part of our series on Video Game Technology, when we analyze the effects of sounds and music in the player’s experience.

References:

Bracken, C. C. (2005). Presence and image quality: the case of high-definition television. Media Psychology, 7(2), 191-205. doi: 10.1207/S1532785XMEP0702_4

Bracken, C. C. (2006). Perceived source credibility of local television news: the impact of television form and presence. Journal of Broadcasting & Electronic Media, 50(4), 723-741. doi: 10.1207/s15506878jobem5004_9

Bracken, C. C., & Skalski, P. (2009). Telepresence and video games: the impact of image quality. PsychNology Journal, 7(1), 101-112. Retrieved from: https://www.researchgate.net/profile/Cheryl_Bracken/publication/220168895_Telepresence_and_Video_Games_The_Impact_of_Image_Quality/file/3deec52e14793e7ab5.pdf

Skalski, P., Whitbred, R., & Lindmark, P. (2009, November). Image vs. sound: a comparison of formal feature effects on presence, video game enjoyment, and player performance. Paper presented at the 12th annual international workshop on presence, Los Angeles, CA.