Rock bands and record players: The psychology of musical sources in Anime.

Does it matter if the background music is coming from a rock band or a record player?

Note: This article contains potential spoilers for K-On!, Tamako Market and Toradora.

It’s not uncommon for Anime series to use music as a way to enhance the mood of a scene. In an episode of Tamako Market, for example, the title character meets with a classmate in a coffee shop. Asagiri had to spend the last afternoon with Tamako and her family. Despite her social anxiety, she managed to have fun and wants to thank her. Asagiri tries to gather the strength to express this, and she eventually does. To her embarrassment though, her words come out in an awkwardly loud voice. Seeing her reaction, Tamako comforts her with a kind smile and a playful “I did too”. The scene is filled with cute facial expressions and gestures. It’s also accompanied by a slow but beautiful piano piece. The music feels sad enough to increase our sympathy for Asagiri, and appreciate Tamako’s kindness; making the interactions even more adorable. Musical attributes like the instrument being played or it’s tempo are frequently attributed with this type of effects. But there might be another factor in play.

Tamako and her friends visit this coffee shop frequently. During each of these scenes, a record player is shown either being set-up or playing tracks from different genres. Explaining where the background music is coming from might have the purpose of presenting the owner as a sophisticated and wise individual. Which makes sense given the role of passive counselor he plays in the series. But there could be another reason. Film theorists believe that, when music is unexplained, it appears as an obvious attempt to manipulate the audience. Placing its source within the character’s environment should reduce this perception, making it more effective. The idea is supported by an experiment in which a chase scene in a mall was perceived as more tense, when the music sounded like it came from the building speakers (Tan, Spackman, & Wakefield, 2008) (1). It’s possible then that showing the owner picking up an album and setting it up made the referred segments in Tamako Market more emotionally intense. Sadly, directors can’t put a record player or a loudspeaker on every single scene. There are, nevertheless, other, less mechanic alternatives.

Tamako Market's team went as far as composing a sixties-style song, hiring a french interpreter
and designing a fake album cover, just to give the right background to another, bitter-sweet, scene.

It’s not unusual for series to include musicians among their cast of characters; providing another method for inserting music into the story. Series about bands, like K-On!, have the advantage of being able to explain the presence of songs through rehearsal and concert scenes (2). The plot doesn’t necessarily have to revolve around music though. The fantasy drama Angel Beats!, for example, has a couple of characters capable of singing and playing instruments, ready to perform as soon as the plot allows it. Animated musicians don’t even have to be that skilled. In an episode of Tamako Market, the protagonist's father is revealed to have composed a song for his late wife, while they were both in high-school. His performance, shown through flashbacks, is clearly amateurish. But instead of disrupting the story, it ends up making it more believable and endearing.

Including an animated performance might have other benefits. People tend to rate acapella or instrumental compositions better when they are able to see the interpreter (Wapnick, Darrow, Kovacs, & Dalrymple, 1997; Wapnick, Mazza, & Darrow, 1998). And while, in the case of instrumental melodies, this doesn’t make people experience more positive, tense or exciting moods (Vines, Krumhansl, Wanderlev, Dalca, & Levitin, 2011; Vuoskoski, Gatti, Spence, & Clarke, 2016), it can make the audience more interested on the music, or even happier, depending on how expressive the performer is (Broughton & Stevens, 2009; Vines, Krumhansl, Wanderlev, Dalca, & Levitin, 2011). The results suggest that the presence of a performer adds very little to the experience of music, at least in the case of single-instrument pieces. Interestingly, they also imply that, when their appearance does make a difference, it has something to do with the way they move (3).

Unusually detailed or realistic movements during animated performances
might have an ulterior motive.

Musicians’ behavior tends to receive a special treatment by animators. Particularly in music driven stories. The openings for series like Sakamichi no Apollon, K-On! and even Angel Beats! depict musical performances with an unusual level of detail and realism. In the intro for Angel Beats!, for example, some notes are accompanied by shots of Kanade’s fingers pressing specific piano keys. This type of synchronization could be making the referred notes more noticeable, facilitating the identification of features like tempo and structure. Two variables that seem to play an important role in the induction of tension and excitement through music (van der Zwaag, Westerink, & van den Broek, 2011), as well as on its enjoyment (Rolison & Edworthy, 2013), respectively. The referred sequence also displays the performer’s hands, torso and head moving in the same manner as an experienced interpreter’s would. The depiction of both fine and gross musical gestures might permit a vicarious participation in the performance. In other words: Being able to see in great detail how an artist produces music, could be allowing viewers to feel like they are singing or playing an instrument, in the same way watching an avatar's hands turn a steering wheel make gamers feel like they are driving. There are plenty of reasons to believe that the additional effort put into these animations didn’t go to waste. But we should also remember that this particular technique hasn’t been properly tested. So it’s still possible that it has no measurable effects on the audience whatsoever.

Behavior isn’t the only attribute that could enhance the experience of music. Experiments have shown that sang or instrumental pieces are rated better when the performer is more attractive (Wapnick, Darrow, Kovacs, & Dalrymple, 1997; Wapnick, Mazza, & Darrow, 1998). A factor that can be particularly useful in a highly stylized medium like Anime. In Toradora, the female lead secretly rehearses a song for the school’s Christmas party. The dress and hairstyle she wears during her performance, as well as the delicate gestures she uses, contrast with the informal appearance and rough behavior displayed in previous episodes. The effort Tiga puts into her interpretation and the reaction this provokes in her friend Ryuji are probably intended to insinuate an increasing attraction between the characters. But her charming appearance could also have made the hopeful song seem more moving, and the scene more romantic. Putting the viewers in the right mood for the emotionally charged episodes that follow (4).

Character designs could become particularly influential during musical performances.

Placing the sources of music within the characters’ environment has, apparently, various benefits. It can make soundtracks more persuasive, while also enriching the experience with the behavior and aesthetics of performers. What makes this technique so interesting though is that it can take various forms. Music can make a gradual appearance as street performer is introduced into a shot. It can also stop just as particular character takes her headphones off. All in synchrony with the development of the plot and the emotions being displayed. Source or Diegetic music, as it’s known, doesn’t just enhances the effectiveness of a soundtrack. It adds a layer of meaning to the scene. Something that can make the whole experience more engaging, moving and memorable for the audience.


Notes:


1. The experiment revolved around a scene in Minority Report, which features a man and a young, debilitated woman, struggling to hide and escape from their pursuers in a mall. All while a slow-beat romantic theme is heard as coming from the building loudspeakers. The melody is probably intended to make the viewers think about a couple caring for each other. This contrasts with the rough and fast interactions between the protagonists, increasing the feeling of urgency and threat of the scene. Interestingly, the experiment found that when the music sounded like a normal soundtrack (with higher volume and quality than it would have if it came from the mall), the interactions between the characters were perceived as more cooperative and the scene as less tense. Our interpretation serves as an example of how mood incongruent melodies could contribute in a positive way to cinematic experiences. But, more importantly, the experiment suggests that, even when the mood depicted by the music is incongruent with a scene, placing it in the fictional world can still enhance its effects.

2. Sub-plots involving musicians can also justify the existence of particular lyrics and melodies. Around the final episodes of K-On!, for example, it becomes increasingly difficult for Azusa, to cope with the fact that her older friends will be leaving both the school and the band after graduation. As usual she tries to control her feelings, but in the day of their graduation this becomes almost impossible. It’s at this point where the girls reveal that they have composed a song specially for her. The piece, titled “Touched by an angel!” expresses the positive impact Azusa has had in their lives, and how much they appreciate this. The band proceeds then to perform the song in front of Azusa. Who, while still emotional, also appears comforted and thankful.

3. It’s seems a bit incongruent that adding video to a musical performance provokes music to be rated better, while at the same time it doesn’t lead to more positive or exciting experiences. One possible explanation is that the mentioned studies used different types of stimuli (Wapnick, Darrow, Kovacs, & Dalrymple, 1997, used a singer, while Wapnick, Mazza, & Darrow, 1998 focused on violinists) and collected different measures (like low vs. high ratings of music performance, as well as valence and arousal of the experienced mood). Another cause for the different results observed is the use of relatively simple stimuli, like a single instrument performance. In the lab, this setting prevents the interference of external variables. However, it also makes it more difficult to generalize the results to Anime music, were different types of instruments and electronic techniques are used to create the composition. Finally, other variables like the effect of watching a beloved character compose, and perform music, haven’t been studied. So, there is both, space for speculation, and additional research to be done.

4. It should be noted that while Christmas is a more family oriented holiday in many countries, in Japan is treated as a romantic event, similar to Valentine’s Day.

Note:

I want to give special thanks to Dr. Siu-Lan Tan, for being kind enough to give me additional information about her research on Diegetic Music in live-action films. You can find her Psychology Today blog here:

https://www.psychologytoday.com/blog/what-shapes-film

References:

Broughton, M., & Stevens, C. (2009). Music, enjoyment and marimba: An investigation of the role of movement and gesture in communicating musical expression. Psychology of Music, 37(2), 137-153. doi: 10.1177/0305735608094511

Rolison, J. J., & Edworthy, J. (2013). The whole is greater than the sum of its parts: Local and structural features in music listening. Psychomusicology: Music, Mind and Brain, 23(1), 33 - 48. doi: 10.1037/a0032442

Tan, S. -L., Spackman, M. P., & Wakefield, E. M. (2008, August*) Effects of diegetic and non-diegetic presentation of film music on viewers’ interpretation of film narrative. Conference Proceedings for the 2008 International Conference of Music Perception and Cognition (pp. 588-593). Hokkaido, JP.
[*Revised and expanded version provided by first author]

van der Zwaag, M. D., Westerink, J. H. D. M., & van den Broek, E. L. (2011). Emotional and psychophysiological responses to tempo, mode and percussiveness. Musicae Scientiae, 15(2), 250-269. doi: 10.1177/1029864911403364

Vines, B. W., Krumhansl, C. L., Wanderley, M. M., Dalca, I. M., & Levitin, D. J. (2011). Music to my eyes: Cross-modal interactions in the perception of emotions in musical performance. Cognition, 118(2), 157– 170. doi: 10.1016/j.cognition.2010.11.010

Vuoskoski, J. K., Thompson, M. R., Clarke, E. F., & Spence, C. (2014). Crossmodal interactions in the perception of expressivity in musical performance. Attention, Perception, & Psychophysics, 76(2), 691-604. doi: 10.3758/s13414-013-0582-2.

Vuoskoski, J. K., Gatti, E., Spence, C., & Clarke, E. F. (2016). Do visual cues intensify the emotional responses evoked by musical performance? A psychophysiological investigation. Psychomusicology: Music, Mind and Brain, 26(2), 179-188. doi: 10.1037/pmu0000142

Vuoskoski, J. K., Thompson, M., Spence, C., & Clarke, E. F. (2016). Interaction of sight and sound in the perception and experience of musical performance. Music Perception, 33(4), 457-471. doi:10.1525/mp.2016.33.4.457

Wapnick, J., Darrow, A. A., Kovacs, J., & Dalrymple, L. (1997). Effects of physical attractiveness on evaluation of vocal performance. Journal of Research in Music Education, 45(3), 470-479. doi: 10.2307/3345540

Wapnick, J. Mazza, J. K., & Darrow, A.-A. (1998). Effects of performer attractiveness, stage behavior, and dress on violin performance evaluation. Journal of Research in Music Education, 46(4), 510-521. doi: 10.2307/3345347

How Anime makes you cry.

If there is something that distinguishes Anime from other mediums is its capacity to provoke strong emotional responses. This is particularly true in the case of sad stories. It’s not uncommon to hear fans recall particularly sad scenes or even recommend a show based on its capacity to trigger crying episodes. But why is Japanese animation so good at this? What makes it such an effective tear-jerking machine?

One of the reasons Anime is capable of making people cry is because of how much the audience ends up caring for their characters. This might have something to do with the way they are designed and animated. Characters like Nayuki from Kanon (Futono, Nakamura, Hatta, Nakayama, & Ishihara, 2006) and Mirai from Beyond the Boundary (Tanaka, Senami, Saito, Nakamura, & Ishidate, 2013), for example, are given delicate features, and cute mannerisms. At first glance this seems to be a superficial choice, but apparently it can make characters more emotionally relevant. Research has shown that people tend to care more for aesthetically pleasant movie characters (Konijn & Hoorn, 2005). Protagonists don’t have to be perfect though. While Nayuki and Mirai are presented as both emotionally and physically strong, they are also clumsy and immature in at least one aspect of their life: Ayu needs an army of alarm clocks in order to wake up early, and Mirai has trouble making enough money to eat properly. Writers probably do this to give them an occasional image of helplessness. An attribute that could elicit a protective attitude in the viewer. Regardless of this, many experts believe that the spectators’ tendency to care for a fictional character has more to do with how they treat other people. Affective Disposition Theory argues that one of the first things an audience does is assess the characters’ moral behavior. If they are good, they wish for their success and fear their misfortune. If they are bad, the opposite happens (a detailed explanation of this process can be found in Zillmann, 2011). Research has found, for example, that people tend to feel more emotionally involved towards a movie character if they show good intentions (Konijn & Hoorn, 2005). Making their protagonists kind isn’t a rule creators follow strictly though. The title character from The Melancholy of Haruhi Suzimiya (Ishihara, 2006), is both attractive and energetic. But her tendency to impose her will onto others and physically abuse them may have ended up alienating some viewers. This changes in the movie, were the story focuses on the efforts of her schoolmate Kyon, trying to restore his timeline with the help of the extremely shy Yuki. Following more sympathetic characters might have allowed viewers to experience a higher level of involvement; which would explain why the film received more positive reviews.

Kindness seems to go a long way when trying to win the viewer’s heart.

While the audience disposition towards characters explain why they feel sad, it doesn’t tell us how emotional arousal reaches high enough levels to trigger a crying episode. Excitation Transfer Theory (Zillmann, 1996) states that residual arousal from previous events can combine with those provoked by a current situation, elevating the intensity of the experienced emotion. This makes sense given that certain aspects of the emotional experience, like the feeling of excitement, is maintained by the release of hormones in the bloodstream. A process that can take a couple of minutes to dissipate. The tone of the previous experience also seems important: People tend to feel more involved when watching a depressing scene, if they’ve previously seen another sad clip, instead of a happy one (Zillmann, Mody, & Cantor, 1974). Excitation Transfer would explain why people feel progressively sad with each adversity the brothers from the movie Grave of the Fireflies (Hara & Takahata, 1988) have to experience. But what happens in the case of a TV series? People watching Clannad (Ishihara, 2007) had to wait days between sad events; and even months just for the second season to start. More than enough time for their arousal levels to normalize. In spite of this though, the series is referred to as one of the most emotionally intense dramas ever produced. How is this possible? Well, one explanation could be that watching Tomoya and Nagisa face another misfortune, reminded viewers of all the past difficulties the protagonists had to go through. And, given that recalling unhappy memories is enough to induce people into a sad mood (Westermann, Kordelia, Stahl, & Hesse, 1996), it’s possible that these past scenes may have increased the emotional arousal of viewers. Something that would have made the current event seem even more depressing. If this is true, it could explain why viewers feel more emotionally involved with every additional sad scene, even when they are separated by several days or weeks. Interestingly, it also suggests that, as a series progresses and the depressing moments accumulate, the capacity of a scene to remind viewers of past adversities will have a stronger effect on their emotions. Maybe even more so than actual tragedies. One of the saddest moments in Clannad, for example, isn’t the result of suffering or dead, but of Ushio, standing in a flower field, crying over the loss of a toy Tomoya had just bought her. When he tries to explain that they can buy another one, she calls it: “Dad’s first gift”. A phrase that reminds the audience of all the time together they have lost in the past years.

Longer dramas might benefit from reminding the viewer of the
protagonists’ past misfortunes.

The repeated exposure to sad scenes eventually puts the viewers in a position where they can barely control themselves. And given that we are naturally biased to express intense sadness through crying, tearing-up becomes almost unavoidable. But just to be sure, Anime directors like to introduce a very effective technique: They make their characters cry. The referred scene in Clannad, for example, is followed by a close-up to Tomoya’s face trying to contain his tears, only to burst up crying. Exposure to this type of content has a predictable effect on people. In an experiment with vignettes, participants reported a higher tendency to feel bad and offer their support to someone in trouble, if they saw them crying (Hendriks, Croon & Vingerhoets, 2008). This result can also be improved by the appropriate soundtrack. A recent study showed that when people listen to sad music, they tend to rate images of individuals crying as more pleasant and kind (Hanser, Mark, Zijlstra, & Vingerhoets, 2015); making them even more deserving of sympathy. Additionally, watching someone tear up has the potential to stimulate any related ideas in the user’s mind; a process known as Spreading Activation (Collins, & Loftus, 1975). It’s possible then that a scene depicting a beloved character’s eyes filling with tears could be promoting the activation of a similar response, in an already highly aroused viewer. In other words, crying scenes would have a dual effect: They push the audience to an unbearable emotional state, while also showing them a way out of it (1).

Well animated tears could make crying unavoidable during states of
high emotional arousal.

The theories and studies we’ve discussed allow us to understand how Anime manages to provoke such strong emotional responses. But there might be other variables involved. For instance: People could feel either engaged or detached depending on the type of adversity the main characters have to face. Viewers of The Pet Girl of Sakurasou (Ishizuka, 2012) may feel fully invested in the love-triangle, but completely disregard the vocational setbacks the male lead has to go through. It’s also possible that individuals with different age, gender or beliefs would develop opposite attitudes towards the same character. There are still many questions to answer. Sadly, while there is plenty of research in respect to video games and movies; empirical studies that focus specifically on Anime are hard to find. We hope that this type of articles motivates fans and academics to perform their own investigation. Maybe even their own experiments.


Notes:


1. While crying is a natural response, it’s not considered an acceptable behavior in most social contexts. As a result, many people actively try to avoid crying while watching a movie. The effort put into blocking any unwanted behavioral response could work in favor of the emotional experience though. According to an experiment, people who inhibit themselves from crying while watching a sad short, show higher arousal levels than those who allowed themselves to tear up (Labott & Teleha, 1996). This would facilitate the accumulation of emotional arousal. Thus improving the involvement experienced in future scenes.


References:


Collins, A. M., & Loftus, E. F. (1975). A Spreading-Activation Theory of Semantic Processing. In Psychological Review, 82(6), DOI: 10.1037/0033-295X.82.6.407.

Futono, N. (Producer), Nakamura, S. (Producer), Hatta, Y. (Producer), Nakayama, Y. (Producer) & Ishihara, T. (Director). (2006). Kanon [Television series]. Japan: Kyoto Animation.

Hanser, W. E., Mark, R. E., Zijlstra, W. P., & Vingerhoets, Ad. J. J. M. (2015). The effects of background music on the evaluation of crying faces. In Psychology of Music, 43(1). DOI: 10.1177/0305735613498132

Hara, T. (Producer), & Takahata, I. (Director). (1988). Grave of the Fireflies [Motion picture]. Japan: Studio Ghibli.

Hendriks, M. C. P., Croon, M. A., & Vingerhoets, AD J. J. M. (2008). Social reactions to adult crying: The help-soliciting function of tears. In The Journal of Social Psychology, 148(1). DOI: 10.3200/SOCP.148.1.22-42

Ishihara, T. (Director). (2006). Suzumiya Haruhi no Yūutsu [Television series]. Japan: Kyoto Animation.

Ishihara, T. (Director). (2007). Clannad. [Television series]. Japan: Kyoto Animation.

Ishizuka, A. (Director). (2012). Sakura-sō no Pet na Kanojo [Television series]. Japan: J.C. Staff.

Konijn, E. A., & Hoorn, J. F. (2005). Some like it bad: Testing a model for perceiving and experiencing fictional characters. In Media Psychology, 7(2). DOI: 10.1207/S1532785XMEP0702_1

Labott, S. M., & Teleha, M. K. (1996). Weeping propensity and the effects of laboratory expression or inhibition. In Motivation and Emotion, 20(3). DOI: 10.1007/BF02251890

Tanaka, G. (Producer), Senami, R. (Producer), Saito, S. (Producer), Nakamura, S. (Producer) & Ishidate, T. (Director). (2013). Kyōkai no Kanata [Television series]. Japan: Kyoto Animation.

Westermann, R., Kordelia, S., Stahl, G., & Hesse, F. W. (1996). Relative effectiveness and validity of mood induction procedures: A meta-analysis. In European Journal of Social Psychology, 26(4). DOI: 10.1002/(SICI)1099-0992(199607)26:4<557::AID-EJSP769>3.0.CO;2-4

Zillmann, D. (1996). Sequential dependencies in emotional experience and behavior. In Kavanaugh, R.D., Zimmerberg, B., & Fein, S. (Eds), Emotion: Interdisciplinary Perspectives. (pp. 243-272). New Jersey: Erlbaum.

Zillmann, D. (2011). Mechanisms of emotional reactivity to media entertainments. In Doveling, K., von Scheve, C., & Konijn, E. A. (Eds), The Routledge Handbook of Emotions and Mass Media (pp. 101-115). New York: Routledge.

Zillmann, D., Mody, B., & Cantor, J. R. (1974). Empathetic perception of emotional displays in films as a function of hedonic and excitatory state prior to exposure. In Journal of Research in Personality 8(4). DOI: 10.1016/0092-6566(74)90025-7

Can video games make you sexist? Part I: A complex relationship.

In the last couple of years there has been an intense discussion regarding the presence of sexist content in video games. Critics have pointed out, for example, how there aren’t enough avatars designed towards women. They have also accused developers of using female characters to make the male protagonist look strong and brave, provide sexually arousing stimulus or even satisfy fantasies about overpowering and harming women. The most common argument against this type of content, however, is based on the belief that it can have long term effects on how the players think and behave. Something that could end up increasing discrimination and even violence against women. But is there any scientific basis for these claims? Can video games make you sexist?

Despite some references to research, studies about sexist games are mostly ignored.

Researchers have been studying the negative effects of media for decades. But while there are probably hundreds of papers about the link between video games and violence, works that focus on the effects of sexist game content are relatively scarce. In spite of this, however, researchers have been able to obtain some interesting results. Let’s take a look at correlational studies, for example.  In this type of work, researchers collect two pieces of data from each participant. Then they analyze if an increase or a decrease in one variable is followed by a change in the other. If this happens so frequently that it can't be attributed to chance, then it’s safe to assume there is some sort of relationship between them. So, is there a relationship between playing video games and sexism?

Well, studies have found no correlation between video game usage and general sexist beliefs (“men should be in charge”) (Breuer, Kowert, Festl, & Quandt, 2015). They did found a connection, though, with hostile forms of sexist attitudes (“women want to take control away from men”) as well as irrational beliefs about sexual violence (“women in a revealing dress are asking for it”) (Fox & Potocki, 2015). On the other hand, playing violent video games has been associated with negative attitudes toward women (Dill, 2009), a lower tendency to interpret unwanted sexual advances as harassment (Dill, Brown, & Collins, 2008), and acceptance of myths about rape (“woman refuse to have sex, even though they want to) (Dill, 2009; Dill, et al., 2008). In other words, there seems to be a link between video game usage and both negative attitudes against women and tolerance towards sexual violence against them.

There seems to be a link between video game usage and at least some forms of sexism.

Now, some people might see this as proof that video games promote sexism, but the truth is a bit more complicated than that. While correlations are an excellent way to find connections, they don’t tell us what type of relationship there is. Results could indicate that games make people sexist. But it could also mean that sexist individuals prefer video games over other kinds of media, especially those with violent content. In order to solve this problem we have to find out if there is a causal relationship between playing video games and sexism, and that’s something only experiments can tell us.


Visit us in two weeks for the second part of this series, when we will analyze experiments that test if exposure to sexualized female characters or sexist behavior can change the way people think.


References:

Breuer, J., Kowert, R., Festl, R., & Quandt, T. (2015). Sexist Games = Sexist Gamers? A longitudinal study on the relationship between video game use and sexist attitudes. Cyberpsychology, behavior, and social networking, 18(4). doi: 10.1089/cyber.2014.0492

Dill, K. E. (2009). Violent video games, rape myth acceptance, and negative attitudes towards women. In: E. Stark and E. S. Buzawa (Eds.), Violence Against Women in Families and Relationships: Volume 4, The Media and Cultural Attitudes (pp. 125-140). Westport, CT: Praeger.

Dill, K. E., Brown, B. P., & Collins, M. A. (2008). Effects of exposure to sex-stereotyped video game characters on tolerance of sexual harassment. Journal of Experimental Social Psychology. 44(5). 1402-1408. doi: 10.1016/j.jesp.2008.06.002

Fox, J., & Potocki, B. (2015). Lifetime video game consumption, interpersonal aggression, hostile sexism, and rape myth acceptance: A cultivation perspective. Journal of Interpersonal Violence. DOI: 10.1177/0886260515570747

Can we use games to control our mood?

Video games are very complex experiences. They involve paying attention to virtual objects, vehicles and characters; all working together to offer both a challenge and a narrative. As a result they provoke a wide range of emotions. Games can make us anxious and scared, but also relaxed. They can help change boredom into excitement, and sadness into happiness. They seem to be able to change how we feel. But how reliable are they? Can we use games to switch our emotions at will? Can we use them to control our mood?

One area of research that can help us understand the relationship between video games and emotions is Mood Management Theory. According to it, people tend to use media to improve how they feel. They can do this, for example, by seeking more positive and arousing content. Media can also distract people from the emotions, thoughts and memories that are making them feel bad. Research on this area has found some support for the potential of video games for mood repair. Let’s analyze this studies more closely.

Evidence suggest that there’s a positive relationship between how many actions a player executes in a game and how good he or she feels. For example: Playing a Wii boxing game with motion controllers was followed by a reduction in negative mood, which didn’t happen for those who played a flash boxing game with less actions available and through the use of a keyboard (Chen & Raney, 2009). This seems to suggest that titles involving additional and more realistic user actions are better for the reduction of negative mood. But there are additional differences between both games (like graphics and sound quality) that could explain the users’ change in mood. Another experiment, however, found more conclusive results. People who watched a warplane simulator (Lock-on: Modern Combat) in autopilot showed less improvement in affect than those who had to control the speed and direction of the plane (with a flight stick and a throttle), which also showed a bigger increase on affect than those who additionally had to control landing gears and flaps as well as air, wheel and parachute brakes (with a keyboard and a mouse) (Bowman, 2012). This indicates that, to promote positive affect, some demand for the intervention of the user is better than none. But when the task becomes too demanding, it starts having negative effect on mood. So if you need to feel better after receiving some bad news, playing a game would probably be more effective than just watching an un-narrated gameplay video on YouTube. However, this wouldn’t necessarily apply to reviews or “Let’s play” videos. After all, these type of content includes expert analysis as well as jokes made by successful entertainers, increasing their chances of influencing the user’s mood. Also, we wouldn’t recommend playing games with unusually complex controls and interfaces like “Dragon Ball Z: Budokai Tenkaichi” or “Master of Orion”, unless you already know how to play them pretty well.

Playing a confusing game might not be the best way to improve your mood.

A recent study could help us understand why interactivity tends to make people feel better. This experiment found that playing Pac-Man was better at reducing depressive mood than watching a clip of the game or just waiting (Rieger, Frischlich, Wulf, Bente, & Kneer, 2014). Interestingly, it was also better at reducing involvement towards a previously seen sad movie clip (Rieger, et al. 2014). This result is congruent with (but it doesn’t prove) the idea that interactivity improves mood by distracting the user from negative content. So if you find yourself experiencing a deep feeling of sadness (and social support isn’t available at the moment), try playing a game that can capture your attention for long periods of time. Whether it is a puzzle or an online racing game, it might help you avoid negative thoughts or memories that would only have made you feel worse.

Research has also found support for a positive effect of interactivity on arousal. People that chose to spend a break playing a video game instead of just waiting, experienced a higher increase in self-reported arousal (Reinecke & Trepte, 2008). It’s possible though that those who choose the game had better expectations towards the improvement of their mood, so a placebo effect can’t be ruled out as an explanation. Nevertheless, another study found more solid evidence of the relationship. This experiment found that playing Pac-Man was better than watching a clip of the game at increasing self-reported arousal but not electro-dermal activity (a physiological measure of arousal) (Rieger, Frischlich, Wulf, Bente, & Kneer, 2014). This suggest that interactivity can make users’ feel more energetic, but won’t necessarily trigger the physiological responses associated with it. Interestingly, studies on movies have found that sad content is an effective way to increase arousal, and that it works better than comedy (Rieger, Reinecke, Kneer, Frischlich, & Bente, 2013; Rieger, Bowman, Frischlich, & Bente, 2014). So playing an emotional interactive experience like the ones provided by “Final Fantasy IV” or “Shadow of the Colossus” could be a good way to make an under-stimulated individual feel more excited. However, if someone is feeling both bored and depressed, the use of sad content could make things worse. In this case, games with less emotional storylines like “Mario RPG” or more action oriented like “Marvel vs. Capcom” or “FIFA” would be a better choice.

Emotionally intense content seems to make people more excited.

As we have seen, there’s enough evidence to believe that interactivity has a positive effect on people’s mood. But this doesn’t necessarily mean video games are always reliable as a source of mood repair. For example: While Pac-Man has been shown to decrease depressive feelings, it hasn’t been found as effective in the promotion of a happy mood (Rieger, Frischlich, Wulf, Bente, & Kneer, 2014). On the other hand, titles like Hitman: Blood Money (a third person stealth game), Call of Duty 2 (a war-themed first person shooter) or Madden: 2007 (an American football game) aren’t particularly useful for the reduction of sadness or hostility (Ferguson & Rueda, 2010). One way to explain this is that other game variables are affecting how the users feel. Maybe Pac-Man is challenging enough to distract people from negative memories, but the animation and sounds aren’t enjoyable enough to promote positive affect. And while Hitman and Call of Duty are known for their action packed missions, they also tend to include sad music and storylines that we wouldn’t recommend for players feeling depressed.

Some games can be surprisingly ineffective at reducing negative mood.

One of the lessons we can obtain from these studies is that providing different and interesting opportunities to interact with the game is a good way to keep the players happy and excited. Megaman X, for example, achieved this by letting the players jump from walls as well as charge their weapon. In Tenchu: Stealth Assassins, the user could hide behind a wall to avoid detection, turning an otherwise passive element like the environment into an important tool. Actions doesn’t even have to be important for the in-game objectives. Creators of Grand Theft Auto V, for example, included simple features like being able to switch between radio stations, turn the lights of the car on and off or even make rude gestures to other drivers as a way to prevent the players from becoming too bored during long road trips. The demand for user intervention should be limited though. As we have seen, overwhelming controls can have a negative effect on mood. Nevertheless, we should remember that this didn’t prevent the first person shooter “Arma II” to gain a considerable amount of followers. This reminds us of how important it is to know your target audience, the experience they want and what they are willing to overlook in order to obtain it.

Simple actions, like playing with the lights of a car, prevent users from
switching their attention away from the game.

A final advice would be to avoid any situation that takes the control away from the user. A good example are quick-time events. This technique (introduced in the Shenmue series and included in recent games like the new Tomb Raider) consists of triggering a complex set of actions (like a martial arts move) by pressing a single button at the right time. Quick-time events are used to insert cinematic scenes during gameplay without completely eliminating the intervention of the user. One way to improve this technique would be to use a more natural approach, like pressing right or left to avoid a falling object instead of just pushing the “action” button. This would help smooth the transition between real-time action and “interactive” cut-scenes.

Video games seem to be a good way to change how you feel. Experienced players probably even know what titles and genres match their mood repairing needs. But these effects are probably only temporal and do not substitute social support or professional counseling. Games are not a long term solution. At best, they can help you take a break from angry thoughts, depressive feelings or even traumatic events. They might give you the space you need to see things more clearly. To set a course of action and start working on your problem.

References:

Bowman, N. D., & Tamborini, R. (2012). Task demand and mood repair: The intervention potential of computer games. New Media & Society, 14(8), 1339-1357. doi: 10.1177/1461444812450426

Chen, Y., & Raney, A. A. (2009, May). Mood management and highly interactive video games: An experimental examination of Wii playing on mood change and enjoyment. Paper presented at the annual meeting of the International Communication Association, Chicago, IL.

Ferguson, C. J., & Rueda, S. M. (2010). The hitman study. European Psychologist, 15, 99–108. doi: 10.1027/1016-9040/a000010

Reinecke, L. & Trepte, S. (2008). In a working mood? The effects of mood management processes on subsequent cognitive performance. Journal of Media Psychology: Theories, Methods and Applications, 20(1), 3-14. doi: 10.1027/1864-1105.20.1.3*

Rieger, D., Bowman, N. D., Frischlich, L., & Bente, G. (2014, May). “I’m pumped, but I don’t feel like it!” The differential effects of affect and arousal regulation on mood repair and recovery. Paper to be presented at the annual meeting of the International Communication Association, Seattle.

Rieger, D., Frischlich, L., Wulf, T., Bente, G., & Kneer, J. (2014). Eating Ghosts: The Underlying mechanisms of mood repair via interactive and noninteractive media. Psychology of Popular Media Culture.

Rieger, D., Reinecke, L., Kneer, J., Frischlich, L., & Bente, G. (2013, June). Media induced recovery: The effects of positive versus negative media stimuli on recovery experience, cognitive performance and vitality. Paper presented at the 63th Annual International Communication Association Conference. London: UK.

Are skilled players more likely to experience flow?

What is the relationship between skill, challenge and flow?

One of the most attractive aspects of video games is the possibility to experience flow. This happens when a user engages in an activity that fits his set of skills while still maintaining a certain level of challenge. The individual then becomes so focused in the task that the outside world slowly starts to disappear. Flow can make people feel better about themselves, have fun and forget about their problems. This makes it one of the most desirable objectives for both gamers and developers. But what attributes are expected to produce flow?

Most experts think there is a strong relationship between how skilled a player is, the amount of challenge he or she faces and the experience of flow. But finding what levels of challenge and skill can produce flow has not been that easy. For example: Since flow is associated with reward, then higher levels of difficulty should have a positive effect on flow, right? Well, not exactly. A recent experiment found no significant effects of difficulty on flow in a Tower Defense game (Schmierbach, Chung, Wu & Kim, 2012). Additionally, two experiments involving Wii games showed that the level of perceived challenge didn’t affect flow on “Trauma Center: New Blood” (a surgical simulator that uses haptic feedback on hand-held controllers) or “Need for Speed” (a racing game played, in this experiment, with a steering wheel) (Jin, 2011). It did, however, made the surgical instruments feel more real (physical presence) and increased the sensation of being inside a race (spatial presence), which had a significant effect on flow (Jin, 2011). This seems to suggest that challenge can have an indirect effect on flow. The high number of threats a player has to face in Ninja Gaiden, for example, could increase the importance of in-game events, making everything feel more real. This would make victories more rewarding but could also increase engagement, making the players forget about their problems.

Ninja Gaiden’s difficulty could have increased both presence and flow.

Cognitive skills, on the other hand, seem to have a rather unstable relationship with flow. A study on first person shooters, for example, revealed that the ability to hit a static target improved flow for players using a gamepad, but not for those using a motion controller (Bowman & Boyan, 2008). This suggests that the capacity of a skill to promote flow can be affected by different aspects of gameplay, like the type of interface being used. One possible explanation could be that when the characteristics of a game change, different abilities are required. In other words, the right set of abilities is more important than having many skills. This would increase performance and, therefore, the feeling of competence; both variables that have been associated with higher levels of flow (Bowman & Boyan, 2008; and Jin, 2012, respectively). The idea is congruent with an experiment in which players who rated better their ability to play a racing game achieved higher levels of flow while playing it (Jin, 2011). So, an ability for planning and leading organized attacks might improve your chances of experiencing flow in “World of Warcraft” and “League of Legends”, but it might not be that useful in a more chaotic environment like an online “Call of Duty” match.

An ability that produces flow in a racing game could be useless in a surgical simulator.

How much challenge or skill is needed for flow to occur? A series of studies on Wii titles showed that, more or less, low levels of challenge or skill tend to produce low levels of flow (Jin, 2011; 2012). This is especially clear in the case of “Need for Speed” (Jin, 2011) and “Need for Speed: Nitro” (Jin, 2012). This suggests that, at least in the case of racing games, a minimum level of demand or ability is required to experience flow. However, an experiment that measured the playfulness trait on people playing “Wii Fit” showed an interesting result. When playfulness was low, the effect of low challenge and skill on flow was, as expected, usually low (Jin, 2012). But when participants were highly playful, low levels of challenge and skill produced the highest levels of flow (Jin, 2012). This probably happens because the absence of challenge or skill means there will be little punishment for errors or reward for victories. This means that there will be less negative consequences if the player engages in free exploration, something that tends to attract highly playful people. So, if you start playing a Real Time Strategy games like Red Alert or StarCraft for the first time, try going through the tutorial campaign first. This will help you improve your skill, increasing your chances of experiencing flow later on. Unless you are the type of person that enjoys finding out the different features of a new game first; like listening the character’s voices or watching the different types of deaths.

Forgetting the mission and exploring the surroundings can be fun too.

Researchers believe that balance between challenge and skill should produce the highest levels of flow. This makes sense as a challenge superior to the player’s abilities would end up promoting frustration, and a level of difficulty inferior to the user’s skills would make the task boring. The hypothesis is supported by part of the evidence: Balance between skill and challenge has been associated with flow for computer games like “Pac-Man” (Engeser & Rheinberg, 2008) and “Bloons Tower Defense 4” (Schmierbach, Chung, Wu, & Kim, 2012), as well as for Wii titles like “Need for Speed” (Jin, 2011), “Need for Speed: Nitro” and “Mad World (a gory beat-em-up) (Jin, 2012). This didn’t happen, however, in the case of “Trauma Center: Second Opinion” and “Wii Fit” (Jin, 2012). One way to explain this is that specific game attributes made the experience of failure less negative or the positive feedback obtained from victories more rewarding, preventing a game that is too difficult or too easy to become less enjoyable. Maybe the absence of virtual enemies in “Trauma Center” made the player care less about losing and “Wii Fit” users were more interested in exploring the capabilities of the Kinect than scoring the highest score. In any case, evidence suggest that there is no sure way to provoke flow, but balance between challenge and skill is definitely a good start.

Balance between challenge and skill seems to be less important in games like “Wii Fit”.

It’s seems difficult to predict when or even if flow will happen. But that doesn’t mean there isn’t something we can do to increase our chances of either enter the state or provoke it. The first lesson we can obtain from the reviewed studies is that some skill is better than no skill. And while no player starts a game with a complete understanding of the mechanics, going through a tutorial or keeping the “hints” option activated could help users make a faster transition from “Wow, easy there!” to “Ok, let’s do this!”. The second lesson is that the relationship between challenge and skill seems to play an important role in the occurrence of flow. This means that it’s particularly important for developers to provide different difficulty levels the user can choose from, as well as carefully verify if the challenge increases with each stage just enough to match the player’s newly acquired skills. The final lesson we must remember is this: The circumstances of play can change what does and what doesn’t promote flow. Different game types and user preferences can, as we have seen, prompt unexpected situations, like a highly playful individual entering a deep state of flow just by ignoring all the objectives and do what he or she wants.

Flow is a complex phenomenon. Probably as complex as different types of games and players are out there. Finding a formula that consistently triggers intense and prolonged flow states in every single player might be impossible. But thanks to the work of both researchers and developers, we can expect flow to become a more accessible type of experience with every new generation of games.

References:

Bowman, N. D., & Boyan, A. C. (2008, May). Cognitive skill as a predictor of flow and presence in naturally mapped video games. Paper presented at the 58th annual convention of the International Communication Association, Montreal, Canada.

Engeser, S., & Rheinberg, F. (2008). Flow, performance and moderators of challenge-skill balance. Motivation and Emotion, 32(3), 158-172. doi:10.1007/s11031-008-9102-4

Jin, S. A. (2011). I feel present. Therefore, I experience flow: A structural equation modeling approach to flow and presence in video games. Journal of Broadcasting & Electronic Media, 55(1), 114-136. doi: 10.1080/08838151.2011.546248

Jin, S. A. (2012). Toward integrative models of flow: Effects of performance, skill, challenge, playfulness, and presence on flow in video games. Journal of Broadcasting & Electronic Media, 56(2), 169-186. doi: 10.1080/08838151.2012.678516

Schmierbach, M., Chung, M., Wu, M. & Kim, K., (2012, May). No one likes to lose: Game difficulty, motivation, immersion and enjoyment. Paper presented at the annual meeting of the International Communication Association, Phoenix, AZ.

The Power of Video Game Music

Music has become an essential part of the video game experience. Developers and composers have managed to make stories more compelling and challenges more attractive with the help of carefully designed soundtracks. Games like Ninja Gaiden for the NES or successful series like Metal Gear or Bioshock probably wouldn´t be the same without the beautiful music especially prepared for them. However, we still know very little about the how and why music affect us the way it does. Why does some songs seem playful and other serious? Why some music make us hurry while other tends to relax us? And if music is capable of affecting how we feel, could it also makes us better players?

Let’s start by talking about the relationship between music and emotions. Video game music is frequently associated with positive affect. The highly motivational theme we hear while waiting for a Battlefield map to load or the playful tune we listen to in the wizard town of Mysidia in Final Fantasy IV are good examples. But music can also be used to provoke negative feelings. The presence of music, for example, was shown to increase stress levels in a group of people playing Quake III (Hébert, Renée, Dionne-Fournelle, Crete & Lupien, 2005). It was also found to make players less relaxed and increase how dangerous everything felt in The Lord of the Rings: The Two Towers game (Lipscomb & Zehnder, 2004). This suggest that listening to certain types of music can make players feel more afraid of negative in-game events, like being shot by the enemy or falling off a cliff. This capacity to elicit negative affect isn’t necessarily a bad thing. In Resident Evil 2, for example, the fear of a surprise encounter with an enemy forces players to pay close attention to any suspicious movement or sound, increasing both engagement and emotional involvement. Also, the constant threat of dying and losing all the unsaved progress turns into a profound feeling of relief when a typewriter (a tool used to save the player’s game) is eventually found. So, even when music promotes negative affect, this could indirectly lead to more immersive and rewarding experiences.

Skyrim’s outdoor theme switches to a more action oriented tune in the presence of an enemy.

Developers seem to believe that fast paced music can increase the fear and anxiety provoked by an approaching threat. Super Mario Bros. titles, for example, tend to accelerate the speed of music when time is running out. Castlevania III: Dracula’s Curse, on the other hand, uses a high tempo song (appropriately named “Pressure”) in a level where the player must hurry to escape constantly rising water levels. But how does this work? One possibility is that people use the speed of music as a way to measure time. The faster it goes, the more it feels like additional time has passed. This could also make us feel like we are moving slowly. The idea seems to be congruent with an experiment in which fast paced music made players drive faster in a racing game (Cassidy & MacDonald, 2010). So, if you are running away from an enemy, then chances are a fast song will make you feel like you are about to lose your life.

Castlevania III added a high-tempo song to a flooding level to increase the feeling of an approaching threat.

While altering tempo seems to be a very reliable way to affect gameplay, media producers also depend heavily on the use of highly arousing music. This is especially obvious in the trailers for summer blockbusters, were intense compositions are used to transmit a feeling that something great and important is happening (just watch the theatrical trailer for Sam Raimi’s Spiderman 2). Video games are no exception. Arousing music tends to accompany the most important events in a level or game. Scenes like the Battle of Stalingrad in the original Call of Duty wouldn’t have transmitted such strong emotions if it wasn’t for the epic but sad theme that played in the background. However, this type of songs might not go with every type of game. People playing Project Gotham Racing 3, for example, found high arousal music less appropriate and enjoyable as well as more distracting than low arousal songs (Cassidy & MacDonald, 2010). It also made them reach higher speeds and achieve shorter lap times, as well as hit more obstacles (Cassidy & MacDonald, 2010). This is congruent with the idea that highly arousing music motivates individuals to act with more energy and speed, but also more recklessly. So if you want start a carefully planned campaign in a Real Time Strategy game, we don’t recommend listening to John Williams´ “Duel of Fates”.

Highly arousing music made players go faster but also more reckless in Project Gotham Racing 3.

As we have seen there are different ways to influence the player’s emotions and behavior through music. Video game companies know this and invest a considerable amount of resources in designing the best possible soundtrack. This involves carefully matching different styles of songs to specific types of games or scenes: A heavy metal song may go well with violent action scenes as well as a theme of revenge, but it may feel out of place in a stealth mission or a scene depicting the dead of a beloved character. Still, some players think games should give users the option to listen to their own music. This would allow them to build a more personalized experience, choosing less distracting and more arousing songs, eventually increasing both engagement and emotional involvement. In the previous experiment, for example, listening to self-selected music was not only preferred but also improved the participants ability to avoid obstacles randomly placed in the racing track (Cassidy and MacDonald, 2010). Also, incongruence between soundtrack and audiovisual content seems to have a very limited effect in immersion, the feeling of being inside the content and how realistic the events depicted seem; at least in the case of movies (Petey, Campanella, Rubenking, Buncher & Gress, 2010). Still, developers won’t necessarily appreciate the idea of having less control over the final feel of their product and additional evidence would be required before most of them adopt this approach.

GTA: San Andreas and WWE 13 allow the the user to add their own music, increasing replay value.

Research on the relationship between music and the user experience could have an important effect in the future of video games. Especially considering the current interest in the development of soundtracks that adapt to the user’s feelings and behavior. But instead of using this newly acquired knowledge to provoke a more intense experience, developers could just employ it to obtain more precise effects. After all, triggering a very specific sets of emotions or reduce the levels of distraction to improve performance could be enough to make players perceive a significant improvement in video game quality. Findings could also be applied to different fields like the use of music in work-place to reduce stress without affecting performance. In the end research about video game soundtracks could end up affecting not only how we play at home, but also how we spend our time in the office.

References:

Cassidy, G. G. & MacDonald, R. A. R. (2010). The effects of music on time perception and performance of a driving game. Scandinavian Journal of Psychology, 51(6), 255-464. doi: 10.1111/j.1467-9450.2010.00830.x.

Hébert, S., Renée, B., Dionne-Fournelle, O., Crete, M., & Lupien, J. (2005). Physiological stress response to video-game playing: The contribution of built-in music. Life Sciences, 76(20), 2371-2380

Lipscomb S. D. & Zehnder, S. M. (2004). Immersion in the virtual environment: The effect of a musical score on the video gaming experience. Journal of Physiological Anthropology and Applied Human Science, 23(6), 337-343.

Petey, G., Campanella, C., Rubenking, B., Buncher, M., & Gress, E. (2010). Telepresence, soundscapes and technological expectation: Putting the observer into the equation. Virtual Reality, 14(1), 15-24. doi: 10.1007/s10055-009-0148-8

Video game technology. Part 3 of 3: Immersion through sound.

Can audio really make movies and games more immersive and fun?

What is the best video game you have ever played? Don’t answer yet. What the first thing that came to your mind when you read that question? Was it a specific level in a first person shooter? A character in a role playing game? In any case you first thought was probably associated with visual images. A gun fight from a first person perspective, a character jumping between platforms or a flash of light after an explosion. Every time we think about video games we tend to think in terms of visuals. Nevertheless, sound also plays an important role in our gaming experiences. Now, you are probably thinking about the rewarding sound you heard after picking up an item or finishing an enemy. Maybe you are just remembering how emotionally moving was the soundtrack of the last game you played. And while sounds and music are not the first thing we associated with movies or video games, it is still a very important part of the experience. They can tell if there are enemies around us and even communicate the emotional tone of a scene. But can sound really affect how immersed we feel or how much we enjoy a movie or video game? Is it really that important?

Silent hill reduced visibility and forced the player to rely on sound effects.

Sounds are an essential feature of our environment. They gives us information about the objects and events around us. They can even tell us about what is happening out of our field of view. It’s no surprise then that the presence of sound has been shown to increase the feeling of being inside an interactive virtual environment (presence) (Larsson, Västfjäll, Olsson & Kleiner, 2007). It has also been associated with higher levels of involvement towards the sensory stimulus, story and characters (immersion), the positive feeling of focusing on a cognitively demanding task (flow) and reduced tension while playing a first person shooter (Nacke, Grimshaw & Lindley, 2010). In this last experiment, however, sound didn’t provoke any significant changes in the physiological indicators of emotional valence (facial muscles activity) and arousal (electro-dermal activity) (Nacke, et al. 2010). This means that either the change in emotions wasn’t strong enough to affect the physiology of players, or that the presence of sound increased their expectations, provoking them to overrate how immersed and less tense they felt. This makes the results of the second study somehow inconclusive, but there’s still an important amount of evidence of how audio can affect the experience of media.

Audio can also provide a considerable amount of information about the things and events that provoked them. The sound an object makes when falling to the ground can give us clues about what it’s made of, how heavy it is or how fast it was going. Placing speakers in different positions around the user allows him or her to use the sound as a source of spatial information. For example: If a sound comes from a speaker placed to the left of the user, he or she will feel like it’s coming from this side of the virtual environment. Sounds can also provide information about the position of its source and the acoustics of the environment, something that tends to make them easier to identify as well as feel more realistic (Larsson, Västfjäll, Olsson & Kleiner, 2007). This has also been shown to increase the feeling of being inside the content (Larsson, et al. 2007). After all, people tend to use sounds to discover the position of objects as well as the shape of their surroundings in real life.

Outdoors and enclosed spaces sound different, even in video games.

If increasing the number of audio channels can help the user better identify the position of sound sources, then audio systems that use various speakers in different positions should be able to provoke a higher sense of presence and immersion, right? According to research, using five audio channels instead of two didn’t increase the feeling of being inside the content or how realistic the events felt in the case of a movie (Freeman & Lessiter, 2001), but it did when the media was a first person shooter (Skalski, Whitbred & Lindmark, 2009). The additional number of channels also made both, the experience of the movie and the video game, more enjoyable (Freeman & Lessiter, 2001 and Skalski, et al. 2009, respectively). This suggest that additional audio channels only increase presence and realism when the user is able to interact with the content, but that this isn’t necessary in the case of enjoyment. Maybe the additional audio channels are able to make the movie experience more interesting. But in video games, where the user’s actions can affect what happens next, the information coming from the additional audio channels becomes more important. The sounds now reveal the position of potential threats and rewards, increasing levels of attention towards this type of stimuli. In this circumstances the player is able to better appreciate the details and experience more intense emotions, perceiving the content as more realistic. Also, the additional levels of engagement will eventually make him or her forget about the technology providing the experience, producing a feeling of being inside the content.

Sound direction is more relevant in games, where it can reveal the position of an enemy.

Adding bass to the audio channels while watching a movie seems to increase how clear and exciting the audio seems, but also how uncomfortably loud it may feel (Freeman & Lessiter, 2001). This might have happened because people tend to associated strong vibrations with loud noises, like a sound system turned up to its maximum volume. Interestingly, the additional bass channel also made the visual stimuli more exciting (Freeman & Lessiter, 2001). This could be attributed to a halo effect, where the additional audio quality made the subjects overstate how well other stimuli felt.

The bass channel was also found to increase the feeling of being inside the movie as well as involvement and the believability of the events depicted on it (Freeman & Lessiter, 2001). It’s possible that the viewers not only heard the bass sounds but also felt the strong vibrations associated with them. The additional tactile information could have helped the viewers experience how it would have felt to be a part of the portrayed events, increasing both presence and perceived realism.

Strong bass sounds can make the viewer feel like the floor is moving,
increasing realism and presence.

Regardless of the apparently obvious benefits of using a speaker-based surround system, some people still prefer to wear headphones. One reason to choose them could be that they help isolate the user from external noise. This would make it easier for him or her to focus on the content, increasing engagement and maybe even involvement. However, research on the effects of using headphones or speakers in the experience of a movie is still inconclusive; with one study showing an increase in engagement and involvement when using headphones (Campanella, Pettey, Guha, & Rubenking, 2010) and another one finding no significant differences between each condition (Pettey, Campanella, Rubenking, Buncher & Gress, 2010). Nevertheless, both studies found that using headphones or speakers didn’t produce any significant differences in the feeling of being inside the content or how realistic the depicted events seemed (Campanella, et al. 2010; Pettey, et al. 2010). So maybe, in the end, it’s all about comfort.

What is better for playing video games then: Headphones or speakers? The previous study indicates that there seems to be no significant differences in the case of movies. Still, we should remember that, as previous research has demonstrated, the fact that an audio variable doesn’t affect the experience of watching a movie doesn’t necessarily means that the same will happen with video games (e. g. Freeman & Lessiter, 2001 and Skalski, Whitbread & Lindmark 2009). Maybe the relationship between the isolation provided by headphones and immersion becomes stronger when the user is able to interact with the media. Still, additional studies are necessary to clarify this last question.

While improvements in the field of images and animation has been more obvious, significant advances have happened too in respect to audio technology. Built-in speakers are still available, but users now have the option to invest several hundreds of dollars in complex surround systems and highly advanced headphones. However, additional studies are required before we can assure consumers that a specific piece of hardware will make them feel more immersed, at least in the case of video games. In the meantime just ask yourself this question: Does a device helps you focus on the game? Does it make you more involved in the story and characters? Did you found yourself suddenly feeling like if you were inside the virtual environment? If the answer is yes, then you probably made the right choice.

References:

Campanella, C., Pettey, G., Guha, T., & Rubenking, B. E. (2010). Sound out small screens and telepresence. The impact of audio, screen size and pace. Journal of Media Psychology, 22(3), 125-137. doi: 10.1027/1864-1105/a000017

Freeman, J. & Lessiter, J. (2001, August). Here, there and everywhere: The effects of multichannel audio on presence. Proceedings of the 1001 International Conference on Auditory Display, Espoo, Finland.

Larsson, P., Västfjäll, D. Olsson, P. & Kleiner, M. (2007, October). When what you hear is what you see: Presence and auditory-visual integration in virtual environments. Proceedings of the 10th Annual International Workshop on Presence, Barcelona, Spain.

Nacke, L. E., Grimshaw, M. N. & Lindley, C. A. (2010). More than a feeling: Measurement of sonic user experience and psychophysiology in a first person shooter game. Interacting with Computers, 22(5), 336-343. doi: 10.1016/j.intcom.2010.04.005

Pettey, G., Campanella, C., Rubenking, B., Buncher, M., & Gress, E. (2010). Telepresence, soundscapes and technological expectation: Putting the observer into the equation. Virtual Reality, 14(1), 15-24. doi: 10.1007/s10055-009-0148-8

Skalski, P., Whitbred, R., & Lindmark, P. (2009, November). Image vs. sound: a comparison of formal feature effects on presence, video game enjoyment, and player performance. Paper presented at the 12th annual international workshop on presence, Los Angeles, CA.