For dramatic VR to succeed, music must become the director

상호작용 음악 / VR 체험

We’ve all experienced music’s power to change our mood or drum up feelings of nostalgia, fear or excitement. And beyond entertainment, numerous studies have proven that music can inspire action and actually impact change.

Retailers have used music to drive shoppers to higher price points and casinos have played music to encourage more gambling, to name just a couple of examples. It’s this use of music — not as a background mood enhancer, but actually as a director of sorts, that will be critical as virtual reality and augmented reality content creators struggle with how to direct the viewer’s gaze in 360 environments.

Figuring out music’s role will be key to the mainstream adoption of VR dramatic content. But so far music is still being relegated to a background mood feature. This needs to change, and it needs to start with a mind shift in how we think about music.

gettyimages-550766037.jpg

A paradigm shift

The use of music in film historically has been confined to on-screen music (a piano player in a bar) or off-screen music (a full 90-piece orchestra accompanying a love scene). But in VR films, the lines between on-screen and off-screen music have blurred. The on-screen gramophone in the virtual reality storyteller Kismet fills the room with an evocative gypsy violin melody, but when you progress, it enters the backing soundtrack in an orchestral arrangement. Because we are immersed in the world, it’s almost as if the music is “playing inside our head,” linking the scenes to each other with a familiar mnemonic.

How we use music in VR film now must be thought of a little differently, as we are no longer the passive observer but a participant, perhaps even the protagonist. This is a paradigm shift in the storytelling dynamic not experienced since the advent of video games. In VR, the user can make decisions that impact the story and choose where to look to the point they could miss crucial story elements. The challenge here is keeping the audience focused on what we want them to see, and music’s power to draw our attention or subtly influence our behavior can be used to this end.

Game technology will lead the way

Truly the biggest barrier to this happening now is our mindset, because the technology is here and improving every day to enable storytellers to use music to direct the viewer’s gaze. Video game audio engines like Audiokinetic’s Wwise are leading the VR music revolution, and new technology is emerging to push the boundaries of spatial audio to produce a true to life experience.

As the VR film world matures and we are able to guide our own journey through the story, music and sound in VR will take the shape of game music on steroids. The technology we use to make this happen will have to not only adapt to the speed in which a user interacts with the experience and evolve based on the changes in order, but it will have to do this seamlessly and without drawing attention to itself, as if it had been composed for that moment.

We are seeing this in VR horror experiences like VRWERX’s Paranormal Activity, where the user can navigate the experience at their own pace, and sound and music jump up at certain moments to scare, add suspense or misdirect. Here, music is used to lead you in what to do next. Because of learned behaviors from horror films, we know what specific audio cues mean. Sudden silence looking down a dark hallway? A creepy onscreen music box playing a childlike melody? A low ominous tone? I found myself compelled to move through the experience in part driven by music and sound triggers, unlike a traditional horror film where you are hand-held by the director to achieve the desired effect. Music needs to become an invisible puppet master doing the job film editors and directors used to do.

Baby steps

Early VR film, such as Chris Milk’s Clouds Over Sidra, have changed the very nature of storytelling by placing the audience into a series of 360 vignettes in a refugee camp tied together by a voice-over narration and emotional piano soundtrack.

You are compelled to look around in each scene and are thrust into the life of Sidra, a 12-year-old Syrian refugee, and her friends. The voice-over and sound directs your attention and guides you through the story. The music serves as an emotional underscore playing “inside your head,” guiding how you feel and linking the vignettes together in an emotional arc.

Here, the music’s role is to focus your attention on the interactions between the children in these scenes, the faces of the refugees and their lives. You “see” what the director wants you to see as the music plays to a specific side of our emotional bias. In this way, the music gently directs your attention within a 360 environment to the human story and keeps you focused.

Content creators need to work together to implement music into VR

We are being held back from moving VR storytelling forward by an outdated concept of how we use music in film. We need to work together to look at the stories as music and look at the music as stories.

Dynamic music is an option now for storytelling, and content creators need to realize the power of music as a tool to not only enhance the emotion in a story but to shape an audience’s perception of every element of their experience, directing their gaze to the desired story arc, placing them in a time, location and aesthetic, and opening their eyes to the meaning in an often confusing experience.

VR directors working together with interactive music composers and implementors can begin to pull back the veil on what is possible in interactive music design, treating music as the wordless narrator that guides us through virtual worlds and glues the narrative together.

 

This blog was originally published on TechCrunch 

 

 

 

Audiokinetic has launched its second Wwise online certification program: the Wwise-201 Certification, which focuses on interactive music.

Subscribe

 

 

Elias Constantopedos

CTO and Composer

CORD Worldwide

Elias Constantopedos

CTO and Composer

CORD Worldwide

 @elasticsounds

댓글

댓글 달기

이메일 주소는 공개되지 않습니다.

다른 글

Wwise를 사용하여 보다 복잡한 MIDI로 제어되는 샘플 악기만들기

MIDI 기반 음악으로 되돌아가는 데에 관심이 있는 사람들은, PS4의 내장 신디사이저를 이용할 수 없으며(존재하지 않음) PC 사운드카드의 GENERAL MIDI에 의존할 수도...

28.7.2020 - 작성자: 다니엘 벡 (DANIEL BECK)

동적 음악 설계에 관하여 - 제 1부: 설계 분류하기

설계 계기 저는 2015년에 오디오 게임 엔지니어로서 처음 일을 하게 되면서 그 당시 저의 아트 디렉터를 통해 Wwise를 접하게 되었습니다. 그전에 저는 게임 음악을 작곡하는...

7.10.2020 - 작성자: 천종 호우 (Chenzhong Hou)

Wwise로 현실 세계의 상호작용 음악 만들기

저는 몇 년 전부터 상호작용 오디오를 좀 더 깊게 탐구해보기로 했습니다. 제가 하는 작업과 연관된 프로젝트로 만들되, 지루하지 않게 흥미로우면서도 배울 것이 있는 프로젝트를...

24.8.2021 - 작성자: 리사 슈바르츠발트 (Ressa Schwarzwald)

게임 음악은 단순히 그냥 음악이 아니다: 제 1부

게임 음악이란 무엇일까요? 상호작용 음악이란 무엇일까요? 이 질문에 답하기란 생각만큼 그리 간단하지 않습니다. 올리비에 더리비에르(Olivier Derivière)는 이 글을 통해...

20.10.2021 - 작성자: 올리비에 더리비에르 (OLIVIER DERIVIÈRE)

Wwise에서 음악 설계 템플릿을 사용하여 독보적인 사운드 디자인을 만들어내는 방법

Foxface Rabbitfish(폭스페이스 레빗피시)의 작곡가 겸 오디오 디렉터인 가이 휘트모어(Guy Whitmore)는 2019 Wwise 상호작용 음악 심포지움에서 현대...

24.11.2021 - 작성자: 가이 휘트모어 (Guy Whitmore)

Audiokinetic Theater 발표 세션 요약 | GDC 2024

GDC 2024에서 개최된 Audiokinetic Theater에서는 많은 오디오 전문가들을 초대해 소중한 정보들을 나눌 수 있어 매우 영광이었습니다. 첫 번째 영상들이 올라와...

22.5.2025 - 작성자: Audiokinetic (오디오키네틱)

다른 글

Wwise를 사용하여 보다 복잡한 MIDI로 제어되는 샘플 악기만들기

MIDI 기반 음악으로 되돌아가는 데에 관심이 있는 사람들은, PS4의 내장 신디사이저를 이용할 수 없으며(존재하지 않음) PC 사운드카드의 GENERAL MIDI에 의존할 수도...

동적 음악 설계에 관하여 - 제 1부: 설계 분류하기

설계 계기 저는 2015년에 오디오 게임 엔지니어로서 처음 일을 하게 되면서 그 당시 저의 아트 디렉터를 통해 Wwise를 접하게 되었습니다. 그전에 저는 게임 음악을 작곡하는...

Wwise로 현실 세계의 상호작용 음악 만들기

저는 몇 년 전부터 상호작용 오디오를 좀 더 깊게 탐구해보기로 했습니다. 제가 하는 작업과 연관된 프로젝트로 만들되, 지루하지 않게 흥미로우면서도 배울 것이 있는 프로젝트를...