For dramatic VR to succeed, music must become the director

상호작용 음악 / VR 체험

We’ve all experienced music’s power to change our mood or drum up feelings of nostalgia, fear or excitement. And beyond entertainment, numerous studies have proven that music can inspire action and actually impact change.

Retailers have used music to drive shoppers to higher price points and casinos have played music to encourage more gambling, to name just a couple of examples. It’s this use of music — not as a background mood enhancer, but actually as a director of sorts, that will be critical as virtual reality and augmented reality content creators struggle with how to direct the viewer’s gaze in 360 environments.

Figuring out music’s role will be key to the mainstream adoption of VR dramatic content. But so far music is still being relegated to a background mood feature. This needs to change, and it needs to start with a mind shift in how we think about music.

gettyimages-550766037.jpg

A paradigm shift

The use of music in film historically has been confined to on-screen music (a piano player in a bar) or off-screen music (a full 90-piece orchestra accompanying a love scene). But in VR films, the lines between on-screen and off-screen music have blurred. The on-screen gramophone in the virtual reality storyteller Kismet fills the room with an evocative gypsy violin melody, but when you progress, it enters the backing soundtrack in an orchestral arrangement. Because we are immersed in the world, it’s almost as if the music is “playing inside our head,” linking the scenes to each other with a familiar mnemonic.

How we use music in VR film now must be thought of a little differently, as we are no longer the passive observer but a participant, perhaps even the protagonist. This is a paradigm shift in the storytelling dynamic not experienced since the advent of video games. In VR, the user can make decisions that impact the story and choose where to look to the point they could miss crucial story elements. The challenge here is keeping the audience focused on what we want them to see, and music’s power to draw our attention or subtly influence our behavior can be used to this end.

Game technology will lead the way

Truly the biggest barrier to this happening now is our mindset, because the technology is here and improving every day to enable storytellers to use music to direct the viewer’s gaze. Video game audio engines like Audiokinetic’s Wwise are leading the VR music revolution, and new technology is emerging to push the boundaries of spatial audio to produce a true to life experience.

As the VR film world matures and we are able to guide our own journey through the story, music and sound in VR will take the shape of game music on steroids. The technology we use to make this happen will have to not only adapt to the speed in which a user interacts with the experience and evolve based on the changes in order, but it will have to do this seamlessly and without drawing attention to itself, as if it had been composed for that moment.

We are seeing this in VR horror experiences like VRWERX’s Paranormal Activity, where the user can navigate the experience at their own pace, and sound and music jump up at certain moments to scare, add suspense or misdirect. Here, music is used to lead you in what to do next. Because of learned behaviors from horror films, we know what specific audio cues mean. Sudden silence looking down a dark hallway? A creepy onscreen music box playing a childlike melody? A low ominous tone? I found myself compelled to move through the experience in part driven by music and sound triggers, unlike a traditional horror film where you are hand-held by the director to achieve the desired effect. Music needs to become an invisible puppet master doing the job film editors and directors used to do.

Baby steps

Early VR film, such as Chris Milk’s Clouds Over Sidra, have changed the very nature of storytelling by placing the audience into a series of 360 vignettes in a refugee camp tied together by a voice-over narration and emotional piano soundtrack.

You are compelled to look around in each scene and are thrust into the life of Sidra, a 12-year-old Syrian refugee, and her friends. The voice-over and sound directs your attention and guides you through the story. The music serves as an emotional underscore playing “inside your head,” guiding how you feel and linking the vignettes together in an emotional arc.

Here, the music’s role is to focus your attention on the interactions between the children in these scenes, the faces of the refugees and their lives. You “see” what the director wants you to see as the music plays to a specific side of our emotional bias. In this way, the music gently directs your attention within a 360 environment to the human story and keeps you focused.

Content creators need to work together to implement music into VR

We are being held back from moving VR storytelling forward by an outdated concept of how we use music in film. We need to work together to look at the stories as music and look at the music as stories.

Dynamic music is an option now for storytelling, and content creators need to realize the power of music as a tool to not only enhance the emotion in a story but to shape an audience’s perception of every element of their experience, directing their gaze to the desired story arc, placing them in a time, location and aesthetic, and opening their eyes to the meaning in an often confusing experience.

VR directors working together with interactive music composers and implementors can begin to pull back the veil on what is possible in interactive music design, treating music as the wordless narrator that guides us through virtual worlds and glues the narrative together.

 

This blog was originally published on TechCrunch 

 

 

 

Audiokinetic has launched its second Wwise online certification program: the Wwise-201 Certification, which focuses on interactive music.

Subscribe

 

 

Elias Constantopedos

CTO and Composer

CORD Worldwide

Elias Constantopedos

CTO and Composer

CORD Worldwide

 @elasticsounds

댓글

댓글 달기

이메일 주소는 공개되지 않습니다.

다른 글

하이브리드 상호작용 음악의 시대가 올 것인가? 제 1부 -상호작용 음악의 R&D 플랫폼으로 Get Even 사용하기

저는 게임 음악을 작곡할 때 어떻게 하면 플레이어에게 의미있게 다가갈 수 있을까 항상 고민합니다. 작곡가는 보통 크리에이티브 디렉터, 오디오 디렉터와 함께 이야기, 감정, 주제,...

22.10.2019 - 작성자: 올리비에 더리비에르 (OLIVIER DERIVIÈRE)

Jurassic World: VR Expedition (쥬라기 월드: VR 탐험)

Jurassic World: VR Expedition은 위치 기반의 경험을 제공하는 인터렉티브 시네마틱 가상 현실(VR)로서, 플레이어들을 시각적으로 아주 아름다운 이슬라 누블라의...

18.3.2020 - 작성자: Hexany Audio (헥사니 오디오)

음악 디자인 & "히라에스 프로젝트(Project Hiraeth)": 내가 존재하지 않는 게임을 위해 1년을 보낸 이유

히라에스 프로젝트(Project Hiraeth)는 완전히 어뎁티브하고, 전문적으로 믹싱 되고, 라이브 오케스트라와 함께 녹음된 세계 최초의 비디오 게임 스코어 라이센스입니다....

15.4.2020 - 작성자: Seth Wright

Mystralia의 마법적이고 역동적인 음악 사운드스케이프 만들기

Mages of Mystralia는 주인공 지아(Zia)가 마법의 기술을 배우는 매력적이고 다채로운 액션 어드벤처 게임입니다. Borealys Games의 작곡자이자 사운드...

23.6.2020 - 작성자: 안토이네 바숀(ANTOINE VACHON)

게임 음악은 단순히 그냥 음악이 아니다: 제 1부

게임 음악이란 무엇일까요? 상호작용 음악이란 무엇일까요? 이 질문에 답하기란 생각만큼 그리 간단하지 않습니다. 올리비에 더리비에르(Olivier Derivière)는 이 글을 통해...

20.10.2021 - 작성자: 올리비에 더리비에르 (OLIVIER DERIVIÈRE)

Audiokinetic Theater 발표 세션 요약 | GDC 2024

GDC 2024에서 개최된 Audiokinetic Theater에서는 많은 오디오 전문가들을 초대해 소중한 정보들을 나눌 수 있어 매우 영광이었습니다. 첫 번째 영상들이 올라와...

22.5.2025 - 작성자: Audiokinetic (오디오키네틱)

다른 글

하이브리드 상호작용 음악의 시대가 올 것인가? 제 1부 -상호작용 음악의 R&D 플랫폼으로 Get Even 사용하기

저는 게임 음악을 작곡할 때 어떻게 하면 플레이어에게 의미있게 다가갈 수 있을까 항상 고민합니다. 작곡가는 보통 크리에이티브 디렉터, 오디오 디렉터와 함께 이야기, 감정, 주제,...

Jurassic World: VR Expedition (쥬라기 월드: VR 탐험)

Jurassic World: VR Expedition은 위치 기반의 경험을 제공하는 인터렉티브 시네마틱 가상 현실(VR)로서, 플레이어들을 시각적으로 아주 아름다운 이슬라 누블라의...

음악 디자인 & "히라에스 프로젝트(Project Hiraeth)": 내가 존재하지 않는 게임을 위해 1년을 보낸 이유

히라에스 프로젝트(Project Hiraeth)는 완전히 어뎁티브하고, 전문적으로 믹싱 되고, 라이브 오케스트라와 함께 녹음된 세계 최초의 비디오 게임 스코어 라이센스입니다....