In this article, we will explore various strategies used in the interactive music systems of Assassin's Creed Shadows. We’ll look at some detailed examples of integration techniques in Audiokinetic's Wwise and some conceptual and aesthetical approaches on dynamic music.

In Shadows, players can have a journey in feudal Japan by playing as Naoe, a deadly shinobi, or Yasuke, a strong samuraï.

I'm Jullian Hoff, composer specialized in electroacoustics and interactive systems and I worked on AC Shadows as music designer.

A subtractive approach to exploration music

Any non-monophonic music is composed from superposition and juxtaposition of various sonic elements. These elements have various spectro-morphologies that can be defined – based on the nature of the sounds – by parameters like: shape over time, harmonicity, spectral balance, granularity of the texture, and so many others that are convenient to summarize as timber and dynamic.

Depending on their unique blend, these sounds can:

• mask each other, where one sound becomes difficult to discern due to the presence of another.
• exist distinctly alongside one another, each contributing its own character to the overall soundscape.
• fuse together, creating something new and different from the sum of its components.

And of course, any possible interpolation of these states, varying through time and dependent on the perception and attention of the listener. 

The Flight, the talented composer duo behind the music for AC Shadows, were extremely generous and trusting by providing their creations as individual tracks. Anyone who has ever been in contact with the individual tracks of a complexly stratified piece of music knows that within these orchestrations, hidden gems can only be revealed to those who have the ability to isolate them. 

These reflections led me to adopt a subtractive approach to create variations of the exploration music. Instead of always adding, this method involves thoughtfully removing elements from the original composition to create distinct listening experiences.

Therefore, each exploration music piece in the game comes with 5 variations that are diverse combinations of their components. These variations are edited using this subtractive approach and each offers a unique listening perspective on the same piece. This minimalist approach also suits the Japanese aesthetic of the game, elegant and refined. Finally, this design choice thoughtfully leaves space for our dynamic sound environments, rich with the sounds of nature and the beings that live within it.

An exploration music and two possible variations

Let’s dive into some of the music systems in Assassin's Creed Shadows.

Hidden in shadows

Light and shadows play an important role in our game. Detectability of the protagonist varies depending on illumination, and skillfully hiding in shadows is key to successful infiltration. In these tense moments, the music plays a vital role in building atmosphere and subtly reflecting Naoe's state of concealment.

With the help of programmer Hugo Provencher, we worked on a system to dynamically impact infiltration music when players are hidden in the shadows. The goal was to create an intuitive connection between the player's visibility & the sonic environment, enhancing the feeling of being truly hidden.

In Wwise, this required several iterations to fine-tune the correct balance. Indeed, if the effect was too obvious, it could be perceived as a bug, as we are not used to such a diegetic relationship. Too subtle, and it became anecdotal. 

Several constraints were defined. This system only affects infiltration music when playing Naoe, and only while indoors. 

The Wwise integration is as follows:

Being in interior sends stealth music to a dedicated aux bus and mutes their output bus

High-shelf and Low-pass cutoff frequencies are driven by player illumination on the aux bus

Early reflections and Dry level are driven by player illumination

Slew rate is a key parameter to fine-tune this type of reactive system

Music of the sixth sense

Naoe possesses a heightened ‘Ninja Sense’, which allows her to perceive nearby enemies. This increased perception is achievable thanks to her sharp auditory acuity and attentive listening. 

In this mode, the game highlights certain sounds, while minimizing others, providing players with crucial information about their surroundings. The music plays a key role in supporting this feature, creating space for these vital cues.

To achieve this, all infiltration and exploration music in the game has been processed using a digital signal processing chain in Reaper. Interestingly, the special "phantasmagoric" visual treatment applied in-game during Ninja Sense directly inspired the design of the following DSP chain:

DSP chain of the cooked “ninja sense” music

Naoe in Ninja Sense

Ninja Sense is integrated at the sub-track level

Danger and assassination opportunities

AC Shadows leaves a lot of room for infiltration, fully embracing the assassin's fantasy embodied by our shinobi, Naoe. But Yasuke can also adopt a predatory approach, especially when equipped with a bow. 

Numerous UI elements – deactivatable in the options – help players perceive threats and out-of-frame opportunities and anticipate their detection by NPCs. The music supports this critical aspect of gameplay. With programmer Philippe Laflamme, we worked on a system that provides an RTPC when an assassinable target is within reach of the various gadgets at our characters' disposal. 

This creates a feeling of danger or opportunity and reinforces the dramatic impact of assassinations. 

Once again, the attack and release times of the RTPC have been adjusted to accelerate its reactivity during moments of increased tension and to slow down its release to maintain tension after the assassination.

Custom RTPC drives LPF and volume on various musics used in a stealth context 

Blending 2D and 3D

Assassin's Creed has a long heritage of diegetic (in-world) music - and Shadows is no exception. Players will have numerous opportunities to listen to traditional music while roaming feudal Japan. 

Several systems allow for a harmonious blend between diegetic music and our score. For instance, during some cinematics, several tracks play in Wwise: some are diegetic and spatialized, while  others are not. Their activation is managed via states to ensure synchronization and coherence with the on-screen action. 

Let's detail another system, simple but elegant, that blends diegetic music with the score from The Flight. While both are incredibly rich, there is a certain tonal homogeneity between these pieces that allows us to layer them without excessive dissonance.

Therefore, when diegetic music plays, exploration music is not completely muted; but instead, affected by it. The score is gradually filtered (HPF), attenuated, and injected into a multichannel reverb to give it a distant, yet enveloping character.

Diegetic musicians’ play outputs RTPC

Diegetic musicians modulate score’s volume and HPF

Diegetic musicians modulate score’s reverberation

Dynamic music system

At the heart of the gameplay music of Shadows stands the Dynamic Music System (DMS), which is schematically represented below:

Schematic representation of the Dynamic Music System (DMS) used in gameplay

This system is designed to create a constantly adapting and responsive musical experience, enhancing tension, action, and immersion based on real-time gameplay. It's structured around four main states provided by the NPCs’ AI: 

• AI UNAWARE: When enemies are not alerted to the player's presence, typically accompanied by ambient, atmospheric music.
• AI SEARCH: As enemies actively look for the player after an alert, often with heightened awareness and seeking musical motifs.
• AI LKP (Last Known Position): When enemies have lost sight of the player but are actively investigating their last known whereabouts, characterized by suspenseful, investigative music, often driven by high percussions. 
• AI FIGHT: During direct combat, with intense, combat-driven themes.

To these core AI states, several additional dynamic systems are layered, including transitions for specific player actions and outcomes.

We've already mentioned the dynamic aspect of stealth music (linked to AI UNAWARE), and Naoe’s ninja sense (which can be overlaid on both UNAWARE and TENSION states). Now, I'd like to delve deeper into the other critical elements that make the DMS so reactive:

Dynamic Threat Level

With programmers Rémi Julien, Alexandru-Gabriel Harapu, and assistant director for Fight, Félix Laliberté, we completely revamped the Threat Level (TL) system for Shadows. 

The design is simple: we observe the level differences between each enemy engaged in combat versus the player and apply a weight to them. The total provides the TL. 

By design, it offers several advantages:

• Below a minimal threshold, combat music does not start, preventing music from playing for a fight we anticipate to be very brief.
• The 3 levels are dynamic and defined by a factor recalculated during combat, depending on enemies joining the battle and those being eliminated. Particularly long combat phases benefit greatly from this dynamism.
• The increase in intensity is quick but the release is slow, to maintain the momentum of the action. 

Here are two calculators used for pre-calibrating the TL during the pre-production phase. The top calculator allows us to compute the minimal number of NPCs of each tier needed to reach a certain threshold, based on variables such as:

• The weights associated with the relative levels of NPCs versus the player
• The threshold values 

The bottom calculator is a simple simulation of different tier NPCs engaged in combat to test various configurations.

(These calculators were simply used during the design phase and the final balance and final values ​​were set using debugging tools in the game engine.)

Spreadsheet used to compute TL during design phase

Because The Flight's music has a good density, it was often easy to create these three levels of intensity. However, I sometimes had to use a few strategies to achieve this. These include:

• Spectral division: The more intensely an instrument is driven, the brighter its timbre and the more pronounced the transients. Observing this basic rule of psychoacoustics, I spectrally divided numerous instrumental layers, particularly the percussion, using 6dB slope high-pass and low-pass filters. The cutoff frequency was determined subjectively by listening, track by track, taking into account the nature of the material and its context.

6dB per octave LPF and HPF

• Duplicating tracks: Duplicating a track and applying a 6dB attenuation to each copy and then distributing them within different threat levels has proven to be an effective solution in some cases, particularly for Trapanese remixes, with less dense orchestration and where spectral division alone was not enough.

Posture attacks

In Shadows, players have the ability to hold attacks, causing characters to enter a stance that can be maintained. This allows them to charge attacks but also to wait for a window of opportunity. This is also part of the samurai fantasy.

With combat designer Gabriel Jobin-Roy, we ensured that the combat music supports this. Holding a stance will transition to a specific version of the music, also sensitive to the TL (two variations spread across the three TL thresholds), and will remain in this state for as long as the stance is held. This allows for moments of dramatic tension or particular suspension.

This feature was the subject of trial and error in the transition times. The transition to the stance is quite slow, so as not to create frequent and rapid changes that could be perceived as bugs. The feature shines most during moments of held stance, particularly during duels.

Yasuke posturing

Special ability

In combat, players can perform special, often devastating, abilities. Each weapon has several abilities that can be unlocked throughout the game.

I worked with combat designers Gabriel Jobin-Roy and Hugo Dansereau to ensure the music supports this. We divided them into two categories.

• For the vast majority, a simple and subtle dynamic mixing system will place more emphasis on sound effects.
• But for a narrower selection of abilities, music plays a major role. In the event of a successful first hit from the ability, the music will transition to a suspenseful element, maintain it throughout the ability, then quickly transition back to where it was suspended. A simple solution to create this kind of effect would have been the use of stingers, but, while more complex, using our system allows the music to be suspended rather than attenuated. This creates a frozen moment in time but also allows for the use of an entry transition as well as an exit transition, for more finesse in the production.

Yasuke using a special ability

Executions

When players perform a stance attack on the last enemy in a fight, it triggers an execution sequence.

With combat designers Gabriel Jobin-Roy and Hugo Danserau, we created a system to support executions with music.

Each weapon has several execution choreographies. By observing the duration of each, I was able to group them into two categories and define average durations that give a satisfactory sense of synchronization for all the animations.

Thus, when fights end with such an attack, the music concludes with specific variations to accentuate the dramatic effect.

Execution endings are triggered as stingers

Each fight music has two possible execution ending

Execution stingers drives an RTPC value

Execution stinger ducks Combat bus with musically synced crossfading

Post-fight

Transitions between combat music and exploration music are always a delicate subject in video games. The music must adapt to this situation because that is its function. If an orchestra were to play in real time to support the action, the score would undoubtedly adapt to emphasize the transition from exploration to combat and vice versa. However, making this transition too obvious lacks elegance. Also, combat sequences often flow seamlessly into one another.

The post-fight is an intermediate state that allowed us to soften this transition. These are edited segments that serve as a link between these pieces of music. They play a few seconds after a fight, synchronized with the other pieces of music, and the transition is done by crossfading.

A blending of fight and exploration tracks used as post-fight

Quest music in an open world

In AC Shadows, players are largely free to explore the world as they wish, regardless of the quests they are currently pursuing. This freedom, while empowering, obviously poses a major challenge: how do you effectively support specific narrative quests with unique  music without interrupting the flow of exploration or combat?

With Technical Design Director Fanny Campagnie, we developed several scripts to play these quest-specific music tracks while still imposing clear control over their activation & deactivation.

First, the quests must be available for the associated music to be ready to play. Then, various conditions determine their start or stop:

• The player's presence within a defined area
• The proximity of NPCs or specific interactive elements
• The completion of sub-objectives
• The triggering of certain scripted elements

Finally, it should be noted that Shadows' quest music also uses DMS, to create an experience consistent with other game contexts and offer the same dynamics as in free-roaming or during the infiltration of hostile zones.

A dynamic quest music structure in Wwise

Acknowledgments

Creating a game of this magnitude is a colossal team effort. In addition to the close collaborators mentioned in the text with whom we worked on the (non-exhaustive) systems described, I would like to mention:

Simon Nicole – Audio Quality Assurance Specialist.

Given the systemic nature of our approach, Simon was invaluable in fine-tuning these systems.

But also:

Greig Newby and Arnaud Libeyre – Audio Director and Associate Audio Director

Jérome Angelot and Bénédicte Ouimet – Music Supervisor and Music Supervisor

Roberto Bender – Audio Programmer

Daniel Sykora – Audio Technical Director

As well as my colleagues on the audio team and the many other people who helped with the creation of Shadows' music systems.