“The Times They Are a-Changin’” - Bob Dylan

One of the more tedious things that comes up in game audio design is accounting for variations in time. Sometimes that means making multiple versions of a sound for several specified durations. Other times, a change in spec can require remaking a completed sound to match new timing. Of course, it's possible for both of these to happen - multiple versions of a sound are made, and then multiple versions of a sound need to be remade to accommodate a change. This can require a lot of time and energy to wrangle.

In some basic cases, loop-based audio can help with this. That works when a sound is more or less constant. Tools for less simple sounds are limited: runtime timestretching often has a very small range before spectral artifacts change a sound's character undesirably. In-engine traditional synthesis, too, can offer a degree of time flexibility, yet that approach limits the designer to the internal synth’s sounds and features. All in all, sounds that need progression often end up fixed in their original timing. 

After acquiring Soundseed Grain several years ago for use making vehicle sounds, I began to experiment with other possible uses for it. At some point I realized it could be a powerful solution for decoupling a sound from its original timing, offering temporal flexibility for sounds made with any tools.

I’ve made some examples in Unreal to show this in action. For clarity in the Blueprint screenshots below, I’ve collapsed most non-audio things into macros.

Buildups 

Here are two example videos. One features a balloon inflating. The other shows a fantastical cube that bounces around as it builds up with magic energy and ultimately vaporizes. Sonically, these are very different, but their setups are nearly identical. Each example uses a single Wwise event that leverages Soundseed Grain to conform its content across a variety of durations. (Of note, the “pop” at the end in each design is simply garnish for a sense of completeness. It is not part of the granular design.)

Here’s a glance at the setups for these.

Inflation - Wwise Setup

In Wwise, a source audio file is loaded into Soundseed Grain. It’s a long, steady, evolving sound of a balloon gradually being inflated.

Being a granular synth, Soundseed Grain will chop the audio up into tiny grains of sound, centered around wherever the Position value is pointed.

Varying that Position value flexibly is the secret sauce. Here, it’s driven via a Real Time Parameter Control so that a stream of values sent from the game engine can control the Position setting directly,

That RTPC, “Progression_Balloon,” uses a range of values between 0 and 1 to move through the sound. By sending increasing values along that range over time, the game engine progressively updates the Position value in Soundseed Grain. In doing so, it effectively “scrubs” through the audio file. It can do this across a remarkably flexible range of durations.

A side note about the other setting, ‘Output Level,’ seen here - I wanted more of a build in intensity, so I used the same RTPC to drive a 6dB volume ramp, too.

Inflation - Unreal Setup

An event on the Level Blueprint gets a reference to the Balloon Blueprint and sends it a duration value as it triggers the balloon’s “Inflate” event.

Here’s a look at that Inflate event on the Balloon Blueprint.

The keystone of this Blueprint is a timeline that drives the scale of the balloon mesh, creating the appearance of inflation as the timeline progresses. It is a very simple curve.

Using that same value, the “Progression_Balloon” RTPC can scrub through the sound in tandem with the visual inflation.. (The “Inflate Update” logic has been positioned in its own event below the main logic path for visual clarity.)

Once the timeline reaches the end, its Finished pin will execute. At that point the inflate sound is stopped and the final “pop” sound is played.

That’s all there is to it. Pushing a different duration value when calling the Inflate event changes the rate of the timeline, and the timeline’s stream of output values drive the progress of both the sound and the visual inflation in perfect lockstep.

It’s worth noting that anything in the game that can send progressing values between 0 and 1 can be used for this. It does not have to come from a Timeline.

Cube Predestruct - Layering

The only significant difference between the cube and balloon setups is that the cube uses a blend of two Soundseed Grain layers.

While they share an RTPC for progression through the sound, this layered separation allows different granularization settings in each, to be tweaked for optimal results. Also, the separate layers allow for independent treatment of additional settings in Wwise. Here, the more ethereal “Cube_Grain” layer is set to ramp up steadily, while the knocking/rattling “Cube_Grain_Breakage” layer stays mostly subtle until a dramatic crescendo at the end.

Reload

As a more grounded sound example, here’s a complex gun reload sequence. Even that detailed sound can flex smoothly to match varied animation durations.

The setup on this is different from the others. The game engine side of things is incredibly simple.

Reload - Unreal Setup

The major difference here is that the RTPC, ReloadDuration, is set only once. 

In the previous examples the game engine was sending a continuous stream of changing RTPC values to Wwise in order to progress the audio. In this case a ReloadDuration value is sent just once, and Soundseed Grain will do the heavy lifting of scrubbing through the audio for that duration.

Reload - Wwise Setup

As before, the source sound is set up in Soundseed Grain.

With only a single ReloadDuration value to work with, it will need to be applied in a couple of places. First, modulator 4 is set to Saw+ and assigned to control Soundseed Grain’s Position value. The ReloadDuration RTPC is then used to set modulator 4’s Period. 

This allows modulator 4 to advance the Position value forward at a rate set by the ReloadDuration value. Next, the sound also needs to be stopped at the right time or it may loop.

Stopping is achieved with a modulation envelope mapped to Output Level. This envelope has super short attack/release times, and the Sustain Time value is fed the ReloadDuration RTPC. To make sure it stops at the end, the “Stop playback after release” box on the Modulator settings is ticked.

This approach takes a bit more setup in Wwise, but has the advantage of needing minimal setup in the game engine. Giving credit where credit is due, I’d like to give a shout out to Jeshua Whitaker for this clever idea. 

In Closing

Like anything, this technique has caveats and limits. Here are a few things to consider:

• Stretching things to be too long tends to invite sonic artifacts. In general, better results seem to come from a source file that’s longer than the use duration.
• Too short can sometimes be an issue, too. I think the 3 second reload in the above video is getting a touch into “iffy” territory (the original design was ~6 seconds). If this were more than a demo I’d probably handle the barrel spin separately.
• This tends to work easily for otherworldly sounds. More grounded sounds can be challenging and take more finessing to keep sounding natural.
• Try experimenting. Different settings are sometimes needed to get optimal results from different kinds of source material. Play with attack, duration multiplier, shape settings, using a modulator to add tiny Position modulations on top of the main progression, etc. Explore and tinker!
• Nothing requires the use of whole files. The RTPC can be mapped across a smaller range of Position values within the source file.
• Be careful with Position values at the extreme edges since Soundseed Grain’s playback can wrap around. To keep from wrapping backwards from 0, it can be helpful to use modulators that only add forward (positive) modulation. To keep from wrapping from the source’s end back to the beginning, it can be helpful to map a range of Position values that stop a bit short of 100% coupled with source material that either leaves a tail or levels off near the end, depending on desired results.

That’s it. I hope this helps some people. Hopefully it improves workflows and inspires some cool, creative designs that are resilient in the face of timing changes.

I suspect I’ve only scratched the surface here, and I’m excited to see where people might take this. Got questions? Suggestions? Made something cool and want to share? Feel free to get in touch.