Runtime Crash in AkMemCpy()

Question

General Information:

• Crashes on unhandled read violation at nullptr (0x0000000000000000)
• Unreal 4.22
• Wwise 2019.1.1.6977
• Engine Plugin
• Vanilla Third Party DLL's (straight from the download, but we have the source for debugging)
• Occurs unpredictably at runtime while the game is posting events (reliably within 5 minutes of play)

Here's the callstack, I'm fairly certain this is a threading error in the AK engine:

  vcruntime140.dll!00007ff92c4014d0() Unknown

  [Inline Frame] [Redacted].exe!AKPLATFORM::AkMemCpy(void *) Line 351 C++

  [Redacted].exe!Bypass_Native_NChan(AkAudioBuffer * io_pInBuffer, AkAudioBuffer * io_pOutBuffer, unsigned long uRequestedSize, AkInternalPitchState * io_pPitchState) Line 63 C++

  [Redacted].exe!CAkResampler::Execute(AkAudioBuffer * io_pInBuffer, AkAudioBuffer * io_pOutBuffer) Line 372 C++

  [Redacted].exe!CAkVPLPitchNode::ConsumeBuffer(AkVPLState & io_state) Line 176 C++

  [Redacted].exe!CAkLEngine::RunVPL(CAkVPLSrcCbxNode * in_pCbx, AkVPLState & io_state) Line 2244 C++

  [Inline Frame] [Redacted].exe!CAkLEngine::ProcessSources(bool) Line 2554 C++

  [Redacted].exe!CAkLEngine::SoftwarePerform() Line 2325 C++

  [Redacted].exe!CAkLEngine::Perform() Line 805 C++

  [Redacted].exe!CAkAudioMgr::Perform() Line 562 C++

  [Redacted].exe!CAkAudioThread::EventMgrThreadFunc(void * lpParameter) Line 74 C++

  [External Code]

 

 

From what I can discern, the AkMemCpy is copying from a nullptr in the pitch resampler. The buffer which is attempting to be read passes an assert in ConsumeBuffer() that verifies it has a valid pointer, but when the Bypass_Native_NChan() call comes through later the buffer has become nullptr. We have the source code for making some necessary integration changes to the sound engine, but this crash comes from the vanilla third party .dll's which we're testing before re-making our internal changes as we upgrade Unreal and Wwise. Our changes worked fine on UE 4.20 with 2018.1.0.6714 but we need to upgrade to UE 4.22 and our audio team needs features in Wwise 2019. 

I'm going to be tweaking the source and rebuilding this to attempt to keep this from crashing, but any help or knowledge on this issue would be greatly appreciated. 

Let me know if there's any information that I can provide that I missed here. 

Answer 1

After more investigation, it appears that the task queue used to move events from the calling-thread to the audio-thread is no longer fully thread-safe. The issue is that in our implementation we would call functions in the AkAudioDevice from callbacks on the audio thread. The solution is to simply not do this. It was fine to do in Wwise 2018, but the newer version of Wwise 2019 does not handle this and will crash as shown above.

Actionable note:
Do not call Wwise functions from any non-game thread in Unreal. Feel free to use "ensure(IsInGameThread())" to verify your calls are all coming from the correct thread when upgrading to 2019.

Comments

Which third party plugin was this?  Hundreds of other games are calling functions from any threads without issue, daily.  I'm more suspicious of a memory stomp due to the third party plugin itself.

---

As far as I understand, this failure isn't related to a third-party plug in. The crash occurs in the software pipeline in the AkAudioLib which isn't a part of any third-party code, I can confirm that I was capable of reproducing this issue when playing both sounds that were mostly vanilla (non-spatialized sounds with no third-party effects) and heavily modified sounds (spatialized, trash-boxed, reverbed, etc).

The issue may have been explicitly calls from the audio thread, the call that was most to blame for the failure in our implementation was SeekOnEvent which we used to sync time to events in game. I haven't looked closely at any changes in the CAkEvent queue implementation that might have made this operation change between 2018 and 2019.

If I get some time to burn in the near future, I can modify our project to strip all third party plugins out and attempt to reproduce the problem then. If you'd like to test the issue yourself, you can re-implement the worst-offender we had which was to call SeekOnEvent() when the event callback fired with the callback type of AK_Duration.

---

We are facing similar issues on UE4 integration 2019.2.1.7250, is this note still the same with 2019.2.1.7250? Please let us know because we are calling Ak functions from non-game threads for sure... (animation threads -> AK notify)

---

Howdy,
I'm fairy certain the Anim Notify thread is safe since there are vanilla AK notify events that call on that thread. It's likely that only AK's callback thread is one of the few crashing locations. This is easy to test as you'll definitely crash if implemented incorrectly.