Deadlock in UE4 cancelling callbacks during GC

Question

Hi there,

  We are experiencing occasional but serious deadlocks using Wwise (2018.1.5.6835.1218) in UE4 (4.21.2).  

 

Basic Repro (low probability due to threading issues):

1. Have a streaming level with an AkEvent that has a Blueprint callback

2. Unload the streaming level

3. Deadlock during GC

 

Some Notes:

- The game objects on each thread seem to be unrelated

- This seems to be a deadlock on the hash table lock between the AKManualEvent wait during GC that I'm guessing is waiting on the event thread and the Event Thread attempting to dispatch a new callback.

- While we have only seen this during level streaming, it seems like any garbage collection pass is at risk for this deadlock.

 

Callstack for our Game Thread:

[External Code]
[Inline Frame] IllGame.exe!AkManualEvent::Wait() Line 47 C++
IllGame.exe!CAkPlayingMgr::CancelCallbackGameObject(unsigned __int64 in_gameObjectID) Line 120 C++
IllGame.exe!FAkComponentCallbackManager::UnregisterGameObject(unsigned __int64 in_gameObjID) Line 224 C++
[Inline Frame] IllGame.exe!FAkAudioDevice::UnregisterComponent(UAkComponent *) Line 2380 C++
IllGame.exe!UAkComponent::UnregisterGameObject() Line 733 C++
IllGame.exe!UActorComponent::BeginDestroy() Line 469 C++
IllGame.exe!UObject::ConditionalBeginDestroy() Line 954 C++
IllGame.exe!UnhashUnreachableObjects(bool bUseTimeLimit, float TimeLimit) Line 1645 C++
IllGame.exe!CollectGarbageInternal(EObjectFlags KeepFlags, bool bPerformFullPurge) Line 1591 C++
IllGame.exe!TryCollectGarbage(EObjectFlags KeepFlags, bool bPerformFullPurge) Line 1700 C++
IllGame.exe!UEngine::ConditionalCollectGarbage() Line 1230 C++
IllGame.exe!UWorld::Tick(ELevelTick TickType, float DeltaSeconds) Line 1647 C++
IllGame.exe!UGameEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 1365 C++
IllGame.exe!UILLGameEngine::Tick(float DeltaSeconds, bool bIdleMode) Line 194 C++
IllGame.exe!FEngineLoop::Tick() Line 3699 C++
[Inline Frame] IllGame.exe!EngineTick() Line 62 C++
IllGame.exe!GuardedMain(const wchar_t * CmdLine, HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, int nCmdShow) Line 174 C++
IllGame.exe!GuardedMainWrapper(const wchar_t * CmdLine, HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, int nCmdShow) Line 145 C++
IllGame.exe!WinMain(HINSTANCE__ * hInInstance, HINSTANCE__ * hPrevInstance, char * __formal, int nCmdShow) Line 276 C++

Callstack for our AK Event Manager Thread:
[Inline Frame] IllGame.exe!Windows::EnterCriticalSection(Windows::CRITICAL_SECTION *) Line 173 C++
[Inline Frame] IllGame.exe!FWindowsCriticalSection::Lock() Line 50 C++
[Inline Frame] IllGame.exe!FUObjectHashTables::Lock() Line 316 C++
[Inline Frame] IllGame.exe!FHashTableLock::{ctor}(FUObjectHashTables &) Line 343 C++
IllGame.exe!StaticFindObjectFastInternalThreadSafe(FUObjectHashTables & ThreadHash, UClass * ObjectClass, UObject * ObjectPackage, FName ObjectName, bool bExactClass, bool bAnyPackage, EObjectFlags ExcludeFlags, EInternalObjectFlags ExclusiveInternalFlags) 
IllGame.exe!StaticFindObjectFastInternal(UClass * ObjectClass, UObject * ObjectPackage, FName ObjectName, bool bExactClass, bool bAnyPackage, EObjectFlags ExcludeFlags, EInternalObjectFlags ExclusiveInternalFlags) Line 563 C++
IllGame.exe!MakeUniqueObjectName(UObject * Parent, UClass * Class, FName InBaseName) Line 1861 C++
IllGame.exe!StaticAllocateObject(UClass * InClass, UObject * InOuter, FName InName, EObjectFlags InFlags, EInternalObjectFlags InternalSetFlags, bool bCanRecycleSubobjects, bool * bOutRecycledSubobject) Line 2371 C++
IllGame.exe!StaticConstructObject_Internal(UClass * InClass, UObject * InOuter, FName InName, EObjectFlags InFlags, EInternalObjectFlags InternalSetFlags, UObject * InTemplate, bool bCopyTransientsFromClassDefaults, FObjectInstancingGraph * InInstanceGraph, bool bAssumeTemplateIsArchetype) Line 3155 C++
[Inline Frame] IllGame.exe!NewObject(UObject * Outer) Line 1175 C++
IllGame.exe!UAkEventCallbackInfo::Create(AkEventCallbackInfo * AkEventCbInfo) Line 83 C++
IllGame.exe!AkCallbackTypeHelpers::GetBlueprintableCallbackInfo(AkCallbackType CallbackType, AkCallbackInfo * CallbackInfo) Line 42 C++
IllGame.exe!FAkBlueprintDelegateEventCallbackPackage::HandleAction(AkCallbackType in_eType, AkCallbackInfo * in_pCallbackInfo) Line 52 C++
IllGame.exe!FAkComponentCallbackManager::AkComponentCallback(AkCallbackType in_eType, AkCallbackInfo * in_pCallbackInfo) Line 100 C++
IllGame.exe!CAkPlayingMgr::CheckRemovePlayingID(unsigned long in_PlayingID, CAkPlayingMgr::PlayingMgrItem * in_pItem) Line 196 C++
IllGame.exe!CAkPlayingMgr::Remove(unsigned long in_PlayingID, CAkTransportAware * in_pPBI) Line 595 C++
IllGame.exe!CAkPBI::Term(bool __formal) Line 211 C++
[Inline Frame] IllGame.exe!CAkURenderer::DestroyPBI(CAkPBI * in_pPBI) Line 931 C++
IllGame.exe!CAkURenderer::PerformContextNotif() Line 915 C++
IllGame.exe!CAkAudioMgr::Perform() Line 555 C++
IllGame.exe!CAkAudioThread::EventMgrThreadFunc(void * lpParameter) Line 74 C++

I'll be investigating potential solutions here, but I'm wondering if this is a known issue and/or if there is a workaround or fix?

 

Thanks,

 - Chance

 

 

Comments

Deferring UAkCallbackInfo creation until the AsyncTask is run seems like the easiest and most correct solution.  Something like below:

// ILLFONIC CHANGE BEGIN - chance.lyon - Making this thread-safe by deferring UObject creation until the game thread
        /* OLD CODE
        UAkCallbackInfo* CachedAkCallbackInfo = AkCallbackTypeHelpers::GetBlueprintableCallbackInfo(in_eType, in_pCallbackInfo);
        EAkCallbackType BlueprintCallbackType = AkCallbackTypeHelpers::GetBlueprintCallbackTypeFromAkCallbackType(in_eType);
        auto CachedBlueprintCallback = BlueprintCallback;
        AsyncTask(ENamedThreads::GameThread, [CachedAkCallbackInfo, BlueprintCallbackType, CachedBlueprintCallback]()
        {
            CachedBlueprintCallback.ExecuteIfBound(BlueprintCallbackType, CachedAkCallbackInfo);
        });
        */

        AkCallbackInfo* ClonedInfo = AkCallbackTypeHelpers::CloneCallbackinfo(in_eType, in_pCallbackInfo);
        auto CachedBlueprintCallback = BlueprintCallback;

        AsyncTask(ENamedThreads::GameThread, [ClonedInfo, in_eType, CachedBlueprintCallback]()
        {
            EAkCallbackType BlueprintCallbackType = AkCallbackTypeHelpers::GetBlueprintCallbackTypeFromAkCallbackType(in_eType);
            UAkCallbackInfo* CachedAkCallbackInfo = AkCallbackTypeHelpers::GetBlueprintableCallbackInfo(in_eType, ClonedInfo);
            CachedBlueprintCallback.ExecuteIfBound(BlueprintCallbackType, CachedAkCallbackInfo);

            delete ClonedInfo;
// ILLFONIC CHANGE END

---

I am also seeing this issue. Is there an official fix for this?