Rapid memory leak when ID3D12Debug::EnableDebugLayer used with Address Sanitizer (ASAN)

[BeamRaceMuppet]

When ID3D12Debug::EnableDebugLayer is used with Address Sanitizer (ASAN) a rapid memory leak (over 3GB/minute on my machine) can be observed in many, but not all, D3D12 samples. Affected samples include even the most basic ones:

D3D12HelloTriangle
D3D12HelloTexture

D3D12HelloTriangle hits 10.3GB of Process Memory after running for 3 minutes and usage just keeps growing:

This does not affect all samples though, such as D3D12HelloFrameBuffering, which levels off near 600MB:

I have never observed this problem during years of using D3D11. When ASAN is used in general it is normal to observe an initial process memory growth for perhaps a minute or so but this eventually levels off. ASAN tries to keep around some freed memory for awhile in order to detect use-after-free bugs, but it is not supposed to keep growing unbounded.

To reproduce the issue:

1. Open the D3D12HelloWorld solution.
2. Right click on the D3D12HelloTriangle project, select Properties.
3. Make sure the current Configuration is Debug (top left dropdown).
4. Under C/C++ -> General set Enable Address Sanitizer to Yes.

5. Run the project.

Expected behavior: relatively small amount of memory growth that eventually stops.

Observed behavior: rapid unbounded memory growth, multiple gigabytes in minutes.

Disabling this code that enables the D3D12 debug layer allows ASAN to function normally:

// Load the rendering pipeline dependencies.
void D3D12HelloTriangle::LoadPipeline()
{
    UINT dxgiFactoryFlags = 0;

#if defined(_DEBUG)
    // Enable the debug layer (requires the Graphics Tools "optional feature").
    // NOTE: Enabling the debug layer after device creation will invalidate the active device.
    {
        ComPtr<ID3D12Debug> debugController;
        if (SUCCEEDED(D3D12GetDebugInterface(IID_PPV_ARGS(&debugController))))
        {
            debugController->EnableDebugLayer();

            // Enable additional debug layers.
            dxgiFactoryFlags |= DXGI_CREATE_FACTORY_DEBUG;
        }
    }
#endif
/* ... */
}

[BeamRaceMuppet]

Update: there is a workaround for this issue already hinted at in my previous post. The workaround is to queue frames the way D3D12HelloFrameBuffering.cpp does.

I figured this out by noting that D3D12HelloTriangle.cpp exhibits the issue whereas D3D12HelloFrameBuffering.cpp does not. Why? Compare their source code side-by-side and both are almost identical. Both just draw a triangle. But D3D12HelloFrameBuffering uses an array of command allocators, one per back buffer, whereas D3D12HelloTriangle uses just one command allocator. D3D12HelloFrameBuffering defines two functions called WaitForGpu() and MoveToNextFrame() whereas D3D12HelloTriangle uses just one called WaitForPreviousFrame().

So I just tried modifying one of the samples that has the ASAN memory leak to perform its frame queuing the same way that D3D12HelloFrameBuffering does and then use WaitForGpu() and MoveToNextFrame().

And this worked. (The sample I modified was D3D12HelloTexture.)

So I'm posting this for the sake of any future web searchers for "D3D12 ASAN memory leak" who land here, in case we must wait a long time for a fix. Of course, I still don't know why the leak happens, and I don't know why modifying the leaking samples to queue up frames after the manner of D3D12HelloFrameBuffering.cpp fixes the leak.

[BeamRaceMuppet]

Update 2: the workaround of queueing frames after the manner of D3D12HelloFrameBuffering.cpp is only a partial workaround. I am observing the memory leak happening whenever a full GPU flush occurs. Full GPU flushes typically happen when, for example, swapchain resize needs to happen, so every time I transition between windowed and fullscreen a few MB of memory is leaked.