GC_suspend can hang attempting to suspend DllMain thread

[tobil4sk]

It seems that in some cases, the SuspendThread call within GC_suspend may hang and never return if it is trying to suspend the thread that is running DllMain to detach itself:

bdwgc/win32_threads.c

Line 457 in b435e76

if (SuspendThread(t->handle) != (DWORD)-1) {

This seems related to the previously fixed issue: #704. Again I am running bdwgc in the neko vm on Windows with DllMain enabled, with this commit: 5658d91, modified with a few debug print statements.

[ivmai]

Is it 64-bit build?

[tobil4sk]

Yes, that's correct: x86-64

[tobil4sk]

I was previously debugging by adding printf statements, however, I have now run a proper debugger on the program and removed the print statements to avoid their potential side effects. It seems that without modification, gc.dll actually hangs on GC_win32_MessageBoxA, which it attempts to run due to ResumeThread failed in suspend loop. This error happens because it runs ResumeThread on a NULL handle. It looks like between the SuspendThread and the ResumeThread, the thread handle has started closing the thread and set it to NULL.

I'm not sure why MessageBoxA hangs since the message box isn't actually showing, but I guess GC_suspend shouldn't abort on the ResumeThread failure if it is caused by a NULL handle.