[ocaml5-issue] Sys stress test may fail or timeout

[jmid]

The revision in #304 of the current Sys tests can either fail or timeout, with the latter being most likely.
This has been observed on both macOS (Intel and ARM64) with 5.3/5.4/trunk and on Linux ARM64 and MSVC with 5.3.

Here's an example of macOS Intel with trunk timing out after 3h:
https://github.com/ocaml-multicore/multicoretests/actions/runs/15207484605/job/42773854908?pr=304

Starting 2-th run

random seed: 531872336
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 1000     0.0s STM Sys test sequential
[ ]    0    0    0    0 / 1000     0.0s STM Sys test sequential (generating)
[✓] 1000    0    0 1000 / 1000     8.8s STM Sys test sequential

[ ]    0    0    0    0 / 1000     0.0s STM Sys stress test parallel
[ ]  142    0    0  142 / 1000    51.6s STM Sys stress test parallel
[ ]  273    0    0  273 / 1000   111.7s STM Sys stress test parallel
[ ]  497    0    0  497 / 1000   171.8s STM Sys stress test parallel
Error: The operation was canceled.

Here's an example of a strange Linux ARM64 failure with 5.3 after 54m (way below the workflow timeout) causing a cancellation,
with little useful info in the CI log:
https://github.com/ocaml-multicore/multicoretests/actions/runs/15207485548/job/42773857490?pr=304

Starting 5-th run

random seed: 28017438
generated error fail pass / total     time test name

[ ]    0    0    0    0 / 1000     0.0s STM Sys test sequential
[ ]    0    0    0    0 / 1000     0.0s STM Sys test sequential (generating)
[✓] 1000    0    0 1000 / 1000     1.5s STM Sys test sequential

[jmid]

I've worked on reproducing this and after some manual shrinking I am now down to the following which triggers consistently on macOS with either an Intel or ARM64 CPU. I was also able to trigger it locally on an Intel-based MacBook Pro:

                                       |                   
                                Mkdir ([], hhh)            
                              Mkfile ([hhh], iii)          
                              Mkdir ([hhh], hhh)           
                                       |                   
                    .------------------------------------.
                    |                                    |                   
         Mkdir ([hhh; iii], eee)          Rename ([hhh; iii], [iii; ccc])    
         Rename ([hhh; hhh], [])                  Rmdir ([], hhh)            
           Rename ([bbb], [])                   Mkdir ([hhh], iii)           
                                                  Rmdir ([], hhh)          

When run, the symptom is that the system locks up and becomes non-responsive - likely a deadlock.
Locally, I've thus not been able to interact with an lldb debugger to understand better what is going on.

Removing Sys.rename generation, this failed to trigger with random test inputs, which indicates that the function is (partly) responsible. I've therefore studied caml_sys_rename which doesn't register its arguments with the GC. AFAICS it should be safe not to do so, as no GC can happen before both potential GC roots are dead (unused).

Interestingly, all the cmds in the parallel legs will raise an exception, which makes me wonder if this is exception related.
The issue could also be timing or scheduling-related, explaining why it primarily shows up on macOS.
This reproducer does not trigger on Linux ARM64 (as mentioned above), though. 🤔