VMM Test Flakiness Master List

[smalis-msft]

I will do my best to keep this list updated with the current status of things:

Description	Status	Example logs
Shutdown IC frequenty doesn't work	Nobody investigating, most tests have moved to pipette instead
Pipette connection disappears (linux guest only?)	Logging has been added, we have hits, investigating	Link
OpenVMM + VBS + Ubuntu attempts an invalid synic MSR write	Alex is poking a hypervisor debugger repro, tests have been disabled	Link
ARM UEFI VMBus appears to have some sort of race condition	Logging has been added, waiting for another hit	Link
OpenVMM PCAT has some sort of race condition	Mitigation added to detect stalls and reboot, but VMs crash after reboot, nobody investigating currently	Link Link 2
TDX VM reports invalid x2apic ids, then drops sint messages, then crashes	Nobody investigating	Link, Link 2
SNP VPs stall out	Nobody investigating	Link
SNP VMs OHCL crash with not implemented: ghcb usage BASE	Known feature gap
TDX and SNP VMs guest bugcheck with MEMORY MANAGEMENT	Nobody investigating	Link Link 2
HyperV TMKs fail, no logs	Nobody investigating	Link

[mattkur]

@smalis-msft , I think I have a recent hit of "pipette connection disappears" here: https://openvmm.dev/test-results/test.html?run=18479962055&job=x64-windows-amd-vmm-tests-logs&test=x86_64__storage__openvmm_openhcl_uefi_x64_ubuntu_2504_server_x64_openhcl_linux_storvsp_dvd_nvme#log-4286

In this case, the handshake started but never completed (at least, I don't see tracing::info!("completed pipette handshake"); in the output).

[smalis-msft]

Yep, looks like it thanks! We'll dig in.

[mattkur]

vmm_tests::tests multiarch::hyperv_openhcl_uefi_aarch64_ubuntu_2404_server_aarch64_memory_validation_gp_heavy is timing out. I'll try a PR that reduces the openhcl log level in case it's a bad interaction of my change (set logs to debug) and reverting the change that sets the com3 log level to error.

RFC: memory/nvme: track if memory blocks are persistent, handle fallback in nvme keepalive save/res…
⁠CI: Try to speed up the loom tests even more · microsoft/openvmm@c9ee049

[smalis-msft]

vmm_tests::tests multiarch::hyperv_openhcl_uefi_aarch64_ubuntu_2404_server_aarch64_memory_validation_gp_heavy is timing out. I'll try a PR that reduces the openhcl log level in case it's a bad interaction of my change (set logs to debug) and reverting the change that sets the com3 log level to error.

RFC: memory/nvme: track if memory blocks are persistent, handle fallback in nvme keepalive save/res… ⁠CI: Try to speed up the loom tests even more · microsoft/openvmm@c9ee049

No need, I'm 99% sure the instance you linked is the pipette connection disappearing issue