Shomei state corruption on OOM

[Filter94]

Description

After failing OOM, Shomei may get its state corrupted and get stuck without being able to import blocks after a restart

Steps to Reproduce (Bug)

• Spin up Shomei with any non-empty chain. Set a 500 MB memory limit for it
• Produce some blocks (in our case it was around 8k)
• Given that much memory it's highly likely to fail OOM
• After a Shomei spinup, it may fail with invalid trie node... in the logs

Expected behavior: [What you expect to happen]
Shomei should be able to at least import blocks when it's given enough memory after failing with OOM

Actual behavior: [What actually happens]
Shomei gets stuck after failing OOM and doesn't import blocks even if you give it enough memory

Frequency: [How regularly does it occur?]
Sometimes

Versions (Add all that apply)

• Software version: ?
• Java version: 21.0.3
• OS Name & Version: ?
• Docker Version: ?
• Cloud VM, type, size: ?

[garyschulte]

I am presuming this occurs in kubernetes specifically.

How was the 500mb limit imposed, was there a limit on the heap memory, or only on the container itself? Can you confirm what the max heap size is in this config.

Is the process killed by k8s oomkiller or does it get an out of memory for heap space and is recycled by liveness/readiness?

[Filter94]

It's not K8s, but it's some kind of containerized system that kills the process if memory quota is exceeded. So it wasn't set via -Xmx