Experiment: Retain more blocks in memory

[mediocregopher]

Describe the feature

We should experiment with retaining more blocks in memory prior to flushing them to disk.

Currently we have --engine.persistence-threshold default to 2 blocks, and --engine.memory-block-buffer-target is ignored. This means every 2 blocks we flush all blocks to disk.

Some things worth trying:

• Increasing the persistence threshold to a higher limit, like 6 or 10.

• Re-introducing the memory block buffer target at some intermediate value, like 2. This would mean we always have at least 2 blocks in memory. With this in place we would have to pull reverts from disk significantly less often.

Additional context

No response

[gakonst]

What was the tradeoff? I believe ExEx related and with 3rd party apps that do direct db access? That seems fine as I've said before -- if you want to go MUCH faster, you need to give up direct db access at the tip; I think?

[mediocregopher]

What was the tradeoff? I believe ExEx related and with 3rd party apps that do direct db access? That seems fine as I've said before -- if you want to go MUCH faster, you need to give up direct db access at the tip; I think?

This I wasn't aware of, the main tradeoffs in my mind are that the more blocks in memory the bigger the overlays become, and so the bigger allocations. We can generate the overlays async between blocks, but at some point they'll become so big it'll eat into the subsequent block anyway.