Geth constantly falling out of sync #465
Description
I'm running the latest version of maru from git with geth 1.16.6 on mainnet. Geth has been fully synced from scratch yesterday.
Once sync had completed, geth started following the chain. It does however not import block by block, but continues to lag behind for a few blocks, then catches up again. This behaviour never stabilized.
What I see on geth is that is imports a couple of blocks (3-15 usually):
Imported new chain segment number=25,359,912 hash=9fd61f..67b62c blocks=3 txs=8 mgas=1.078 elapsed=13.655ms mgasps=78.979 triediffs=9.53MiB triedirty=207.83MiB
For some reason, it seems to receive more beacon updates for the same block:
WARN [11-06|22:03:31.116] Ignoring already known beacon payload number=25,359,912 hash=9fd61f..67b62c age=12s
WARN [11-06|22:03:36.116] Ignoring already known beacon payload number=25,359,912 hash=9fd61f..67b62c age=17s
WARN [11-06|22:03:41.116] Ignoring already known beacon payload number=25,359,912 hash=9fd61f..67b62c age=22s
WARN [11-06|22:03:46.115] Ignoring already known beacon payload number=25,359,912 hash=9fd61f..67b62c age=27s
Then it appears the blocks from ...912 to ...921 are missing, and it restarts syncing:
WARN [11-06|22:03:51.125] Ignoring payload with missing parent number=25,359,922 hash=5d5ec5..711ac3 parent=bfd640..6166b3 reason="chain gapped, head: 25359912, newHead: 25359922"
INFO [11-06|22:03:51.127] Restarting sync cycle reason="chain gapped, head: 25359912, newHead: 25359922"
INFO [11-06|22:03:51.156] Syncing beacon headers downloaded=298 left=0 eta=0s
WARN [11-06|22:03:51.156] Retrieved pivot header from local number=25,359,858 hash=c05bd4..88087b latest=25,359,922 oldest=25,359,913
Which leads to the gap being filled, the blocks imported and everything starts from the top:
INFO [11-06|22:03:51.323] Imported new chain segment number=25,359,922 hash=5d5ec5..711ac3 blocks=10 txs=26 mgas=5.265 elapsed=52.349ms mgasps=100.568 triediffs=9.74MiB triedirty=207.85MiB
On the maru side, except for occasional warnings like this:
[WARN ] 2025-11-06 22:10:06.128 [ELSyncPoller-6193999a-9819-4c32-911d-f1453054edc1] InOrderFanoutSubscriptionManager - empty subscribers list, following data wont be delivered: data=ELSyncStatus.SYNCED
everything looks normal.
I don't understand where these gaps are coming from, as maru doesn't log any errors and continously feeds new heads and I guess this is where the blocks should have been coming from.
Anything I can try, or data to provide, that can help shed some light on this?
Thanks!