Testing sync

[mempirate]

This is a discussion in which I'll post the findings on trying to sync a node.

Context

I ran the node for 30 minutes trying to sync up to block 16289962 with the following command:

reth node --debug.tip 0xa9137f264e276f1c86a7c048e6fd0ef57e016858fa91da973bb3232cec65c376

During this time, it wasn't able to import any headers from the p2p network. This is the state of the db at the end:

2022-12-29T13:11:15.155032Z  INFO Table CanonicalHeaders has 1 entries (total size: 4 KB)
2022-12-29T13:11:15.155045Z  INFO Table HeaderTD has 1 entries (total size: 4 KB)
2022-12-29T13:11:15.155048Z  INFO Table HeaderNumbers has 1 entries (total size: 4 KB)
2022-12-29T13:11:15.155050Z  INFO Table Headers has 1 entries (total size: 4 KB)
2022-12-29T13:11:15.155052Z  INFO Table BlockBodies has 1 entries (total size: 4 KB)
2022-12-29T13:11:15.155054Z  INFO Table BlockOmmers has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155057Z  INFO Table NonCanonicalTransactions has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155059Z  INFO Table Transactions has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155061Z  INFO Table TxHashNumber has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155063Z  INFO Table Receipts has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155065Z  INFO Table Logs has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155067Z  INFO Table PlainAccountState has 8893 entries (total size: 524 KB)
2022-12-29T13:11:15.155069Z  INFO Table PlainStorageState has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155071Z  INFO Table Bytecodes has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155074Z  INFO Table BlockTransitionIndex has 1 entries (total size: 4 KB)
2022-12-29T13:11:15.155076Z  INFO Table TxTransitionIndex has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155077Z  INFO Table AccountHistory has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155079Z  INFO Table StorageHistory has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155081Z  INFO Table AccountChangeSet has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155083Z  INFO Table StorageChangeSet has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155085Z  INFO Table TxSenders has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155087Z  INFO Table Config has 0 entries (total size: 0 KB)
2022-12-29T13:11:15.155089Z  INFO Table SyncStage has 1 entries (total size: 4 KB)

So only the genesis header was written.

The debug logs can be found in this TG message: https://t.me/paradigm_reth/939

Errors

P2P

Looking at the p2p errors we get the following:

• 1653 disconnects
• 1041 UselessPeer errors (of which 763 on inbound connections)
• 455 TooManyPeers errors
• Then a combination of other ones

It was hard to connect to peers, got up to around 8 at the end, but they were useless since reth wasn't requesting any headers from them.

Header stage

Every once in a while we connected to a peer and started requesting headers, but we never really got any. Usually we got EmptyResponse errors:

2022-12-29T12:23:55.621194Z ERROR execute{stage=Headers}: Unexpected error error=EmptyResponse
2022-12-29T12:23:55.621209Z  WARN execute{stage=Headers}: Stage encountered a non-fatal error: Received empty header response.. Retrying stage=Headers

Or just 0 len headers:

2022-12-29T12:28:52.551734Z  INFO execute{stage=Headers}: Received headers len=0
2022-12-29T12:28:52.551752Z  INFO execute{stage=Headers}: Stage made progress stage=Headers stage_progress=0 done=false

At one point I also got a TheMergeMixHashIsNotZero error:

2022-12-29T12:45:00.122357Z ERROR execute{stage=Headers}: Validation error error=TheMergeMixHashIsNotZero hash=0xfa9152bc3a1c7b6f8386cfd184c8382717261eaa2540d46bc81fbca87c5a660d
2022-12-29T12:45:00.122373Z  WARN execute{stage=Headers}: Stage encountered a validation error: Mix hash after merge is not zero stage=Headers bad_block=0

This originates here:

reth/crates/consensus/src/verification.rs

Lines 60 to 61 in de6630a

	if header.mix_hash != H256::zero() {
	return Err(Error::TheMergeMixHashIsNotZero)

This was probably a bad block, can't find the hash anywhere.

Next Steps

I think it might be useful to try to connect reth to a synced geth instance, which is what I'm going to do next.

[mattsse]

1041 UselessPeer errors (of which 763 on inbound connections)

so you're getting incoming connections? The useless peer is then emitted by the remote peer, because we currently don't send this @Rjected

[mempirate]

Yes I'm getting a lot of them, but it looks like most if not all are broken off by the remote peer. Also want to add that I'm seeing a lot of the following ECIES errors: TagCheckDecryptFailed

[entropidelic]

is there a chance you are setting the debug tip incorrectly?
https://etherscan.io/block/16289962

the hash of block 16289962 is

 0x36ab19341fb0f52f2aad3a7e6f287a38e57335dcf407a4fe15146469da76d666 

[mempirate]

You're right, trying again with the correct hash!

[rakita]

@jonasbostoen thank you for mixHash, this check should be removed: #652

[0xMelkor]

Next Steps

I think it might be useful to try to connect reth to a synced geth instance, which is what I'm going to do next.

I have a synced Geth instance in the cloud. We can pair our nodes if you wish.
Just tell me.

Peace