Merge block_hashes.jsons into block_hashes.bincode

[i1i1]

Closes #241

Depends on #440. But anyway, only last 2 commits are relevant to this pr

[kien-rise]

use reference, just like &bytecodes

[kien-rise]

Maybe the data amount is not much, but this is for the sake of consistency.

Also, .clone is widely considered as anti-pattern when using reference (&) is easy.

[i1i1]

Just in case, InMemoryStorage requires iterator with owned values, so its either block_hashes.iter().cloned() or just block_hashes.clone().

pevm/crates/pevm/src/storage/in_memory.rs

Line 21 in 09bb59a

block_hashes: impl IntoIterator<Item = (u64, B256)>,

[kien-rise]

This means InMemoryStorage should be changed to accept a reference.

See https://github.com/risechain/pevm/pull/245/files#diff-d83915c2d163041919e1ac073fb26aaa65dcc05ebb7ceeaab1b70033ab85d5e8L15

[kien-rise]

The spirit is the same. bytecodes and block_hashes are readonly values. InMemoryStorage only "views" them, not own them.

[kien-rise]

I believe a reference is sufficient. Using Arc will be an overkill. Anyway, it's not bad.

Arc comes with a small performance cost. Therefore, please run a benchmark (using cargo bench and crates/pevm/benches/report.py) to make sure that the performance is not affectted.

[i1i1]

It shouldn't though. It is just a fancy pointer to (AtomicUsize, AtomicUsize, T), so, each time you dereference Arc you just read pointer (but now it is with offset of 2 machine words) 🤷‍♂️

[i1i1]

But my take here is that it would be easier to manage Arc over wrangling (now 2) lifetimes.

[hai-rise]

I'm not a fan of premature Arcs over & but 2 Option<&>s with 2 potentially different lifetimes look pretty unpleasant. So yeah, Arc is justified here (not premature).

[kien-rise]

This https://github.com/risechain/pevm/pull/441/files#r1892852724 is a blocking issue. The rest looks good to me.

[i1i1]

Here are benchmarks:

Before (all logs in pastebin and under spoiler below):

Average: x1.89
Max: x4.07
Min: x0.8

Spoiler warning

Block 13287210(1414 txs, 29990789 gas)
2.842 2.457 1.16

Block 46147(1 txs, 21000 gas)
0.003 0.003 1.0

Block 4864590(195 txs, 7985890 gas)
1.791 0.581 3.08

Block 12159808(180 txs, 12478883 gas)
2.893 3.404 0.85

Block 19932703(143 txs, 10421765 gas)
9.907 7.459 1.33

Block 12300570(687 txs, 14934316 gas)
1.392 1.171 1.19

Block 4330482(237 txs, 6669817 gas)
0.678 0.462 1.47

Block 4369999(22 txs, 6630311 gas)
0.644 0.444 1.45

Block 19716145(341 txs, 29995804 gas)
9.726 4.402 2.21

Block 6196166(108 txs, 7975867 gas)
0.653 0.78 0.84

Block 2179522(222 txs, 4698004 gas)
0.384 0.479 0.8

Block 18988207(186 txs, 12398324 gas)
9.403 6.177 1.52

Block 930196(18 txs, 378000 gas)
0.031 0.031 1.0

Block 19934116(58 txs, 3365857 gas)
1.248 1.248 1.0

Block 10760440(202 txs, 12466618 gas)
3.471 1.528 2.27

Block 17034870(184 txs, 29999074 gas)
7.39 3.316 2.23

Block 19932810(270 txs, 18643597 gas)
5.549 2.613 2.12

Block 14545870(456 txs, 29925884 gas)
9.586 3.143 3.05

Block 16257471(98 txs, 20267875 gas)
8.65 5.692 1.52

Block 19923400(24 txs, 1624049 gas)
0.454 0.453 1.0

Block 16146267(473 txs, 19204593 gas)
5.859 2.377 2.47

Block 19606599(367 txs, 29981684 gas)
16.491 6.95 2.37

Block 116525(83 txs, 2625335 gas)
0.172 0.172 1.0

Block 2674998(16 txs, 1915348 gas)
0.083 0.083 1.0

Block 6137495(60 txs, 7994690 gas)
0.833 0.56 1.49

Block 15538827(823 txs, 29981465 gas)
6.432 2.424 2.65

Block 8889776(330 txs, 9996021 gas)
2.076 1.026 2.02

Block 19932148(227 txs, 14378808 gas)
4.914 2.47 1.99

Block 5891667(380 txs, 7980153 gas)
0.622 0.64 0.97

Block 12244000(133 txs, 12450737 gas)
4.944 3.292 1.5

Block 15537394(80 txs, 29983006 gas)
1.906 1.4 1.36

Block 4370000(97 txs, 6609719 gas)
1.385 1.474 0.94

Block 14383540(722 txs, 30059751 gas)
8.34 3.026 2.76

Block 7279999(122 txs, 7998886 gas)
3.36 0.826 4.07

Block 18426253(147 txs, 18889343 gas)
9.188 6.449 1.42

Block 3356896(176 txs, 4033966 gas)
0.362 0.397 0.91

Block 19498855(241 txs, 29919049 gas)
12.774 6.507 1.96

Block 1796867(49 txs, 3917663 gas)
0.195 0.196 0.99

Block 13217637(1100 txs, 29985362 gas)
5.67 2.345 2.42

Block 12965000(259 txs, 30025257 gas)
16.416 4.744 3.46

Block 18085863(178 txs, 17007666 gas)
5.645 3.158 1.79

Block 11743952(206 txs, 11955916 gas)
8.795 9.483 0.93

Block 17666333(961 txs, 29983414 gas)
10.447 5.696 1.83

Block 19860366(430 txs, 29969358 gas)
10.072 4.102 2.46

Block 5526571(143 txs, 7988261 gas)
1.41 0.745 1.89

Block 2462997(9 txs, 484186 gas)
2.242 2.227 1.01

Block 5283152(150 txs, 7979463 gas)
1.756 0.551 3.19

Block 12243999(205 txs, 12444977 gas)
3.213 1.267 2.54

Block 2675000(15 txs, 1312529 gas)
0.072 0.072 1.0

Block 19426586(127 txs, 15757891 gas)
5.756 2.741 2.1

Block 15274915(1226 txs, 29928443 gas)
4.063 2.317 1.75

Block 12047794(232 txs, 12486404 gas)
2.967 3.316 0.89

Block 14396881(1346 txs, 30020813 gas)
3.276 2.498 1.31

Block 2688148(4 txs, 2725844 gas)
0.12 0.12 1.0

Block 19933612(130 txs, 11236414 gas)
5.42 1.556 3.48

Block 15537393(1 txs, 29991429 gas)
1.447 1.449 1.0

Block 12522062(177 txs, 15028295 gas)
2.091 1.02 2.05

Block 19933122(45 txs, 2056821 gas)
0.432 0.431 1.0

Block 7280000(118 txs, 7992790 gas)
3.273 1.829 1.79

Block 1150000(9 txs, 649041 gas)
0.065 0.066 1.0

Block 19505152(417 txs, 29999872 gas)
10.63 3.34 3.18

Block 19929064(103 txs, 7743849 gas)
2.589 1.458 1.78

Block 19807137(712 txs, 29981386 gas)
14.932 7.847 1.9

Block 19444337(417 txs, 29999800 gas)
11.0 3.552 3.1

Block 17034869(93 txs, 8450250 gas)
2.603 1.127 2.31

Block 8038679(237 txs, 7993635 gas)
1.479 0.767 1.93

Block 9068998(3 txs, 3575534 gas)
0.577 0.577 1.0

Block 19469101(469 txs, 26398517 gas)
11.947 6.041 1.98

Block 12964999(145 txs, 15026712 gas)
7.129 3.885 1.83

Block 19910734(0 txs, 0 gas)
0.001 0.001 1.0

Block 19737292(195 txs, 29999921 gas)
6.679 2.924 2.28

Block 19638737(381 txs, 15932416 gas)
4.55 2.269 2.0

Block 14029313(724 txs, 30074554 gas)
5.317 1.754 3.03

Block 19933597(154 txs, 12788678 gas)
2.912 1.75 1.66

Block 14334629(819 txs, 30135754 gas)
6.898 2.567 2.69

Block 15199017(866 txs, 30028395 gas)
6.172 2.153 2.87

Block 11814555(579 txs, 12494001 gas)
1.092 1.04 1.05

Block 19426587(37 txs, 2633933 gas)
2.348 2.353 1.0

Block 12459406(201 txs, 14994849 gas)
4.936 3.329 1.48

Block 15752489(132 txs, 8242594 gas)
1.984 0.994 2.0

Block 9069000(56 txs, 8762935 gas)
2.805 1.692 1.66

Block 12520364(660 txs, 14989902 gas)
1.894 1.376 1.38

Block 11114732(100 txs, 12450745 gas)
2.459 2.849 0.86

Block 2641321(83 txs, 1917429 gas)
0.18 0.181 1.0

Block 19917570(116 txs, 12889065 gas)
4.137 1.638 2.52

Average: x1.89
Max: x4.07
Min: x0.8

After (pastebin)

Average: x1.88
Max: x4.03
Min: x0.8

Spoiler warning

Block 13287210(1414 txs, 29990789 gas)
2.77 2.457 1.13

Block 46147(1 txs, 21000 gas)
0.003 0.003 1.0

Block 4864590(195 txs, 7985890 gas)
1.778 0.58 3.06

Block 12159808(180 txs, 12478883 gas)
2.878 3.404 0.85

Block 19932703(143 txs, 10421765 gas)
9.887 7.417 1.33

Block 12300570(687 txs, 14934316 gas)
1.384 1.174 1.18

Block 4330482(237 txs, 6669817 gas)
0.667 0.464 1.44

Block 4369999(22 txs, 6630311 gas)
0.645 0.443 1.46

Block 19716145(341 txs, 29995804 gas)
9.736 4.421 2.2

Block 6196166(108 txs, 7975867 gas)
0.649 0.786 0.83

Block 2179522(222 txs, 4698004 gas)
0.376 0.472 0.8

Block 18988207(186 txs, 12398324 gas)
9.341 6.148 1.52

Block 930196(18 txs, 378000 gas)
0.03 0.03 1.0

Block 19934116(58 txs, 3365857 gas)
1.246 1.246 1.0

Block 10760440(202 txs, 12466618 gas)
3.461 1.524 2.27

Block 17034870(184 txs, 29999074 gas)
7.36 3.335 2.21

Block 19932810(270 txs, 18643597 gas)
5.55 2.602 2.13

Block 14545870(456 txs, 29925884 gas)
9.555 3.151 3.03

Block 16257471(98 txs, 20267875 gas)
8.691 5.693 1.53

Block 19923400(24 txs, 1624049 gas)
0.448 0.448 1.0

Block 16146267(473 txs, 19204593 gas)
5.821 2.374 2.45

Block 19606599(367 txs, 29981684 gas)
16.484 6.945 2.37

Block 116525(83 txs, 2625335 gas)
0.168 0.169 1.0

Block 2674998(16 txs, 1915348 gas)
0.083 0.083 1.0

Block 6137495(60 txs, 7994690 gas)
0.83 0.559 1.49

Block 15538827(823 txs, 29981465 gas)
6.462 2.437 2.65

Block 8889776(330 txs, 9996021 gas)
2.069 1.028 2.01

Block 19932148(227 txs, 14378808 gas)
4.876 2.457 1.98

Block 5891667(380 txs, 7980153 gas)
0.607 0.632 0.96

Block 12244000(133 txs, 12450737 gas)
4.965 3.262 1.52

Block 15537394(80 txs, 29983006 gas)
1.882 1.394 1.35

Block 4370000(97 txs, 6609719 gas)
1.381 1.473 0.94

Block 14383540(722 txs, 30059751 gas)
8.369 3.068 2.73

Block 7279999(122 txs, 7998886 gas)
3.331 0.827 4.03

Block 18426253(147 txs, 18889343 gas)
9.264 6.453 1.44

Block 3356896(176 txs, 4033966 gas)
0.355 0.398 0.89

Block 19498855(241 txs, 29919049 gas)
12.742 6.498 1.96

Block 1796867(49 txs, 3917663 gas)
0.194 0.193 1.0

Block 13217637(1100 txs, 29985362 gas)
5.689 2.38 2.39

Block 12965000(259 txs, 30025257 gas)
16.453 4.762 3.46

Block 18085863(178 txs, 17007666 gas)
5.653 3.147 1.8

Block 11743952(206 txs, 11955916 gas)
8.76 9.487 0.92

Block 17666333(961 txs, 29983414 gas)
10.512 5.748 1.83

Block 19860366(430 txs, 29969358 gas)
10.018 4.087 2.45

Block 5526571(143 txs, 7988261 gas)
1.404 0.755 1.86

Block 2462997(9 txs, 484186 gas)
2.214 2.194 1.01

Block 5283152(150 txs, 7979463 gas)
1.751 0.554 3.16

Block 12243999(205 txs, 12444977 gas)
3.203 1.268 2.53

Block 2675000(15 txs, 1312529 gas)
0.071 0.071 1.0

Block 19426586(127 txs, 15757891 gas)
5.748 2.727 2.11

Block 15274915(1226 txs, 29928443 gas)
4.031 2.336 1.73

Block 12047794(232 txs, 12486404 gas)
2.929 3.299 0.89

Block 14396881(1346 txs, 30020813 gas)
3.262 2.429 1.34

Block 2688148(4 txs, 2725844 gas)
0.12 0.12 1.0

Block 19933612(130 txs, 11236414 gas)
5.427 1.561 3.48

Block 15537393(1 txs, 29991429 gas)
1.45 1.452 1.0

Block 12522062(177 txs, 15028295 gas)
2.083 1.026 2.03

Block 19933122(45 txs, 2056821 gas)
0.416 0.416 1.0

Block 7280000(118 txs, 7992790 gas)
3.275 1.826 1.79

Block 1150000(9 txs, 649041 gas)
0.065 0.065 1.0

Block 19505152(417 txs, 29999872 gas)
10.543 3.363 3.14

Block 19929064(103 txs, 7743849 gas)
2.581 1.452 1.78

Block 19807137(712 txs, 29981386 gas)
14.954 7.894 1.89

Block 19444337(417 txs, 29999800 gas)
11.006 3.543 3.11

Block 17034869(93 txs, 8450250 gas)
2.613 1.127 2.32

Block 8038679(237 txs, 7993635 gas)
1.47 0.767 1.92

Block 9068998(3 txs, 3575534 gas)
0.578 0.578 1.0

Block 19469101(469 txs, 26398517 gas)
11.942 6.053 1.97

Block 12964999(145 txs, 15026712 gas)
7.146 3.91 1.83

Block 19910734(0 txs, 0 gas)
0.001 0.001 1.0

Block 19737292(195 txs, 29999921 gas)
6.73 2.924 2.3

Block 19638737(381 txs, 15932416 gas)
4.51 2.263 1.99

Block 14029313(724 txs, 30074554 gas)
5.32 1.783 2.98

Block 19933597(154 txs, 12788678 gas)
2.903 1.749 1.66

Block 14334629(819 txs, 30135754 gas)
6.892 2.617 2.63

Block 15199017(866 txs, 30028395 gas)
6.093 2.171 2.81

Block 11814555(579 txs, 12494001 gas)
1.068 1.045 1.02

Block 19426587(37 txs, 2633933 gas)
2.343 2.343 1.0

Block 12459406(201 txs, 14994849 gas)
4.943 3.295 1.5

Block 15752489(132 txs, 8242594 gas)
1.976 0.993 1.99

Block 9069000(56 txs, 8762935 gas)
2.781 1.666 1.67

Block 12520364(660 txs, 14989902 gas)
1.86 1.368 1.36

Block 11114732(100 txs, 12450745 gas)
2.457 2.837 0.87

Block 2641321(83 txs, 1917429 gas)
0.177 0.177 1.0

Block 19917570(116 txs, 12889065 gas)
4.109 1.645 2.5

Average: x1.88
Max: x4.03
Min: x0.88

Most of the blocks within statistical threshold.

[kien-rise]

Here are benchmarks:

Before (all logs in pastebin):
After (pastebin)

pastebin.com refused to connect.

You should upload the files here instead.

[i1i1]

@kien-rise added reports under spoiler in the same comment. Take a look 🙏

[hai-rise]

Few nits but looks great overall!

I just realized storing the block hashes as JSON is fine as they're not that big, and being JSON makes it easy to manually cross-check with an explorer/RPC when things go wrong.

That said, bincode everything to save Mother Earth is the future here. If we need debuggability we can extend the tests or CLI to auto cross-check the checked-in block hashes with RPC :).

[hai-rise]

Nice!