Update simulated geth client wrapper & LogPoller tests#13204
Update simulated geth client wrapper & LogPoller tests#13204
Conversation
…lient ( instead of deprecated backends.SimulatedBackendClient )
This test relied on markBlockFinalized() which has been replaced with finalizedBlocksThrough(). Just needed some slight adjustments to keep testing the same thing.
d5f0bb2 to
9be1d30
Compare
|
I see you updated files related to
|
74c13d9 to
81aa9b7
Compare
Add RegisterHeadNumberCallback() to SimulatedBackendClient, so we can trigger an rpc failover event just after reading a particular block, but before the logs get read for that block. This is the race condition that can happen on optimism chain which BackupPoller was designed to address
be613b7 to
aec7ffe
Compare
|
Taking this out of draft status, because the most complex part of it--getting Backup LogPoller test to work--is done and passing. Most of the tests that broke due to not being able to mark blocks finalized any more have also been updated and are passing. Just have a few more to go through, as well as updating the types in some tests. Seems like a good time to start getting some eyes on it. |
There was a problem hiding this comment.
Let's group these with the other geth lines
There was a problem hiding this comment.
Did you mean to update these functions files? Looks like stray make generate effects that can be discarded
There was a problem hiding this comment.
They can probably be discarded, I just figured it's better to have the latest make generate checked in than something out of date.
One reason I re-ran make generate was because GETH_VERSION was still set to the old version in some of them:
-GETH_VERSION: 1.13.8
-forwarder: ../../../contracts/solc/v0.8.19/KeystoneForwarder/KeystoneForwarder.abi ../../../contracts/solc/v0.8.19/KeystoneForwarder/KeystoneForwarder.bin ed9164cfe4619dff824b11df46b66f4c6834b2ca072923f10d9ebc57ce508ed8
-keystone_capability_registry: ../../../contracts/solc/v0.8.19/CapabilityRegistry/CapabilityRegistry.abi ../../../contracts/solc/v0.8.19/CapabilityRegistry/CapabilityRegistry.bin d4e0661491c2adc7f0d7553287c938fb32664b9f07770f6d57ae6511ce9884cd
-ocr3_capability: ../../../contracts/solc/v0.8.19/OCR3Capability/OCR3Capability.abi ../../../contracts/solc/v0.8.19/OCR3Capability/OCR3Capability.bin 9dcbdf55bd5729ba266148da3f17733eb592c871c2108ccca546618628fd9ad2
diff --git a/core/gethwrappers/operatorforwarder/generation/generated-wrapper-dependency-versions-do-not-edit.txt b/core/gethwrappers/operatorforwarder/generation/generated-wrapper-dependency-versions-do-not-edit.txt
index 1569801b3f..a30823b737 100644
--- a/core/gethwrappers/operatorforwarder/generation/generated-wrapper-dependency-versions-do-not-edit.txt
+++ b/core/gethwrappers/operatorforwarder/generation/generated-wrapper-dependency-versions-do-not-edit.txt
@@ -1,4 +1,4 @@
-GETH_VERSION: 1.13.8
+GETH_VERSION: 1.13.14
Should the above changes get backed out?
In the case of KeystoneForwarder, it was not only using the old version of geth, but the abi itself has changed. I can't remember for sure whether that was the main motivation--it might have been causing some test failures but not sure.
Looks like the only difference in functions_allow_list, functions_coordinator, and llo-feeds were the compiled binary itself, so I assume those can be ignored.
There was a problem hiding this comment.
CI already enforces that the latest generate version is used. If you are able to edit generated files without failing a check, then you found something that is not tracked correctly. There is a gap that old generated files which are not being regenerated are not noticed because they don't produce a diff.
There was a problem hiding this comment.
see: #13138
As a follow-up to prevent this from happening: Our CI process does not see these as problematic, since they are committed and do not produce a diff. We could add a step before make generate that removes all the generated mockery files to ensure that one which do not get re-generated produce a diff that fails CI.
a5a60a5 to
fcae628
Compare
2
Outdated
Presumably, the older verison of simulated.Backend would fill in fake timestamps instead of real ones?
The new Simulated Geth made two changes which affected this test 1. The automatic time interval added to each new block is now 1ns instead of 1s 2. AdjustTime() now automatically calls Commit() so it no longer needs to be called aftewards
- Consolidate go-ethereum imports - Remove extra geth-wrapper changes
fcae628 to
1a6e9ff
Compare
|
Present status:
I think the last one was what prompted me initially to run I'm not sure how many of these errors were already in the branch I'm trying to merge into, but seems like it might be about time to merge it and rebase that branch? |
core/gethwrappers/keystone/generation/generated-wrapper-dependency-versions-do-not-edit.txt
Show resolved
Hide resolved
…aside from the // error at the end that's in CI as well.
|
* bump geth v1.13.14; rm hack; fix simulated backend; run make generate * Update simulated geth client wrapper & LogPoller tests (#13204) * Re-run make generate, fix fluxmonitorv2 & ocr2keeper tests * Update SimulatedBackendClient to wrap simulated.Backend & simulated.Client ( instead of deprecated backends.SimulatedBackendClient ) * Update LogPoller helper * Add support for switching rpc clients in simulated geth * Fix TestLogPoller_BackupPollAndSaveLogsSkippingLogsThatAreTooOld This test relied on markBlockFinalized() which has been replaced with finalizedBlocksThrough(). Just needed some slight adjustments to keep testing the same thing. * Fix TestLogPoller_ReorgDeeperThanFinality * Fix Test_PollAndSavePersistsFinalityInBlocks * Fix Test_PollAndQueryFinalizedBlocks * update listener_v2_log_listener_test * Fix chainreader & config poller tests * Update keeper integration tests * Re-run make generate * Update BackupLogPoller test Add RegisterHeadNumberCallback() to SimulatedBackendClient, so we can trigger an rpc failover event just after reading a particular block, but before the logs get read for that block. This is the race condition that can happen on optimism chain which BackupPoller was designed to address * . * Update TestLogPoller_PollAndSaveLogsDeepReorg * Not sure how this was working before Presumably, the older verison of simulated.Backend would fill in fake timestamps instead of real ones? * Update TestLogPoller_PollAndSaveLogs * Update TestLogPoller_Blocktimestamps The new Simulated Geth made two changes which affected this test 1. The automatic time interval added to each new block is now 1ns instead of 1s 2. AdjustTime() now automatically calls Commit() so it no longer needs to be called aftewards * Update TestLogPoller_BackupPollAndSaveLogsWithPollerNotWorking * Address PR review comments - Consolidate go-ethereum imports - Remove extra geth-wrapper changes * Update types in vrf tests * Update keepers, fluxmointor, transmitter,ocr test types * re-generate KeystoneForwarder * Replace optimismMode with chainFamily enum * Update more types GenesisAlloc, GenesisAccount, llo * Fix some more compilation errors (fluxmonitor2, vrf, ocr2, functions) * Fix lint errors, remove unused logpoller test definitions * Run go-generate again on KeystoneForwarder * Re-run "make generate" one more time, this time without any failures aside from the // error at the end that's in CI as well. * Re-generate generation version db for keystone * cleanup * make generate * cleanup * race-free simulated backend commits * race free commits * fix tests by adding commits * bump geth to 1.13.15 * fix more races * Remove inconsistent named param from return signature * insert temporary skips * make generate * skip more tests * skip more tests; fill zeroed timestamp on insert * fix race for commit * core/services/ocr2/plugins/ocr2keeper/evmregistry/v21/logprovider: run log poller * re-enable some tests * consolidate finalization helpers * fix some tests with real timestamps * simplify * core/services/vrf/solidity_cross_tests: raise gas estimate upper bound in TestMeasureRandomnessRequestGasCost * cleanup and patch for geth bug * new geth error; vrf test fixes * fix ccip tests * fix clo tests * more fixes * fix test var name * fix compile error * upgrade to 14.7 * fix lint * lint issues * make generate * fix more tests * skip failing tests * skip tests; linter issues * fix race & lint issues * BCFR-1047 Fix HT Tests (#14807) * Simulated defaults (#14986) * Simulated defaults * Direct request works * Fix config tests * AsyncEthTx * OCR + FM * Keepers * Functions * Bunch of keeper and VRF tests * More vrf tests * Cleanup * Fix a race * core/chains/evm/txmgr: fix TestEthBroadcaster_ProcessUnstartedEthTxs_OptimisticLockingOnEthTx * core/gethwrappers: regenerate * bump geth to 1.14.11 (#15041) * bump geth to 1.14.11 * remove AdjustTime hacks * Try replace in integration tests * Copy replace around * Preserve fork behaviour * generate * tidy * Work around geth, fix timestamp test * Fix docstring * Fix reader test * Linter * Re-enable CCIP deployment tests * Mod + lint * Remove cltest dependency which invokes pgtest init --------- Co-authored-by: connorwstein <connor.stein@mail.mcgill.ca> * Another cltest remove * Go mod tidy * Regen lint and fix LLO test * Enable some more tests * core: prefer simulated backend interface in order to wrap with syncBackend * fix race: cleanup int conversion * lint * lint & fix * lint * lint * unskip and fix * Fix TestIntegration_OCR2_plugins * Fix more OCR tests * update t.Skip() messages * update changeset * Resurrect CCIP tests * Lint * replaces * lint --------- Co-authored-by: Domino Valdano <domino.valdano@smartcontract.com> Co-authored-by: Domino Valdano <2644901+reductionista@users.noreply.github.com> Co-authored-by: AnieeG <anindita.ghosh@smartcontract.com> Co-authored-by: Dmytro Haidashenko <34754799+dhaidashenko@users.noreply.github.com> Co-authored-by: Connor Stein <connor.stein@mail.mcgill.ca>



In this PR:
Update
SimulatedBackendClientto wrap simulated.Backend & simulated.Client ( instead of deprecated backends.SimulatedBackendClient )Add
SetActiveClient()toSimulatedBackendClientto simulate switching from one rpc server to another while running.Add
RegisterHeadByNumberCallbacktoSimulatedBackendCilent, to register a custom callback function to be called each timeHeadByNumber()is called.Add to
SimulateBackendClientability to run in "optimism mode", which behaves slightly different from geth: returns success rather than "block not found" ifFilterLogs()is called with an invalid or unknown block hash.Update BackupPoller test to use the previous 3 features to simulate failing over.to an out-of-sync optimism rpc server during a race condition between pulling a particular block and its corresponding logs. (This is exactly the case which motivated the creation of Backup LogPoller.) This avoids the need to use rawdb, which is no longer accessible in geth.
Replace
markBlockAsFinalized()andmarkBlockAsFinalizedByHash()in LogPoller tests withfinalizeThroughBlock().Refactor logic in LogPoller tests previously using
markBlockAsFinalized(ByHash)to make use offinalizedThroughBlock()instead.Update some tests for new block timestamp behavior in Simulated Geth: automatic time increment between simulated blocks is 1ns now instead of 1s, and Client.AdjustTime() now automatically commits the current block after adjusting its timestamp
Update rest of tests in LogPoller and other packages to use new data types associated with (1.), instead of deprecated types