Make indexer skip over any heights older than tail.

[robin-near]

Testing strategy is just to bring up an MPC node and see if we see the "skipping to tail height" message. Other than that not sure how to test.

[codecov]

Codecov Report

Attention: Patch coverage is 0% with 23 lines in your changes missing coverage. Please review.

Project coverage is 69.50%. Comparing base (345116c) to head (b88a99b).

Files with missing lines	Patch %	Lines
chain/indexer/src/streamer/mod.rs	0.00%	14 Missing ⚠️
chain/indexer/src/streamer/fetchers.rs	0.00%	9 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master   #13821      +/-   ##
==========================================
- Coverage   69.51%   69.50%   -0.01%     
==========================================
  Files         960      960              
  Lines      185757   185776      +19     
  Branches   185757   185776      +19     
==========================================
+ Hits       129121   129127       +6     
- Misses      51514    51537      +23     
+ Partials     5122     5112      -10     

Flag	Coverage Δ
pytests	1.37% <0.00%> (-0.01%)	⬇️
pytests-nightly	1.45% <0.00%> (-0.01%)	⬇️
unittests	69.11% <0.00%> (-0.01%)	⬇️
unittests-nightly	69.02% <0.00%> (-0.02%)	⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[wacban]

It's definitely worth trying on MPC nodes but I don't think it's the right thing to do in general case. Indexer users such as ReadRPC would not want to skip any heights. Perhaps you can check the same condition (last_synced_block_height < tail) and panic? If this happens the proper solution would be to reboostrap a node in such a way it has all the necessary history.

[wacban]

Was this the issue the MPC nodes were suffering from?

[robin-near]

@wacban the issue specifically happens if the node is bootstrapped using epoch sync, where all the headers from genesis to the epoch sync boundary are missing. We did not quite figure out why, but somehow the indexer grabbed a height already, when it was still at genesis.

If you're concerned about other users of the indexer, I could add an additional condition that this can only be triggered if the previous height is at genesis... though that adds even more untested code. Hmm.

Are ReadRPC nodes regular nodes that are not archival? And I guess we're looking at a case where the ReadRPC node goes down for more than 5 epochs and then we resync it?

[wacban]

In principal if the user configures the indexer with FromInterruption we should not be skipping any blocks.

Are there any drawbacks to just panicking in this case rather than silently skipping to tail? Presumably MPC nodes should configure their indexer with LatestSynced and in that case it should just start at the final head right?

I believe ReadRPC are regular non-archival nodes. cc @khorolets just in case. The scenario you described is what I was thinking but it can probably happen for other reason too.

[khorolets]

ReadRPC are the old-school web 2.0 apps, and they don't use nearcore indexer directly but consume blocks from services like FastNear Data, NEAR Lake that use Indexers directly.

As for the overall idea of skipping blocks in the Indexer Framework (the nearcore one) I am out of context of the problem you're solving here.

he issue specifically happens if the node is bootstrapped using epoch sync, where all the headers from genesis to the epoch sync boundary are missing. We did not quite figure out why, but somehow the indexer grabbed a height already, when it was still at genesis.

Regarding this, I discourage fixing issues by dramatically change well-known behaviour of fundamental tool like Indexer to solve the issues introduced by new method of node-bootstrapping.

I believe if your issue is sneaky you could handle it on your side with restarting the indexer with FromLatest on certain conditions like

the indexer grabbed a height already, when it was still at genesis

If there is a small chance that Indexer start skipping blocks just because it can it may affect a lot of NEAR ecosystem projects.

[robin-near]

Are there any drawbacks to just panicking in this case rather than silently skipping to tail? Presumably MPC nodes should configure their indexer with LatestSynced and in that case it should just start at the final head right?

We do, but somehow the waiting-for-sync step may "slip" and return synced even when the head is at genesis. It's a bug we don't understand :(

[khorolets]

We do, but somehow the waiting-for-sync step may "slip" and return synced even when the head is at genesis. It's a bug we don't understand :(

I am unsure it will help, but the Indexer Framework checks the node status on the nearcore side through the ViewClient so if it gets the "synced" status the culprit is somewhere in nearcore depth. The Indexer is just a "middleman client" in this process.