Skip to content

3.5.0

Compare
Choose a tag to compare
@who-biz who-biz released this 21 Nov 23:49
· 19 commits to master since this release
7f530ea

Overview

This release contains 64 commits since the last major release v3.4.0. Many of which are significant improvements to Epic's node server functionality. The most significant of these changes are detailed below.

v3.5.0 includes a dramatic reduction in blockchain synchronization time, as well as effort involved. Previously, syncing a full node from zero took a very long time, and a lot of manual interaction. Users can now sync in a few hours, without intervention.

Approximate Benchmarking:
Prior full sync time: 72hrs, multiple manual restarts needed
New full-sync performance: 1hr 58min, no intervention necessary

This performance improvement is a combination of work done by:

  • @johanneshahn in writing a much more performant replacement for Grin's HeaderSync code, which preserves slowsync backwards compatibility
  • And work by @who-biz toward a significant overhaul of BodySync phase

None of the differentiating code is present in any upstream codebase.

Note: The above benchmark used the "least PoW verification" configuration, on a dedicated Linux server. I.e. skip_pow_validation = true and disable_checkpoints = true. Please see changelog below for disambiguation on what these options do (and what they do not). Total synchronization time will vary, based on hardware specs.

What's Changed

  • Fixes config file settings for peer_max_count by @who-biz (4966bb9)

    • The correct settings of peer_max_inbound/outbound_count are now detailed properly in generated epic-server.toml
  • Resolves compiler warnings by @who-biz (acc157e, 2ab41cf, 3b58407, 88743b1, 4179d5b, 569961e, f1721fd)

  • Adds hard-coded BlockchainCheckpoints, and skip_pow_validation config setting by @who-biz (3163ac8)

    • A checkpoint is present for every 200,000th block in long-term blockchain history, with a final checkpoint at height 2.2M
    • Setting skip_pow_validation = true in server settings will skip pow validation for all blocks prior to latest checkpoint.
    • This setting is toggled false by default.
  • Adds ussage of GitHub actions/badges and CI for PRs, commits to master branch by @who-biz (5c65fa2)

  • Adds disable_checkpoints setting to config options , refactor checkpoint code by @who-biz (bcf0ecc)

    • Has no effect if skip_pow_validation is not set to true (double safety-switch)
    • Allows users who specify true for both settings to skip PoW validation for the entirety of HeaderSync phase
  • Resolves ScrollBase deprecation in cursive crate - commits taken from upstream Grin by @who-biz (17bac4a)

  • Fixes infinite loop for reorg case in FastHeaders logic by @who-biz (b77a741)

    • In cases of reorg, peers send us the last header that matches our history. This case was not handled in code, and resulted in an inability for HeaderSync to progress
    • This case is now handled gracefully
  • Updates server dependencies (tokio, hyper, and many more) by @johanneshahn (d135732, d899850, 65783a3, 236ff25)

  • Fixes cucumber testing suite by @johanneshahn (88120db)

  • General code cleanup following dependency upgrades by @who-biz

  • Adds a dynamic threshold to force full PoW verification during BodySync of last 1000 blocks by @who-biz (cc05f3d, bfb0924)

  • Adds threshold for ignoring new block/header broadcasts while syncing by @who-biz (4f17617)

  • Enables server to switch from BodySync back to HeaderSync phase by @who-biz (59f5036, 9aada0b)

    • Previously, once HeaderSync phase had completed, code did not support switching back to this phase
    • Due to work-intensive PoW, and poor OrphanBlockPool handling (Grin), BodySync phase was extremely slow
    • If many new blocks were found during this period, we could not switch back to HeaderSync to fetch them
    • Logic has now been added to clear OrphanBlockPool and reset all states so that HeaderSync may resume from a clean state
  • Greatly improves handling of OrphanBlockPool, reducing sync time & cpu usage greatly by @who-biz #105

    • OrphanBlockPool is essentially used as an LRUCache during blockchain synchronization, when we receive blocks out-of-order.
    • Previously, check_orphans function had looping logic that would potentially execute 1000+ times, before function exit
    • Due to OrphanBlockPool being a RwLock<HashMap> type - this was blocking reads/writes while queue continued to grow
    • Some of this looping logic is useful, so we have spliced check_orphans into check_orphans_loop and check_orphans
    • check_orphans now checks for only a single orphan at a time, minimizing prohibitive locking
    • When OrphanBlockPool grows large, we call the looping function to quickly depopulate it, with an early break to prevent problematic locks
    • Finally, MAX_ORPHANS_COUNT was decreased to 60 - again, to minimize prohibitive locking
  • Improves efficiency of peer comms, with less spammy block requests (0991826, 6603488)

  • Version bump to 3.5.0 by @who-biz (7f530ea)


Note: More information on the BodySync improvements is available here: #105
More information on HeaderFastSync code can be found in commits located here: #89

Full Changelog: v3.4.0...v3.5.0