-
Notifications
You must be signed in to change notification settings - Fork 16
Use Merkle trees in ACE #253
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
pct960
wants to merge
57
commits into
main
Choose a base branch
from
ace/merkle-trees
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Also minimise critical section in compare_checksums
* Merges happen only when --rebalance=true is passed
* Prior use of executemany would internally call execute multiple times, thereby making repairs slow because of statement triggers from mtrees
mmols
requested changes
Jun 12, 2025
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Functionality worked well for me - although I exercised it on a very small table. Great work
My feedback is mostly around documentation / presentation.
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR introduces the use of Merkle trees in table-diff with
mtree-diff
. Furthermore, it also adds several optimisations to regulartable-diff
to enable it to run on medium-sized tables. To use this feature:Initialise the Merkle tree objects using:
--analyse=true
can be very time-consuming depending on the size of the table. It's recommended to manually perform anANALYZE <table>
in Postgres and avoid using this flag.Diff can be performed using:
Performing an
mtree-diff
automatically updates the Merkle tree before performing the diff. It's also possible to separately update the trees using:--rebalance=true
will perform splits and merges of blocks based on changes in the underlying keyspace. It's not necessary to perform rebalancing unless it's essential. The default update operation inmtree-diff
takes care of block splits and updates but defers merges. This is to preserve parent-child relationships and avoid costly recursions due to merged blocks.