Skip to content

Use Merkle trees in ACE #253

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 57 commits into
base: main
Choose a base branch
from
Open

Use Merkle trees in ACE #253

wants to merge 57 commits into from

Conversation

pct960
Copy link
Contributor

@pct960 pct960 commented Jan 24, 2025

This PR introduces the use of Merkle trees in table-diff with mtree-diff. Furthermore, it also adds several optimisations to regular table-diff to enable it to run on medium-sized tables. To use this feature:

  1. Initialise the Merkle tree objects using:

    ./pgedge ace mtree build <cluster> <schema>.<table> --max-cpu-ratio=1 --analyse=true --recreate-objects=true
    

    --analyse=true can be very time-consuming depending on the size of the table. It's recommended to manually perform an ANALYZE <table> in Postgres and avoid using this flag.

  2. Diff can be performed using:

    ./pgedge ace mtree diff <cluster> <schema>.<table> --max-cpu-ratio=1
    

    Performing an mtree-diff automatically updates the Merkle tree before performing the diff. It's also possible to separately update the trees using:

    ./pgedge ace mtree update <cluster> <schema>.<table> --rebalance=true
    

    --rebalance=true will perform splits and merges of blocks based on changes in the underlying keyspace. It's not necessary to perform rebalancing unless it's essential. The default update operation in mtree-diff takes care of block splits and updates but defers merges. This is to preserve parent-child relationships and avoid costly recursions due to merged blocks.

pct960 added 30 commits January 24, 2025 18:15
* Also minimise critical section in compare_checksums
* Merges happen only when --rebalance=true
is passed
@pct960 pct960 changed the base branch from REL25_01 to main June 5, 2025 15:17
@pct960 pct960 marked this pull request as ready for review June 10, 2025 20:44
@pct960 pct960 requested a review from mmols June 10, 2025 20:45
Copy link
Contributor

@mmols mmols left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Functionality worked well for me - although I exercised it on a very small table. Great work

My feedback is mostly around documentation / presentation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants