Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

create patch from .tar instead of .tar.gz (alternative solution) #105

Merged
merged 16 commits into from
Feb 12, 2024

Conversation

dennisvang
Copy link
Owner

@dennisvang dennisvang commented Feb 9, 2024

This is a simplified alternative to PR #93, with the same basic idea:

The original implementation creates patches from the diff between subsequent gzipped archives (.tar.gz).

This leads to excessive patch sizes, because small changes in the source can lead to large changes in the gzip-compressed archives. As a result, patches become practically useless.

The new solution adopted in this PR is to create patches from the uncompressed .tar files. This leads to much smaller diffs, making patches useful again.

Note that the archives are still compressed using gzip, in order to save bandwidth and storage space, but they are decompressed before patching, and re-compressed after patching.

One drawback is that patch creation, on the repo-side, now takes a lot longer, simply because an uncompressed archive is (much) larger than a compressed one. Also we are more likely to run into memory size limits, see e.g. 1, 2:

bsdiff is quite memory-hungry. It requires max(17n,9n+m)+O(1) bytes of memory, where n is the size of the old file and m is the size of the new file. bspatch requires n+m+O(1) bytes.

bsdiff runs in O((n+m) log n) time

fixes #69

Verifying individual patches inside the loop does tell us which specific patch failed, but in the end we still mark all patches as .failed, so we'd be doing unnecessary work.
It would be cleaner, and easier to test, if we separated the diff/patch and hash/verify steps. However, we need to verify right after patching, but *before* writing the dst file.
Thus, the hash verification step was pulled inside the patch method. Now, for consistency, it makes sense to pull the hash creation into the diff method.
@dennisvang dennisvang changed the title Issue69 alt create patch from .tar instead of .tar.gz (alternative solution) Feb 9, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Large patch sizes
1 participant