Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add mult-threaded lzma compression support. #79

Open
jcoffland opened this issue Feb 1, 2022 · 2 comments
Open

Add mult-threaded lzma compression support. #79

jcoffland opened this issue Feb 1, 2022 · 2 comments

Comments

@jcoffland
Copy link
Member

The 7-zip LZMA SDK is public domain and supports multi-threaded compression.

@Artoria2e5
Copy link

Artoria2e5 commented Mar 12, 2024

I recommend looking into the public domain XZ utils liblzma instead, at https://github.com/tukaani-project/xz. The liblzma part of XZ is also public domain. There are more pairs of eyes on that code base, because every Linux distribution uses it. There's also real source control with commit messages.

In your boost dependency, there already is an iostream filter for lzma that works through liblzma. I have no idea what your boost version is, but boost 1.83.0 evidently can use multithreaded lzma in its implementation (source code). The amount of code required should, as a result, be pretty close to what you have for bzip2 right now: a thin wrapper on top of boost.

Having lzma iostream is a really good idea, because it compresses usually better than bzip2 and decompresses usually faster. bzip2 should really not be a thing in the compress-once-then-distribute-widely use case now that lzma exists.


(There used to be some comment about the chunk size here. I misread the un-named struct members; boost's defaults are fine. I hate un-named struct initializations.)

@Artoria2e5
Copy link

Artoria2e5 commented Mar 30, 2024

Well I’ll be damned. Some long-time maintainer just put a big backdoor in the last two versions of xz/liblzma and it’s all over the news. At least boost doesn’t rely on these two specific versions… and MT has been present before those two.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

No branches or pull requests

2 participants