Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

lzbench 2.0 - Textual Benchmarking of 6 corpora (2MB, 4MB, 15MB, 20MB, 42MB, 320MB) #187

Open
Sanmayce opened this issue Feb 17, 2025 · 11 comments
Open

lzbench 2.0 - Textual Benchmarking of 6 corpora (2MB, 4MB, 15MB, 20MB, 42MB, 320MB) #187

Sanmayce opened this issue Feb 17, 2025 · 11 comments

Comments

@Sanmayce
Copy link

Sanmayce commented Feb 17, 2025 •
edited
Loading

Testmachine: ThinkPad L490, i7-8565U, DDR4 2400 MHz
OS: Linux Fedora 39
Mode: Performance Mode, as superuser
Compiler: gcc version 13.2.1 20231205 (Red Hat 13.2.1-6) (GCC)

lzbench 2.0 (64-bit Linux)  Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
...
done... (cIters=1 dIters=37 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)

The results sorted by column number 4:
Compressor name         Compress. Decompress. Compr. size  Ratio Filename
kanzi 2.3 -9             1.49 MB/s  1.51 MB/s      420296  21.44 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
bsc 3.3.4 -1             19.9 MB/s  37.0 MB/s      432428  22.06 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
bsc 3.3.4 -6             19.5 MB/s  17.2 MB/s      440908  22.49 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
bzip2 1.0.8 -9           14.8 MB/s  34.7 MB/s      499739  25.49 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
bsc 3.3.4 -3             31.0 MB/s  22.8 MB/s      528932  26.98 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
brotli 1.1.0 -11         0.59 MB/s   517 MB/s      534261  27.25 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lzma 24.09 -9            4.17 MB/s  88.5 MB/s      539694  27.53 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
zstd 1.5.6 -22           2.69 MB/s  1091 MB/s      544017  27.75 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
zstd24LDM 1.5.6 -22      2.69 MB/s  1167 MB/s      544017  27.75 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
tornado 0.6a -16         2.66 MB/s   235 MB/s      547470  27.93 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
crush 1.0 -2             0.55 MB/s   365 MB/s      622886  31.77 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lizard 2.1 -49           2.56 MB/s  1017 MB/s      627818  32.02 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
brieflz 1.3.0 -9         3.26 MB/s   239 MB/s      643037  32.80 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
libdeflate 1.23 -12      5.07 MB/s   932 MB/s      659879  33.66 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lizard 2.1 -39           5.98 MB/s  1577 MB/s      671872  34.27 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lzsse2 2019-04-18 -17    7.56 MB/s  4640 MB/s      691119  35.25 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
zlib 1.3.1 -9            12.1 MB/s   332 MB/s      692283  35.31 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
nakamichi okamigan       0.00 MB/s  1766 MB/s      713253  36.38 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lizard 2.1 -29           2.76 MB/s  1480 MB/s      747500  38.13 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lzav 4.5 -2              46.3 MB/s  1859 MB/s      769116  39.23 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lz4hc 1.10.0 -12         11.7 MB/s  2799 MB/s      781619  39.87 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lizard 2.1 -19           6.24 MB/s  2266 MB/s      782179  39.90 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lzo1x 2.10 -999          6.25 MB/s   430 MB/s      790025  40.30 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
kanzi 2.3 -1              121 MB/s   627 MB/s      813848  41.51 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lzav 4.5 -1               244 MB/s  1609 MB/s      905276  46.18 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
quicklz 1.5.0 -3         39.0 MB/s   672 MB/s      910339  46.43 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
density 0.14.2 -3         415 MB/s   485 MB/s      958175  48.88 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lzf 3.6 -1                263 MB/s   475 MB/s     1074264  54.80 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
yappy 2014-03-22 -99     99.5 MB/s  1717 MB/s     1087315  55.46 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
fastlz 0.5.0 -2           254 MB/s   572 MB/s     1093325  55.77 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
snappy 1.2.1              265 MB/s   835 MB/s     1128890  57.58 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
lz4 1.10.0                403 MB/s  3149 MB/s     1150925  58.71 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt
zstd_fast 1.5.6 --5       405 MB/s  1547 MB/s     1227042  62.59 KAZE_stuff/HERCULE_POIROT_-_THE_COMPLETE_SHORT_STORIES_-_Agatha_Christie.txt

TOP 3 Pareto frontiers (compressors for which no other compressors BOTH compress smaller and decompress faster):
zstd 1.5.6 -22
lizard 2.1 -39
lzsse2 2019-04-18 -17

'Wolfeye' is 1766/1091=1.61x faster-but-weaker than zstd.

lzbench 2.0 (64-bit Linux)  Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
...
done... (cIters=1 dIters=37 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)

The results sorted by column number 4:
Compressor name         Compress. Decompress. Compr. size  Ratio Filename
kanzi 2.3 -9             2.56 MB/s  2.61 MB/s      752464  16.12 KAZE_stuff/KJV_Bible_pg10-h.html
bsc 3.3.4 -1             19.3 MB/s  32.9 MB/s      848448  18.18 KAZE_stuff/KJV_Bible_pg10-h.html
bsc 3.3.4 -6             26.7 MB/s  15.1 MB/s      866782  18.57 KAZE_stuff/KJV_Bible_pg10-h.html
bzip2 1.0.8 -9           14.5 MB/s  38.0 MB/s      996250  21.35 KAZE_stuff/KJV_Bible_pg10-h.html
bsc 3.3.4 -3             32.0 MB/s  22.8 MB/s     1040224  22.29 KAZE_stuff/KJV_Bible_pg10-h.html
lzma 24.09 -9            3.31 MB/s  94.3 MB/s     1055025  22.61 KAZE_stuff/KJV_Bible_pg10-h.html
brotli 1.1.0 -11         0.55 MB/s   600 MB/s     1056053  22.63 KAZE_stuff/KJV_Bible_pg10-h.html
zstd24LDM 1.5.6 -22      2.11 MB/s  1077 MB/s     1070683  22.94 KAZE_stuff/KJV_Bible_pg10-h.html
zstd 1.5.6 -22           2.15 MB/s  1150 MB/s     1070683  22.94 KAZE_stuff/KJV_Bible_pg10-h.html
tornado 0.6a -16         2.20 MB/s   285 MB/s     1074429  23.02 KAZE_stuff/KJV_Bible_pg10-h.html
crush 1.0 -2             0.28 MB/s   398 MB/s     1246547  26.71 KAZE_stuff/KJV_Bible_pg10-h.html
lizard 2.1 -49           2.03 MB/s   984 MB/s     1246641  26.71 KAZE_stuff/KJV_Bible_pg10-h.html
brieflz 1.3.0 -9         2.57 MB/s   258 MB/s     1260077  27.00 KAZE_stuff/KJV_Bible_pg10-h.html
libdeflate 1.23 -12      5.10 MB/s  1118 MB/s     1323850  28.37 KAZE_stuff/KJV_Bible_pg10-h.html
lizard 2.1 -39           5.17 MB/s  1475 MB/s     1370691  29.37 KAZE_stuff/KJV_Bible_pg10-h.html
nakamichi okamigan       0.00 MB/s  1968 MB/s     1390376  29.79 KAZE_stuff/KJV_Bible_pg10-h.html
lzsse2 2019-04-18 -17    6.98 MB/s  4944 MB/s     1399447  29.99 KAZE_stuff/KJV_Bible_pg10-h.html
zlib 1.3.1 -9            10.5 MB/s   317 MB/s     1400530  30.01 KAZE_stuff/KJV_Bible_pg10-h.html
lizard 2.1 -29           2.14 MB/s  1373 MB/s     1467043  31.43 KAZE_stuff/KJV_Bible_pg10-h.html
lzav 4.5 -2              47.7 MB/s  2107 MB/s     1555765  33.33 KAZE_stuff/KJV_Bible_pg10-h.html
lz4hc 1.10.0 -12         9.23 MB/s  2688 MB/s     1578351  33.82 KAZE_stuff/KJV_Bible_pg10-h.html
lizard 2.1 -19           5.41 MB/s  2071 MB/s     1579730  33.85 KAZE_stuff/KJV_Bible_pg10-h.html
lzo1x 2.10 -999          5.28 MB/s   485 MB/s     1601128  34.31 KAZE_stuff/KJV_Bible_pg10-h.html
kanzi 2.3 -1              132 MB/s   831 MB/s     1626388  34.85 KAZE_stuff/KJV_Bible_pg10-h.html
quicklz 1.5.0 -3         45.3 MB/s   706 MB/s     1788136  38.31 KAZE_stuff/KJV_Bible_pg10-h.html
lzav 4.5 -1               262 MB/s  1768 MB/s     1838819  39.40 KAZE_stuff/KJV_Bible_pg10-h.html
density 0.14.2 -3         372 MB/s   415 MB/s     1964749  42.10 KAZE_stuff/KJV_Bible_pg10-h.html
yappy 2014-03-22 -99      112 MB/s  1800 MB/s     2134663  45.74 KAZE_stuff/KJV_Bible_pg10-h.html
lzf 3.6 -1                295 MB/s   504 MB/s     2225799  47.69 KAZE_stuff/KJV_Bible_pg10-h.html
fastlz 0.5.0 -2           244 MB/s   548 MB/s     2249135  48.19 KAZE_stuff/KJV_Bible_pg10-h.html
snappy 1.2.1              293 MB/s  1003 MB/s     2285979  48.98 KAZE_stuff/KJV_Bible_pg10-h.html
lz4 1.10.0                420 MB/s  2885 MB/s     2292717  49.12 KAZE_stuff/KJV_Bible_pg10-h.html
zstd_fast 1.5.6 --5       328 MB/s  1247 MB/s     2293569  49.14 KAZE_stuff/KJV_Bible_pg10-h.html

TOP 3 Pareto frontiers (compressors for which no other compressors BOTH compress smaller and decompress faster):
zstd 1.5.6 -22
nakamichi okamigan
lzsse2 2019-04-18 -17

'Wolfeye' is 1968/1150=1.71x faster-but-weaker than zstd.

lzbench 2.0 (64-bit Linux)  Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
...
done... (cIters=1 dIters=37 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)

The results sorted by column number 4:
Compressor name         Compress. Decompress. Compr. size  Ratio Filename
kanzi 2.3 -9             2.48 MB/s  2.46 MB/s     3423405  22.15 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
bsc 3.3.4 -1             13.9 MB/s  26.1 MB/s     3610384  23.36 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
bsc 3.3.4 -6             23.2 MB/s  9.00 MB/s     3775012  24.43 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lzma 24.09 -9            2.20 MB/s  81.0 MB/s     4336028  28.06 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
zstd 1.5.6 -22           1.60 MB/s   650 MB/s     4404672  28.51 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
zstd24LDM 1.5.6 -22      1.58 MB/s   600 MB/s     4404704  28.51 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
tornado 0.6a -16         1.63 MB/s   198 MB/s     4410641  28.54 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
brotli 1.1.0 -11         0.48 MB/s   417 MB/s     4449854  28.80 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
bsc 3.3.4 -3             24.1 MB/s  15.6 MB/s     4641848  30.04 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
bzip2 1.0.8 -9           13.6 MB/s  30.0 MB/s     4655679  30.13 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lizard 2.1 -49           1.61 MB/s   784 MB/s     5165800  33.43 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
crush 1.0 -2             0.25 MB/s   334 MB/s     5278331  34.16 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
brieflz 1.3.0 -9         1.70 MB/s   205 MB/s     5347424  34.61 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
nakamichi okamigan       0.00 MB/s  1267 MB/s     5676106  36.73 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
libdeflate 1.23 -12      5.50 MB/s   756 MB/s     5763888  37.30 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lizard 2.1 -39           6.23 MB/s  1378 MB/s     5968503  38.63 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
zlib 1.3.1 -9            9.93 MB/s   239 MB/s     6010105  38.89 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lzsse2 2019-04-18 -17    6.97 MB/s  3543 MB/s     6130438  39.67 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lizard 2.1 -29           1.69 MB/s  1177 MB/s     6144922  39.77 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lzav 4.5 -2              34.6 MB/s  1319 MB/s     6392070  41.37 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lzo1x 2.10 -999          5.45 MB/s   382 MB/s     6803187  44.03 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lz4hc 1.10.0 -12         10.9 MB/s  2412 MB/s     6823104  44.16 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lizard 2.1 -19           6.37 MB/s  2050 MB/s     6827559  44.19 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
kanzi 2.3 -1              264 MB/s  1581 MB/s     7054356  45.65 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
density 0.14.2 -3         302 MB/s   358 MB/s     7678956  49.70 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lzav 4.5 -1               187 MB/s  1247 MB/s     7689515  49.76 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
quicklz 1.5.0 -3         36.8 MB/s   635 MB/s     7767555  50.27 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lzf 3.6 -1                209 MB/s   400 MB/s     9169379  59.34 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
fastlz 0.5.0 -2           183 MB/s   433 MB/s     9317027  60.30 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
yappy 2014-03-22 -99     82.2 MB/s  1387 MB/s     9463310  61.24 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
snappy 1.2.1              245 MB/s   729 MB/s     9664948  62.55 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
lz4 1.10.0                356 MB/s  2451 MB/s     9958632  64.45 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar
zstd_fast 1.5.6 --5       348 MB/s  1294 MB/s    10626403  68.77 KAZE_stuff/The_Book_of_The_Thousand_Nights_and_a_Night_17-TXTs-volumes.tar

TOP 3 Pareto frontiers (compressors for which no other compressors BOTH compress smaller and decompress faster):
zstd 1.5.6 -22
nakamichi okamigan
lzsse2 2019-04-18 -17

'Wolfeye' is 1267/650=1.94x faster-but-weaker than zstd.

lzbench 2.0 (64-bit Linux)  Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
...
done... (cIters=1 dIters=37 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)

The results sorted by column number 4:
Compressor name         Compress. Decompress. Compr. size  Ratio Filename
kanzi 2.3 -9             3.21 MB/s  3.22 MB/s     1560996   7.17 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
bsc 3.3.4 -1             21.0 MB/s  44.5 MB/s     2019624   9.28 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
bsc 3.3.4 -6             41.7 MB/s  17.3 MB/s     2234158  10.26 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lzma 24.09 -9            3.95 MB/s   202 MB/s     2266970  10.41 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
zstd 1.5.6 -22           1.25 MB/s  1481 MB/s     2305317  10.59 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
zstd24LDM 1.5.6 -22      1.26 MB/s  1400 MB/s     2310528  10.61 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
tornado 0.6a -16         1.64 MB/s   467 MB/s     2337296  10.74 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
brotli 1.1.0 -11         0.52 MB/s   869 MB/s     2342384  10.76 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
brieflz 1.3.0 -9         2.49 MB/s   525 MB/s     2748920  12.63 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lizard 2.1 -49           1.06 MB/s  1653 MB/s     2818515  12.95 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
bzip2 1.0.8 -9           11.0 MB/s  45.3 MB/s     2846477  13.08 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
crush 1.0 -2             0.65 MB/s   707 MB/s     2994355  13.75 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
bsc 3.3.4 -3             41.7 MB/s  25.3 MB/s     3038902  13.96 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lizard 2.1 -29           1.20 MB/s  2176 MB/s     3188688  14.65 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lzav 4.5 -2              75.4 MB/s  2339 MB/s     3684945  16.93 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
nakamichi okamigan       0.00 MB/s  2532 MB/s     3728452  17.13 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
libdeflate 1.23 -12      3.67 MB/s  1468 MB/s     3778110  17.35 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
zlib 1.3.1 -9            26.8 MB/s   480 MB/s     3960632  18.19 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lizard 2.1 -39           3.04 MB/s  2048 MB/s     3991118  18.33 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
kanzi 2.3 -1              286 MB/s  1622 MB/s     4065665  18.68 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lzsse2 2019-04-18 -17    4.99 MB/s  5410 MB/s     4203558  19.31 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lzo1x 2.10 -999          12.2 MB/s   718 MB/s     4437900  20.39 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lz4hc 1.10.0 -12         17.5 MB/s  3446 MB/s     4483998  20.60 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lizard 2.1 -19           3.38 MB/s  2866 MB/s     4493325  20.64 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lzav 4.5 -1               416 MB/s  2132 MB/s     4571436  21.00 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
quicklz 1.5.0 -3         65.5 MB/s  1086 MB/s     4804643  22.07 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
zstd_fast 1.5.6 --5       591 MB/s  1897 MB/s     6019295  27.65 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lz4 1.10.0                580 MB/s  2830 MB/s     6159719  28.29 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
fastlz 0.5.0 -2           333 MB/s   807 MB/s     6228050  28.61 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
density 0.14.2 -3         318 MB/s   350 MB/s     6233394  28.63 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
snappy 1.2.1              432 MB/s  1302 MB/s     6670113  30.64 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
lzf 3.6 -1                352 MB/s   657 MB/s     6701600  30.78 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar
yappy 2014-03-22 -99      130 MB/s  2093 MB/s     6970595  32.02 KAZE_stuff/Theravada-Buddhism_SuttaCentral_11-HTMLs_utf-8.tar

TOP 3 Pareto frontiers (compressors for which no other compressors BOTH compress smaller and decompress faster):
zstd 1.5.6 -22
nakamichi okamigan
lzsse2 2019-04-18 -17

'Wolfeye' is 2532/1481=1.70x faster-but-weaker than zstd.

lzbench 2.0 (64-bit Linux)  Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
...
done... (cIters=1 dIters=37 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)

The results sorted by column number 4:
Compressor name         Compress. Decompress. Compr. size  Ratio Filename
kanzi 2.3 -9             3.00 MB/s  3.02 MB/s     5237128  12.32 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
bsc 3.3.4 -1             15.6 MB/s  33.6 MB/s     5928776  13.95 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
bsc 3.3.4 -6             29.8 MB/s  11.3 MB/s     6320618  14.87 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lzma 24.09 -9            2.56 MB/s   114 MB/s     7476550  17.59 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
zstd 1.5.6 -22           1.41 MB/s   877 MB/s     7615509  17.92 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
tornado 0.6a -16         1.55 MB/s   270 MB/s     7720074  18.16 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
zstd24LDM 1.5.6 -22      1.59 MB/s   799 MB/s     7811781  18.38 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
brotli 1.1.0 -11         0.52 MB/s   533 MB/s     7882510  18.55 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
bzip2 1.0.8 -9           11.5 MB/s  37.9 MB/s     7977286  18.77 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
bsc 3.3.4 -3             34.5 MB/s  18.8 MB/s     8137992  19.15 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
brieflz 1.3.0 -9         1.86 MB/s   263 MB/s     9222408  21.70 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lizard 2.1 -49           1.41 MB/s   878 MB/s     9293126  21.87 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
crush 1.0 -2             0.37 MB/s   386 MB/s     9606347  22.60 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
nakamichi okamigan       0.00 MB/s  1347 MB/s    10230161  24.07 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lizard 2.1 -29           1.53 MB/s  1218 MB/s    10755016  25.31 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
libdeflate 1.23 -12      4.28 MB/s  1011 MB/s    10813129  25.44 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
zlib 1.3.1 -9            18.8 MB/s   319 MB/s    11327427  26.65 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lizard 2.1 -39           4.73 MB/s  1495 MB/s    11620963  27.34 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lzsse2 2019-04-18 -17    7.19 MB/s  4381 MB/s    11688012  27.50 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lzav 4.5 -2              39.8 MB/s  1492 MB/s    11862438  27.91 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lzo1x 2.10 -999          8.97 MB/s   504 MB/s    12739904  29.98 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lz4hc 1.10.0 -12         13.6 MB/s  2608 MB/s    12909873  30.38 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lizard 2.1 -19           5.16 MB/s  2069 MB/s    12922459  30.41 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
kanzi 2.3 -1              263 MB/s  1277 MB/s    13168974  30.99 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
quicklz 1.5.0 -3         50.0 MB/s   784 MB/s    13731505  32.31 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lzav 4.5 -1               249 MB/s  1480 MB/s    14378562  33.83 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
density 0.14.2 -3         273 MB/s   308 MB/s    14537655  34.21 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
yappy 2014-03-22 -99      107 MB/s  1849 MB/s    17949476  42.23 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
fastlz 0.5.0 -2           229 MB/s   535 MB/s    18388615  43.27 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lz4 1.10.0                422 MB/s  2680 MB/s    18493385  43.51 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
lzf 3.6 -1                274 MB/s   520 MB/s    18519603  43.58 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
snappy 1.2.1              309 MB/s  1104 MB/s    18648583  43.88 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl
zstd_fast 1.5.6 --5       363 MB/s  1260 MB/s    18951858  44.59 KAZE_stuff/En-En_American_Heritage_Dictionary_4_UTF-8.dsl

TOP 3 Pareto frontiers (compressors for which no other compressors BOTH compress smaller and decompress faster):
zstd 1.5.6 -22
nakamichi okamigan
lzsse2 2019-04-18 -17

'Wolfeye' is 1347/877=1.53x faster-but-weaker than zstd.

lzbench 2.0 (64-bit Linux)  Intel(R) Core(TM) i7-8565U CPU @ 1.80GHz
...
done... (cIters=1 dIters=37 cTime=1.0 dTime=2.0 chunkSize=1706MB cSpeed=0MB)

The results sorted by column number 4:
Compressor name         Compress. Decompress. Compr. size  Ratio Filename
bsc 3.3.4 -1             21.1 MB/s  51.0 MB/s    21875452   6.83 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lzma 24.09 -9            3.01 MB/s   304 MB/s    21889563   6.83 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
zstd 1.5.6 -22           1.27 MB/s  2235 MB/s    22118399   6.90 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
tornado 0.6a -16         2.19 MB/s   509 MB/s    23467167   7.32 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
brieflz 1.3.0 -9         1.85 MB/s   880 MB/s    27833172   8.69 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
kanzi 2.3 -9             4.89 MB/s  5.29 MB/s    27947512   8.72 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
zstd24LDM 1.5.6 -22      1.33 MB/s  1769 MB/s    31601578   9.86 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
bsc 3.3.4 -6             34.5 MB/s  13.7 MB/s    34340340  10.72 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lizard 2.1 -49           0.99 MB/s  1868 MB/s    37833141  11.81 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lizard 2.1 -29           1.16 MB/s  2420 MB/s    42569504  13.28 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
brotli 1.1.0 -11         0.59 MB/s   721 MB/s    45327417  14.14 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
nakamichi okamigan       0.00 MB/s  2925 MB/s    45999508  14.35 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
bsc 3.3.4 -3             40.4 MB/s  21.8 MB/s    46800824  14.60 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
bzip2 1.0.8 -9           11.8 MB/s  39.4 MB/s    63824315  19.92 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lzav 4.5 -2              59.8 MB/s  2342 MB/s    66238194  20.67 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
crush 1.0 -2             0.66 MB/s   596 MB/s    66283232  20.68 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
libdeflate 1.23 -12      3.48 MB/s  1222 MB/s    76437879  23.85 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
zlib 1.3.1 -9            23.4 MB/s   427 MB/s    80131114  25.01 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lizard 2.1 -39           1.83 MB/s  2031 MB/s    81204140  25.34 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lzav 4.5 -1               376 MB/s  2132 MB/s    81688987  25.49 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
kanzi 2.3 -1              374 MB/s  1821 MB/s    84326978  26.32 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lzo1x 2.10 -999          11.3 MB/s   737 MB/s    87981076  27.46 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lz4hc 1.10.0 -12         18.1 MB/s  3566 MB/s    88965755  27.76 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lizard 2.1 -19           1.98 MB/s  2884 MB/s    89320791  27.87 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lzsse2 2019-04-18 -17    3.74 MB/s  3988 MB/s    89499005  27.93 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
quicklz 1.5.0 -3         58.7 MB/s  1009 MB/s    98058319  30.60 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
density 0.14.2 -3         362 MB/s   395 MB/s   100655900  31.41 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lz4 1.10.0                556 MB/s  3116 MB/s   117161746  36.56 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
zstd_fast 1.5.6 --5       540 MB/s  1772 MB/s   117967120  36.81 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
fastlz 0.5.0 -2           333 MB/s   829 MB/s   118464663  36.97 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
snappy 1.2.1              435 MB/s  1411 MB/s   124047015  38.71 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
lzf 3.6 -1                328 MB/s   625 MB/s   129614904  40.45 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar
yappy 2014-03-22 -99     99.8 MB/s  2069 MB/s   152032354  47.44 KAZE_stuff/ebible.org_33_English_Bibles_37736-HTMs.tar

TOP 3 Pareto frontiers (compressors for which no other compressors BOTH compress smaller and decompress faster):
zstd 1.5.6 -22
nakamichi okamigan
lzsse2 2019-04-18 -17

'Wolfeye' is 2925/2235=1.30x faster-but-weaker than zstd.

The brutally fast Oodle and LZTurbo are missing, one day when open-sourced they will redefine the roster.
@avaneev
Copy link

avaneev commented Feb 21, 2025

It's interesting lzav -2 is on par or outperforms zlib -9 on larger files. lzav is pure LZ77, and its compression algorithms were tuned for speed, so its stream format can probably be utilized even better.

Hopefully there are no issues with benchmarking. lzav -2 decompressing faster than lzav -1 is a bit suspicious to me.

As far as Oodle and LZTurbo go, we'll probably never know if they are actually faster than lzsse2 on decompression. These are commercial algorithms and advertised benchmarks may not be reliable. I do not think they'll be open-sourced ever.

@tansy
Copy link
Collaborator

tansy commented Feb 24, 2025

@Sanmayce: as far as I know Nakamichi is your codec. Would you mind to add simple statement what the licence of the software is?
Practically all other codecs have their licences, this one could also have.

@tansy
Copy link
Collaborator

tansy commented Feb 24, 2025 •
edited
Loading

lzav -2 decompressing faster than lzav -1 is a bit suspicious to me.

There is nothing 'suspicious', not surprising about it. Exactly same relation applies to lz4hc, zlib, lzma, lizard (for the most part), ucl, tornado, yappy, practically all LZ-scheme codecs.
It's pretty simple to explain - better compressed bit stream contains more, longer matches, that are simply copied from memory to output, and the longer these matches are the faster it will go.

@avaneev
Copy link

avaneev commented Feb 24, 2025 •
edited
Loading

Well, as an author of lzav I generally rarely witnessed -2 being faster than -1 at decompression. It may be faster maybe by 2%, and not by 10%. The contradiction in your statement is exactly about having "more matches" that increases the amount of "branching" per byte in the stream.

Realistically, memory copy in LZ77 decompression is a very minor factor - it's all about branching.

@avaneev
Copy link

avaneev commented Feb 24, 2025

Well, probably I'm incorrect and there can be edge cases.

@tansy
Copy link
Collaborator

tansy commented Feb 24, 2025 •
edited
Loading

The contradiction in your statement is exactly about having "more matches" that increases the amount of "branching" per byte in the stream.

Brachylogy.

You can always test what is the reason for that, but you can't deny the observation, that virtually all LZ-scheme codecs behave this way.

You can make your compressor to use only short matches, and second only long ones, if you keep same compression ratio then you can test the hypothesis that length of the match matters.
If not, than figure out what is the reason for the obseserved behaviour.

It's not always the case. On ARM lzav is rather slow (relatively), especially on ARM32 [1] and `-2' is much slower than `-1'. On ARM64 [2] is almost as fast as LZ4 but levels behave the same.
These tests are significantly skewed though, due to small size of corpus (lzbench executable).
On bigger data corpus like silesia, -2 is faster than -1 [3], even though lzbench binary shows opposite relation [4].
It's not that fast in relation to LZ4 either.
You can find more of these test here [5].

  1. lzbench@ARM32 from here
  2. lzbench@ARM64 from here
  3. silesia@Cortex-A53
  4. lzbench@Cortex-A53
  5. gcc ARM 32-bit [Ubuntu 24.04 Noble]
    Update lzbench to v2.0.1

@avaneev
Copy link

avaneev commented Feb 24, 2025 •
edited
Loading

What matters it's both the number of matches and average match length. Many short matches yields slower decompression - that's what I usually observe with silesia dataset.

Thanks for the links to the benchmarks. However, not too useful as there is no machine info. What's the difference between Cortex-A53 and ARM64?

@avaneev
Copy link

avaneev commented Feb 24, 2025

It would be a good idea for someone to combine Silesia and Manzini datasets. Computing evolved and now benchmarking larger datasets should not be an issue.

@tansy
Copy link
Collaborator

tansy commented Feb 25, 2025 •
edited
Loading

Thanks for the links to the benchmarks.

You're welcome. Here are all of them.

However, not too useful as there is no machine info.

That's the problem with ARM - they don't have human readable, string CPUID, as if it was big problem to add few transistors with one instruction.

What's the difference between Cortex-A53 and ARM64?

ARM32/64 comes from CI tests, Cortex-A53 is my mobile.

Computing evolved and now benchmarking larger datasets should not be an issue.

If you mean CI tests then you are wrong - just check time of these tests - 40 min for 8 MB data set. I don't know Manzini corpus/data set, but Silesia is 200 MB, 25 times more than lzbench.exe. It also means 25 times more time to test it; that's 1000 minutes, 17 hours. In reality even more. If you pay for it, sure they won't care. But it's still looong time.

Of course you can test it with your codec only, in your repo, that's much more sensible and manageable.

@avaneev
Copy link

avaneev commented Feb 25, 2025 •
edited
Loading

Why make it so aggressive, eh? CI tests use 8 MB files. Of course, I'm not talking about CI tests.

@tansy
Copy link
Collaborator

tansy commented Feb 25, 2025 •
edited
Loading

Why make it so aggressive, eh?

?

CI tests use 8 MB files.

7, to be more precise. It uses it's own executable as input ($ ./lzbench -eall ./lzbench).

Of course, I'm not talking about CI tests.

I don't know then.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@avaneev @Sanmayce @tansy