TurboBench: Dynamic/Static web content compression benchmark

[powturbo]

file html8 : 100MB random html pages from a 1m Alexa Top sites corpus.
Number of pages = 1178
Average length = 84886 bytes
The pages (length + content) are concatenated into a single html8 file,
but compressed/decompressed separately using the multiblock mode in TurboBench.

compress: page1,page2,...pageN
decompress : Page1,page2,...pageN

In the speedup (see below) plots you can see the best compressors for content providers:

• brotli 11 is best for static content
• brotli 5 is best until 1MB/s network transfer speed
• libdeflate 6 is best from 1MB/s to 6MB/s (followed by brotli,4)
• igzip 1,2 is best for very fast networks > 10MB/s

Remarks:

• Note that for web-content, the difference in decompression speed between zlib, brotli and others is practically imperceptible to the user.
• Actually there are only web server support for zlib, zlib_ng, slz and brotli.
• zstd is included only as reference as it is not compatible with web content encoding.

Unlike other benchmarks on the net, this is pure memory benchmark without any additional (http) server overhead.

Page Statistics:
Page Length: Minimum = 16kb, Maximum = 128kb

bits histogram:
15:######################## 24%
16:####################################### 39%
17:##################################### 37%

Lenovo Ideapad Pro 5 / CPU 7840HS 3.8-5.1GHz, DDR5 6400MHz

C Size	R%	C MB/s	D MB/s	Name	C Mem	D Mem	C Stack	D Stack
16457986	16.5	1.41	612.34	brotli 11	10,629,680	247,672
18579017	18.6	60.35	703.56	brotli 5	10,711,032	199,384
18996682	19.0	0.53	1787.41	zstd 22	815,429,856	191,952
19615056	19.6	0.34	1003.78	zopfli	33,644,120	14,352
19766557	19.8	7.10	1661.60	libdeflate 12	18,027,032	23,152
19971457	20.0	135.89	716.37	brotli 4	10,153,192	198,904
19974358	20.0	5.64	2092.36	zstd 15	72,091,872	191,952
20282366	20.3	96.65	1700.68	libdeflate 9	1,336,592	23,152
20363869	20.4	55.00	687.49	zlib 9	274,096	14,320
20451000	20.5	84.95	1130.76	zlib_ng 9	36 GB	778,098,680
20485533	20.5	92.93	682.52	zlib 6	274,096	14,320
20502399	20.5	201.65	1702.04	libdeflate 6	1,340,024	23,152
20568950	20.6	78.85	2004.33	zstd 5	5,503,608	191,952
20592048	20.6	173.56	1141.23	zlib_ng 6	36 GB!	778,098,680
21624240	21.6	242.98	702.77	brotli 2	9,235,608	231,792
22159165	22.2	434.23	1733.07	libdeflate 1	407,544	23,152
22484852	22.5	370.72	1483.97	igzip 3	696,336	0	7,340,032	7,340,032
22867303	22.9	374.92	2064.67	zstd 1	1,374,840	191,952
23098360	23.1	766.27	1446.97	igzip 2	663,568	0	7,340,032	7,340,032
23153034	23.2	518.39	678.09	brotli 1	1,200,512	165,704
23287503	23.3	850.51	1444.42	igzip 1	569,336	0	7,340,032	7,340,032
23723266	23.7	238.93	629.48	zlib 1	274,096	14,320
25355327	25.4	982.05	1543.00	igzip 0	1,032	0	7,340,032	7,340,032
28214601	28.2	573.26	692.20	slz 6	0	14,320	65,536
28214601	28.2	576.88	684.35	slz 9	0	14,320	65,536
29476316	29.5	587.78	679.40	slz 1	1,032	14,320	65,536
30040886	30.0	569.15	1078.03	zlib_ng 1	25 GB	778,098,680

R: Compression ratio
Mem: Heap Memory usage in bytes
Stack: Stack Memory usage in bytes
Zstd is added only as indication. There is actually no zstd content-encoding for the web

speedup files

[powturbo]

Benchmark with "silesia.tar"
Lenovo Ideapad Pro 5 / CPU 7840HS 3.8-5.1GHz, DDR5 6400MHz

C Size	R%	C MB/s	D MB/s	Name
64677891	30.5	8.36	1443.74	libdeflate 12
66715898	31.5	45.81	1405.76	libdeflate 9
67511452	31.9	134.54	1420.39	libdeflate 6
67644075	31.9	16.99	560.77	zlib 9
68152563	32.2	30.69	861.26	zlib_ng 9
68228660	32.2	41.74	555.55	zlib 6
68914854	32.5	104.47	871.24	zlib_ng 6
70166917	33.1	218.74	1417.63	libdeflate 3
72490921	34.2	164.66	820.80	zlib_ng 3
72968832	34.4	101.07	558.79	zlib 3
73505577	34.7	334.63	1360.91	libdeflate 1
75094329	35.4	378.76	1222.12	igzip 3
76571415	36.1	672.85	1183.64	igzip 2
77260023	36.5	157.67	521.54	zlib 1
78154519	36.9	700.23	1153.13	igzip 1
87551010	41.3	756.87	1107.45	igzip 0
100929713	47.6	401.93	774.98	zlib_ng 1
211948544	100.0	27554.41	27493.65	memcpy