Add Shared Brotli and Shared Zstandard (zstd)

[summercms]

This feature adds support for using designated previous responses, as an external dictionary for Brotli(or Zstandard)-compressing HTTP responses.

If the browser and server share a dictionary, using the dictionary for compression can significantly reduce the amount of data transferred. This is especially useful for incremental library updates, or when browsing multiple pages that have a lot of common parts in their files.

Compression Ratio Comparison Tests

format	byte	ratio
html	27278	100%
br	5453	20%
sbr	4559	17%

Browser Support

• Shared Zstandard support is available on Google Chrome 118 onwards.
• Firefox: Positive
• Safari: No signal

Chrome 118

Chrome 117

The following flags were enabled on Chrome Canary 117.0.5912.0 and the behaviour was verified.

#enable-compression-dictionary-transport
#enable-compression-dictionary-transport-backend

Demos and samples

https://compression-dictionary-transport-threejs-demo.glitch.me/
https://compression-dictionary-transport-shop-demo.glitch.me/

Info

https://github.com/WICG/compression-dictionary-transport
https://datatracker.ietf.org/doc/draft-meenan-httpbis-compression-dictionary
https://docs.google.com/document/d/1IcRHLv-e9boECgPA5J4t8NDv9FPHDGgn0C12kfBgANg/edit#heading=h.7nki9mck5t64
https://github.com/facebook/zstd
https://en.wikipedia.org/wiki/Zstd

Examples

Bundled JavaScript on separate origin

In this example, www.example.com will use a bundle of application JavaScript that they serve from a separate static domain (static.example.com). The JavaScript files are versioned and have a long cache time, with the URL changing when a new version of the code is shipped.

On the initial visit to the site:

• The browser loads https://www.example.com/ which contains <script src="//static.example.com/app/main.js/123" crossorigin> (where 123 is the build number of the code).
• The browser requests https://static.example.com/app/main.js/123 with Accept-Encoding: sbr,br,gzip.
• The server for static.example.com responds with the file as well as use-as-dictionary: match="/app/main.js*", Access-Control-Allow-Origin: https://www.example.com and Vary: Accept-Encoding,sec-available-dictionary.
• The browser caches the js file along with a SHA-256 hash of the decompressed file and the https://www.example.com/app/main.js* URL pattern.

sequenceDiagram
Browser->>www.example.com: GET /
www.example.com->>Browser: ...<script src="//static.example.com/app/main.js/123" crossorigin>...
Browser->>static.example.com: GET /app/main.js/123<br/>Accept-Encoding: sbr,br,gzip
static.example.com->>Browser: use-as-dictionary: match="/app/main.js"<br/>Access-Control-Allow-Origin: https://www.example.com<br/>Vary: Accept-Encoding,sec-available-dictionary

At build time, the site developer creates delta-compressed versions of main.js using previous builds as dictionaries, storing the delta-compressed version along with the SHA-256 hash of the dictionary used (e.g. as main.js.<hash>.sbr).

On a future visit to the site after the application code has changed:

• The browser loads https://www.example.com/ which contains <script src="//static.example.com/app/main.js/125" crossorigin>.
• The browser matches the https://www.example.com/app/main.js/125 request with the https://www.example.com/app/main.js* URL pattern of the previous dictionary response that is in cache and requests https://static.example.com/app/main.js/125 with Accept-Encoding: sbr,br,gzip, sec-fetch-mode: cors and sec-available-dictionary: <SHA-256 HASH>.
• The server for static.example.com matches the URL and hash with the pre-compressed artifact from the build and responds with it and Content-Encoding: sbr, Access-Control-Allow-Origin: https://www.example.com, Vary: Accept-Encoding,sec-available-dictionary.

It could have also included a new use-as-dictionary: match="/app/main.js*" response header to have the new version of the file replace the old one as the dictionary to use for future requests for the path but that is not a requirement for the existing dictionary to have been used.

sequenceDiagram
Browser->>www.example.com: GET /
www.example.com->>Browser: ...<script src="//static.example.com/app/main.js/125" crossorigin>...
Browser->>static.example.com: GET /app/main.js/125<br/>Accept-Encoding: sbr,br,gzip<br/>sec-fetch-mode: cors<br/>sec-available-dictionary: [SHA-256 HASH]
static.example.com->>Browser: Content-Encoding: sbr<br/>Access-Control-Allow-Origin: https://www.example.com<br/>Vary: Accept-Encoding,sec-available-dictionary

Site-specific dictionary used for all document navigations in a part of the site

In this example, www.example.com has a custom-built dictionary that should be used for all navigation requests to /product.

On the initial visit to the site:

• The browser loads https://www.example.com/ which contains <link rel=dictionary href="/dictionaries/product_v1.dat">.
• At an idle time, the browser sends an uncredentialed fetch request for https://www.example.com/dictionaries/product_v1.dat.
• The server for www.example.com responds with the dictionary contents as well as use-as-dictionary: match="/product/*" and appropriate caching headers.
• The browser caches the dictionary file along with a SHA-256 hash of the decompressed file and the https://www.example.com/product/* URL pattern.

sequenceDiagram
Browser->>www.example.com: GET /
www.example.com->>Browser: ...<link rel=dictionary href="/dictionaries/product_v1.dat">...
Browser->>www.example.com: GET /dictionaries/product_v1.dat<br/>Accept-Encoding: sbr,br,gzip
www.example.com->>Browser: use-as-dictionary: match="/product/*"

At some point after the dictionary has been fetched, the user clicks on a link to https://www.example.com/product/myproduct:

• The browser matches the /product/myproduct request with the https://www.example.com/product/* URL pattern of the previous dictionary request and requests https://www.example.com/product/myproduct with Accept-Encoding: sbr,br,gzip and sec-available-dictionary: <SHA-256 HASH>.
• The server supports dynamically compressing responses using available dictionaries and has the dictionary with the same hash available and responds with a brotli-compressed version of the response using the specified dictionary and Content-Encoding: sbr (and Vary headers if the response is cacheable).

sequenceDiagram
Browser->>www.example.com: GET /product/myproduct<br/>Accept-Encoding: sbr,br,gzip<br/>sec-available-dictionary: [SHA-256 HASH]
www.example.com->>Browser: Content-Encoding: sbr

[LeoColomb]

Thanks for this sharing @summercms.
That said, I'm not sure what can be done with H5BP configs regarding this (still to be published) standard. Are we sure this is compatible with current server software?

[summercms]

@LeoColomb

See slide 4 found here: https://datatracker.ietf.org/meeting/117/materials/slides-117-httpbis-compression-dictionaries-00

Here's what I understand:

Use-As-Dictionary: match="/path/to/target/*"

When the client Use-As-Dictionary transitions to the path specified in , it adds to the request header that it supports CDT in Brotli ( sbr) and has a dictionary that can be applied there, as shown below.

Accept-Encoding: sbr
Sec-Available-Dictionary: <SHA-256 of Dictionary>

The server can use the dictionary specified by the client to compress and return the content.

If compression is performed using a dictionary, the response will be as follows. Since the encoding result of the same content changes depending on the dictionary, add the hash of the dictionary to the key of Sec-Available-Dictionary Cache Vary.

Content-Encoding: sbr
Vary: Accept-Encoding, Sec-Available-Dictionary

Since it is a method that generates a dictionary in advance, it can be said to be a method that is compatible with static site generators.

Was thinking could update this code section, with notes at a later date:

---

Disclaimer: This spec is likely to change still, so expecting this to be put on hold for the next major semantic version of this repo.