Pre-fetching byte-range requests

[daniel-j-h]

Hey folks I just learned that Cloudfront doesn't cache or retrieve files of more than 30 GB. It does however cache individual smaller byte-range requests.

With a S3+Cloudfront setup hosting e.g. the >100GB planet pmtiles this results in

• the map making a byte range request against cloudfront
• cloudfront passing the request onto the file on s3
• s3 returning the bytes to cloudfront
• cloudfront now caching this specific byte range
• cloudfront returning the byte range to the map

This makes me think

1. if it's always beneficial to use cloudfront on top of s3, and
2. if there is a way to prefetch byte-ranges in the pmtiles plugin

With prefetching I mean queuing up byte-range requests e.g. for zoom levels lower/higher (use case: map zoom in, zoom out) or neighboring tiles (use case: map panning). Would prefetching have to happen in maplibre-gl itself (where I haven't found a good answer)?

References

• https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/RangeGETs.html

[bdon]

Pre-fetching generally isn't a great solution because you don't know in advance how the user will behave interactively.

I don't recommend using cloud front to cache byte ranges, CDNs are inconsistent with how they deal with 206 Partial Content.

The Lambda solution https://protomaps.com/docs/cdn/aws is the best way to glue Cloudfront and S3 together.

[daniel-j-h]

I don't recommend using cloud front to cache byte ranges, CDNs are inconsistent with how they deal with 206 Partial Content.

Got it 👍 That is exactly what I'm seeing, too.

Regarding prefetching: below is an example where we do know the camera's path: it's not about user interaction but a pre-defined camera path, think: flyTo or similar in a map story-telling use-case (my use case is similar that's why I'm picking this example from the maplibre docs).

Right now on a good network connection this is what the user experience looks like

Screenshare.-.2023-10-05.12.17.15.PM.webm

[bdon]

I suggest making a feature request in https://github.com/maplibre/maplibre-gl-js - prefetching requests for animations would be useful for any backend, whether pmtiles or another tiled provider

[daniel-j-h]

Alright then let's take it to maplibre/maplibre-gl-js#116 :)