Reduce calls to s3 for downloading files

[elruwen]

Request details

Hi!

When I switched from the filestorage provider to s3, I noticed some decline in performance. Now I got finally the time to have a deeper look.

A request to download a file which is already stored in s3 results into:

• 3 HEAD requests
• 1 actual GET request

I think some form of caching would be great. I know there is already #1983 but that is more towards the UI performance.

I would suggest to cache the head request results via something like https://github.com/ben-manes/caffeine

I would let the user opt-in via the following configuration settings on the s3 provider:

metadataCacheEnabled: true|false - defaults to false
metadataCacheExperiationAfterSeconds: number - defaults to 28800 -> 8h
metadataCacheMaxItems: number - defaults to 1000

We need to exclude SNAPSHOT from caching, since they might update. Otherwise, I would consider regular maven versions to never change. So if the cache has an entry which is gone from s3, the get file since results into a 404.

I don't think there is any need to sync caches between multiple nodes.

I am happy to give the implementation a try if you are interesting in this feature.

[dzikoysk]

Yeah, the UI performance with S3 storage is quite terrible, but I guess that's expected with current solution. It has quite low priority on my side, but if you'd like to improve that, feel free to add a tiny opt-in caching layer there.

Regarding the details, I think we can discuss this directly on the PR, so feel free to give the base implementation a try