-
Notifications
You must be signed in to change notification settings - Fork 21
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fetch/unpack layers concurrently #365
Comments
Looked briefly at this, we probably want containers/containers-image-proxy-rs#31 to enable concurrency on the proxy. That said, it'd also work OK to wrap the proxy in a |
Came across https://docs.rs/futures/0.3.24/futures/stream/trait.StreamExt.html#method.buffer_unordered to help ensure we only fetch e.g. max of 2 at a time. |
We came across this issue last week. We are layering decently sized container on top of FCOS and in development it layers quickly but on various cloud providers it is painfully slow. We are still digging into why the provisioning times are so vastly different but we think that this might be the issue. Any chance of concurrency for layers getting added soon? |
Today we fetch layers serially, which is unnecessarily slow. I think we should start by fetching and unpacking at least 2 (possibly 3) layers in parallel.
This would crucially allow interleaving the fetching of each layer with the storage process.
The text was updated successfully, but these errors were encountered: