Resumable Uploads: Resource-specific upload limits

[Acconut]

As discussed in #2741, this mechanism allows the server to send an Upload-Limit header in informational and final responses. The header indicates a maximum upload size that the server accepts. If the upload exceeds the size, the server may abort it depending on its preference. A client should not continue upload if it knows that its upload won't fit into that size.

[LPardue]

Can the upload limit change during am upload? I don't think we should allow that

[guoye-zhang]

Do we want Upload-Limit to be a dictionary structured field so it can support additional information in the future? E.g. Upload-Limit: bytes=100000000, concurrency=4

[Acconut]

Can the upload limit change during am upload? I don't think we should allow that

Good point, I agree that should not be allowed.

[Acconut]

Do we want Upload-Limit to be a dictionary structured field so it can support additional information in the future? E.g. Upload-Limit: bytes=100000000, concurrency=4

I don't see a reason against this. We could also use this to indicate a limit on the lifetime of an upload resource or the minimum and maximum sizes of individual PATCH requests. Since we get dictionaries for free from structured fields, we can make it a dictionary easily.

[Acconut]

Added the changes. Let me know what you think

[LPardue]

Making it a dictionary is going to need more text than that I'm afraid. You're going to need an IANA registry etc . But most importantly we then need to define a framework for limits and what the expectations are for setting them. Immediately I can see from Guoye's example we are conflating limits on the upload resource with limits on the server more generally, and that sort of generic capability is an interop concern

We're also treading near to the rate limit headers in httpapi. The uses cases are distinct but there was a lot of friction about how to express these things over there so we need to be wary.

I'm not against the idea of Dictionary but this needs more WG discussion before landing

[Acconut]

That makes sense. Maybe we are better of with individual headers to represent the individual limits. I see that individual headers are also more in line with how messages in HTTP are laid as this allows reuse etc.

On the top of my head, I can think of the following limits applying to an upload resource:

• total size
• minimum and maximum size of an upload creation or append request (e.g. limited by proxies or buffers)
• lifetime of the resource
• concurrent upload request (if that's specified in another document in the future)

For some of these there already seem to be existing headers. For example, lifetime could be expressed using the Sunset header.

[Acconut]

At IETF 119, we discussed upload limits and there were no concerns raised about utilizing a dictionary for upload limits. As long as we include language for handling future extensions/changes of the header field, we also apparently don't need a separate registry.

[Acconut]

I extended this PR to include more upload limits. Most of these are already used in production system for resumable uploads, but are only documented in the provider's website, inaccessible to the clients themselves:

• max-size is the maximum size for a single uploaded file.
• min-size is the minimum size. A server might not want to create a resumable upload for a 1KB file and instead use traditional uploads for this.
• min-append-size and max-append-size is the minimum and maximum size of an append request. Many deployments put a limit on this, likely to cap the duration of a single request and help with proxies and other intermediaries. For example, CloudFlare has an lower (5 MiB) and upper (200 MiB) limit (search for chunk size on the linked pages).
• expires is the remaining lifetime of an upload in seconds and let's the client know how long it can resume the upload.

[guoye-zhang]

Would be hesitant defining a min append size. What if the last remaining part is smaller than the min append size?

The min size might also interact badly if we are doing automatic upgrades. The client wouldn't know it beforehand and we don't want the upload to get rejected.

[Acconut]

What if the last remaining part is smaller than the min append size?

Good point, which should be addressed. The last append request (which includes Upload-Complete: ?1) would be allowed to surpass the minimum append size. This is also similar to how multipart uploads to S3 work. Each part must be larger than 5MB, except for the last part, which might be smaller.

The min size might also interact badly if we are doing automatic upgrades. The client wouldn't know it beforehand and we don't want the upload to get rejected.

The minimum append size shouldn't be a problem for automatic upgrades as the client should try to upload the entire file is one request anyway, as far as I understand it. The maximum append size on the other hand, might cause a problem, indeed. However, hopefully the client received in the informational response and is able to resume the upload while abiding to the min/max append sizes. But of course, limits on the append sizes can make the uploads less efficient. In my experience, some services do implement the limits, so I thought it would be better to provide a standardized way to discover those limits. Or do you think it is better to not include such append limits in the protocol to encourage implementers to handle append requests without limits?