reverse_proxy: bodyless HTTP/3 client GET forwarded to an HTTP/2 upstream gets ContentLength -1 instead of 0 (regression in v2.9.0)

[ChintanRaval]

Important

• Symptom was investigated, root cause was found, and this issue was raised - all by Claude (with occasional (re)direction from me)
• This note was authored by me

Description

When a client request arrives over HTTP/3 with no body (e.g. a GET) and reverse_proxy forwards it to the upstream over HTTP/2, Caddy sends the upstream request with an indeterminate body length (ContentLength: -1) instead of 0.

• Lenient HTTP/2 upstreams tolerate it.
• Strict HTTP/2 upstreams (e.g. IIS + Application Request Routing) reject the indeterminate-length request → HTTP 502. There it surfaces as an illegal Transfer-Encoding: chunked request header (forbidden in HTTP/2, RFC 9113 §8.1.1).

The same request is forwarded correctly (ContentLength: 0) when it originates over HTTP/1.1 or HTTP/2, or when the upstream hop is forced to HTTP/1.1.

This is a regression introduced in v2.9.0 (v2.8.4 and earlier are unaffected). It may originate in the bundled HTTP/3 (quic-go) request handling.

Reproduction (two Caddy instances, no other dependencies)

upstream — echoes what it received:

https://up.example.test:9500 {
	tls internal
	templates
	respond "proto={{.Req.Proto}} transfer-encoding={{.Req.TransferEncoding}} content-length={{.Req.ContentLength}}" 200
}

front — HTTP/3 enabled (default), HTTP/2 to the upstream:

https://front.example.test:8443 {
	tls internal
	reverse_proxy https://up.example.test:9500 {
		transport http {
			tls_insecure_skip_verify
		}
	}
}

Send a bodyless GET to the front, once over HTTP/3 and once over HTTP/2, and compare what the upstream reports:

client protocol	upstream sees
HTTP/3	content-length=-1 ❌
HTTP/2	content-length=0 ✅

Add versions 1.1 to the front's transport http block and both become content-length=0 — confirming the trigger is HTTP/2 to the upstream.

Version bisect

version	result	version	result
2.7.6	OK	2.9.1	broken
2.8.4	OK	2.10.0	broken
2.9.0	broken (first)	2.11.4	broken

Real-world impact

Browser (HTTP/3) → Caddy → IIS/ARR upstream (HTTP/2). ARR returns 502.7 (0x80070057, E_INVALIDARG) for every proxied asset. IIS Failed Request Tracing shows the inbound request carrying Transfer-Encoding: chunked.

Workarounds

• transport http { versions 1.1 } on the upstream, or
• disable HTTP/3 on the front.

Environment

• Caddy v2.11.4 (also reproduced on 2.9.0 / 2.9.1 / 2.10.0), Windows amd64.
• Client used for the bisect: .NET HttpClient with Version = 3.0 and VersionPolicy = RequestVersionExact (forces HTTP/3); any HTTP/3 client sending a bodyless GET reproduces it.

[u5surf]

I dug into this. The root cause is upstream in quic-go, but it turns out to be working as intended, so the fix belongs here in Caddy.

Why HTTP/3 gives ContentLength == -1: HTTP/3 has no END_STREAM flag on the HEADERS frame (unlike HTTP/2); the end of a request is signaled by the QUIC stream FIN, which isn't surfaced on the *http.Request. So a bodyless GET arrives with ContentLength == -1 (unknown) instead of 0. I filed quic-go/quic-go#5685 and confirmed with the maintainer that -1 is correct here — and notably golang.org/x/net/internal/http3 does exactly the same thing, so it's the de-facto behavior, not a quic-go bug.

Why it breaks the proxy: with ContentLength == -1 and a non-nil body, Go's http2.Transport forwards the request with an indeterminate length, which strict HTTP/2 upstreams (IIS + ARR) reject — a chunked transfer-coding is illegal in HTTP/2 (RFC 9113 §8.1.1).

Proposed fix (Caddy side): in reverse_proxy's prepareRequest, normalize a GET/HEAD request that has ContentLength == -1 to a definite zero-length body (ContentLength = 0, Body = nil) before forwarding, matching what HTTP/1.1 and HTTP/2 already produce. I scoped it to GET/HEAD because there's no non-blocking way to detect an empty body for methods that may legitimately stream one (and this file already treats GET/HEAD as bodyless during rewrites).

I have a patch + test ready and can open a PR if this approach looks good.

[cs8425]

We encountered a very similar issue a few days ago in production:

Client → HTTP/3 → Caddy → HTTP/2 → IIS 10 → ASP.NET Core (ANCM Out-of-Process)

and initially suspected Caddy as well.

After further investigation, we were able to reproduce the failure without Caddy, HTTP/3, QUIC, or ARR involved at all.

The following requests sent directly to IIS consistently fail only when ASP.NET Core is hosted Out-of-Process:

• HTTP/1.1 GET + Transfer-Encoding: chunked
• HTTP/2 GET with HEADERS followed by one or more DATA frames

Meanwhile, all of the following succeed:

• GET
• GET + Content-Length: 0
• GET + Content-Length: N
• POST + Transfer-Encoding: chunked
• HTTP/2 POST with DATA frames

---

Another additional data point from our investigation:

To better understand the request representation forwarded by Caddy to HTTP/2 upstreams, we used a minimal Go HTTPS server as the backend. The server exposes only a single handler that dumps request headers and request metadata, and we enabled:

• GODEBUG=http2debug=2

to inspect the received HTTP/2 frames.

We also validated the same request patterns using a custom client that can directly send equivalent HTTP/2 requests to the backend, bypassing Caddy entirely.

Observations from Caddy → HTTP/2 backend

HTTP/2 client GET → Caddy → HTTP/2 backend:

1. HEADERS (flags: END_STREAM | END_HEADERS)

HTTP/3 client GET → Caddy → HTTP/2 backend:

1. HEADERS (flags: END_HEADERS)
2. DATA (flags: END_STREAM)

In both cases, the HTTP/2 request does not contain a Transfer-Encoding: chunked header.

Observations with HTTP/1.1 upstream

When Caddy is configured to use HTTP/1.1 toward the backend, both HTTP/2 and HTTP/3 client cases also do not result in a Transfer-Encoding: chunked header being added.

IIS behavior (separate observation)

When sending equivalent requests directly to IIS (without Caddy), we observe:

• Classic ASP reports Transfer-Encoding: chunked
• ASP.NET Core In-Process reports Transfer-Encoding: chunked
• ASP.NET Core Out-of-Process returns 502.3 (0x80070057)

Summary of findings

Across both:

• Go-based HTTP/2 backend inspection
• IIS direct testing

we consistently observe that:

• Transfer-Encoding: chunked is not present on the HTTP/2 wire from Caddy
• yet IIS/ASP.NET Core In-Process reports it in the request state

This suggests that the Transfer-Encoding: chunked representation is not something Caddy explicitly forwards at the HTTP/2 layer, but rather a server-side interpretation of the request body mode.

Based on these observations, we currently consider the IIS/ANCM Out-of-Process behavior to be independently reproducible without Caddy in the request path.

---

Another one observation that may be relevant to the proposed normalization approach:

GET requests carrying a body are accepted by IIS/ASP.NET Core when represented using Content-Length.

For example:

• GET + Content-Length: 0 → OK
• GET + Content-Length: N → OK

Only the streamed representations fail in Out-of-Process mode only:

• GET + Transfer-Encoding: chunked → 502.3
• HTTP/2 GET HEADERS + DATA with END_STREAM flag → 502.3

Because of that, I'm not sure whether treating every GET with ContentLength == -1 as a definite zero-length request is always semantically equivalent. There may be cases where a legitimate GET request body is present.

---

The IIS/ANCM issue can be reproduced independently here:

• ASP.NET Core issue: ANCM Out-of-Process return 502 Bad Gateway on HTTP/2 split-frame GET and HTTP/1.1 chunked GET dotnet/aspnetcore#67423
• Reproduction repository: https://github.com/cs8425/aspnetcore-oop-http2-bug

For anyone affected by this combination, we found two workarounds:

• force using http1.1 to connect with backend (set ["1.1"] in https://caddyserver.com/docs/json/apps/http/servers/routes/handle/reverse_proxy/transport/http/versions )
• disable HTTP/3 on the frontend

Both eliminate the IIS 502 errors in our environment.

Our environment

• Caddy server version:
  • v2.11.2 h1:iOlpsSiSKqEW+SIXrcZsZ/NO74SzB/ycqqvAIEfIm64= @ linux amd64 docker
  • v2.11.4 h1:XKxkMTgNSizEvKG6QHue6cAsFOteU2qA61w2tKkCWi0= @ win amd64 for isolation and reproduce issue
• IIS 10.0
• dotnet 10

[inetol]

Is this related with #6678 ?