reverse_proxy: different active health checks (health_uri/health_headers) against the same upstream address collapse into one shared health state

[73VW]

Issue Details

1. Environment

1a. Operating system and version

Debian 12.14

1b. Caddy version (caddy version)

Detected on v2.11.1 (built with xcaddy).

Reproduced on v2.11.4 (docker. Check the reproduction repo below).

2. Description

2a. What happens

Two reverse_proxy handlers that dial the same upstream host:port but configure different active health checks (different health_uri and/or health_headers) share a single health state. Upstream health is stored in the process-global host pool keyed only by the dial address, so both handlers resolve to the same Host object.

Consequences:

• Only one checker's result is reflected: the shared healthy flag flaps / is last-writer-wins.
• GET /reverse_proxy/upstreams returns a single entry for the address, and caddy_reverse_proxy_upstreams_healthy{upstream="..."} is a single series — so the two distinct health results cannot be represented or graphed separately.
• Real-world impact: a vhost that is genuinely unhealthy on a node can be reported healthy (and vice-versa), and one vhost's outage evicts the node from the other vhost's data path.

2b. Why it's a bug

When #4341 was filed, active checks could not use an alternate URI, so "one dial address" was effectively equivalent to "one health check" and keying by address was consistent. Now that health_uri and health_headers are configurable per active check, two checks against the same address are semantically distinct health targets, but the state model still collapses them onto one address-keyed Host. (This matches the maintainer's own note on #4341 that the host used to be "equivalent to the active health check.")

Typical setup that hits this: Caddy in front of another reverse proxy that routes by Host / X-Forwarded-Host; each vhost checks a different backend health path on the same node. See the "Real-world use case" section below for why this is unavoidable under end-to-end TLS.

2c. Log output

caddy-edge     | {"level":"info","ts":1783665984.586571,"logger":"http.handlers.reverse_proxy.health_checker.active","msg":"status code out of tolerances","status_code":503,"host":"caddy-backend:80"}

08:45:12 healthy=0
08:45:11 healthy=0
08:45:10 healthy=0
08:45:09 healthy=0
08:45:08 healthy=1
08:45:07 healthy=1
08:45:06 healthy=1
08:45:05 healthy=1

2d. Workaround(s)

Give each vhost a distinct backend hostname (DNS alias to the same node) or a distinct port so the dial addresses differ; this yields separate Host objects, separate health state, and separate metric series. This is the "N hostnames for one IP" hack already noted in #4341: functional, but it forces alias management and duplicates connection/state that would otherwise be shared. There is no way to keep one address and still track two health results. (Under end-to-end TLS this workaround is not even a simple rename — see the use case below.)

2e. Relevant links

• Active health check not separated for same upstream #4341 (this behavior; closed but not resolved by the referenced PR)
• reverseproxy: Dynamic upstreams (with support for SRV and A/AAAA lookups) #4470 (dynamic upstreams / SRV + A/AAAA — orthogonal; active health checks don't run on per-request dynamic upstreams, and it still requires distinct names)
• Active health checks with health_port mark all upstreams as unhealthy despite successful manual checks #7544 (related concern about multiple handlers sharing the same upstream gauge)
• Reproduction repo: https://github.com/73VW/caddy-healthcheck-shared-state-repro

3. Tutorial (minimal steps to reproduce)

Self-contained, no external services. Save as /tmp/Caddyfile, then run docker run --rm -p 8080:8080 -p 2019:2019 -v "/tmp/Caddyfile:/etc/caddy/Caddyfile:ro" caddy:2.11.4-alpine.

{
    metrics
    admin 0.0.0.0:2019
    auto_https off
}
 
# Mock backend that DISAGREES per health path on the SAME address (localhost:9000)
:9000 {
    request_header Host {http.request.header.X-Forwarded-Host}
    @s3 host s3.localhost
    handle @s3 {
        handle_path /healthz {
            respond 200
        }
        respond "welcome to the s3 backend" 200
    }
    @admin host admin.localhost
    handle @admin {
        handle_path /health {
            respond 503
        }
        respond "welcome to the admin backend" 200
    }
    handle {
        respond "backend ok" 200
    }
}
 
:8080 {
    @s3 host s3.localhost
    handle @s3 {
        reverse_proxy localhost:9000 {
            health_uri /healthz
            health_fails 3
            health_interval 10s
            health_headers {
                X-Forwarded-Host s3.localhost
            }
        }
    }
    @admin host admin.localhost
    handle @admin {
        reverse_proxy localhost:9000 {
            health_uri /health
            health_fails 3
            health_interval 10s
            health_headers {
                X-Forwarded-Host admin.localhost
            }
        }
    }
}

Then observe:

# STRUCTURAL: one entry for localhost:9000, though two different checks are configured
curl --silent http://localhost:2019/reverse_proxy/upstreams | jq
 
# BEHAVIORAL: single series, value flaps between 1 and 0 (or sticks to one writer)
watch --interval 1 'curl --silent http://localhost:2019/metrics | grep caddy_reverse_proxy_upstreams_healthy'

Expected: the two active checks target the same node via different URIs, so their results should be trackable independently (distinct state and/or distinct metric series).

Actual: a single address-keyed Host holds one shared healthy value; /reverse_proxy/upstreams and the metric show one entry, and the value flaps depending on which checker wrote last.

4. Real-world use case (motivation)

This is not a contrived config — it falls out of running end-to-end HTTPS to backends that have no stable IP.

Setup:

• Two Caddy reverse proxies sit in a DMZ as the single public entry point. They also run an internal ACME CA.

• Backend nodes live on private networks (an on-prem LAN, plus a cloud site reached over a VPN). They have no fixed IP: DHCP assigns their address and updates a dynamic-DNS zone, so each node is reachable only by its dynDNS name (<node>.dyn.example.net).

• Each node obtains its TLS certificate from the edge CA (via ACME), issued for its dynDNS name.

• The edge proxies to the nodes over HTTPS.

• The nodes are all distinct machines — this is not a pool of interchangeable replicas of one service. Each node is unique.

• A single machine may expose one or several hostnames for one or several unrelated services (for example: a Django API backing an Angular front-end, plus a separate Flask app for the same client). Each of those services has its own health endpoint and must be checked independently, even though they share the machine's single dial address.
  The constraint that creates the collision:

• For upstream TLS to validate, the SNI / upstream Host must equal the node's certificate name — i.e. the node's dynDNS name. So the edge pins Host = <node>.dyn.example.net on every upstream connection to that node.

• But a single node serves several virtual services (e.g. an object-store data endpoint and its separate admin endpoint). Because Host is pinned to the machine name for TLS, the real service identity cannot travel in Host; it is carried in X-Forwarded-Host, and the node swaps X-Forwarded-Host → Host internally to route to the right vhost.
  Result: to monitor each service independently, the edge configures one reverse_proxy per service, each with a different health_uri and/or a different X-Forwarded-Host — yet all of them dial the same node address (<node>.dyn.example.net:443). That is exactly the case that collapses today: same dial address, different active health checks, one shared Host object.

Why dynamic upstreams (#4470) do not apply here: those solve address discovery for a pool of interchangeable backends. In this topology the upstreams are not interchangeable and there is no pool to resolve — each machine is unique and hosts unrelated services. The multiplicity is services per machine, not replicas per service, so load balancing / service discovery is orthogonal to the problem. (And active health checks don't run on dynamic upstreams anyway, as noted in 2e.)

Note this also makes the 2d workaround costlier than a cosmetic rename here: giving each service a distinct dial name (<node>-admin.dyn.example.net) would break the "upstream Host must equal the certificate name" invariant, so it additionally requires either issuing an extra certificate per alias or overriding the upstream SNI/Host. In other words, end-to-end TLS to a shared, dynamically-named node structurally pushes multiple distinct health checks onto a single dial address.

Operationally, the workaround also defeats a core goal of this setup: internal names are meant to be fully automatic. Each machine gets exactly one dynDNS record, maintained by DHCP, so adding a new service to an existing machine requires zero internal DNS changes. Minting a distinct internal name per service (per 2d) reintroduces exactly the manual internal-DNS bookkeeping the dynamic-DNS design was built to eliminate — one new record for every service added to a machine. Exposing a service publicly still needs an external CNAME plus an edge config change, and that is expected and unavoidable; but that cost is external. The regression is purely on the internal side — the side deliberately engineered to be zero-touch.

Assistance Disclosure

AI used

If AI was used, describe the extent to which it was used.

AI (Claude) was used to: research the relevant Caddy internals (the address-keyed global host pool) and prior issues (#4341, #4470, #7544); help draft this report; and scaffold the reproduction repository linked above. I reviewed everything, built and ran the reproduction myself, and confirmed the single-entry /reverse_proxy/upstreams output and the metric behavior — originally on v2.11.1 in production and on v2.11.4 with the minimal repro. The analysis and conclusions are my own.

[steadytao]

The metric collision is valid and the active pass/fail counters are shared through the address-keyed Host. One small correction is that the active unhealthy state itself is stored per Upstream so there is not one shared healthy flag. The current reproduction proves the metric collision but not that one handler evicts the address from another handler's data path. Could you add a routing assertion which demonstrates that impact or narrow the report accordingly?