Issue Details
None of the 12 selection policies use observed latency. least_conn and random_choose balance on in-flight counts, so a fast and a degraded upstream look equal until requests pile up; when one backend slows down (GC pause, cold cache), tail latency spikes long before connection counts diverge.
Proposal: a least_latency policy — pick two random healthy upstreams, route to the lower score, where score = peak-sensitive EWMA of response latency × (in-flight + 1). O(1) per request, no coordination, zero required config (EWMA decay and choice count as optional tunables). Latency feedback comes from the existing RecordMetric hook, so no new plumbing.
I've implemented this algorithm in Apache brpc (apache/brpc#3367) and ClickHouse (ClickHouse/ClickHouse#109234); in brpc's benchmark with 1 degraded backend of 4, p99 dropped from 6.5ms (round-robin) to 337µs. Happy to open a PR with the policy, tests, and Caddyfile docs if maintainers are open to it.
Assistance Disclosure
AI used
If AI was used, describe the extent to which it was used.
AI tooling assisted with research and drafting; the algorithm follows my prior brpc/ClickHouse implementations.
Issue Details
None of the 12 selection policies use observed latency.
least_connandrandom_choosebalance on in-flight counts, so a fast and a degraded upstream look equal until requests pile up; when one backend slows down (GC pause, cold cache), tail latency spikes long before connection counts diverge.Proposal: a
least_latencypolicy — pick two random healthy upstreams, route to the lower score, where score = peak-sensitive EWMA of response latency × (in-flight + 1). O(1) per request, no coordination, zero required config (EWMA decay and choice count as optional tunables). Latency feedback comes from the existing RecordMetric hook, so no new plumbing.I've implemented this algorithm in Apache brpc (apache/brpc#3367) and ClickHouse (ClickHouse/ClickHouse#109234); in brpc's benchmark with 1 degraded backend of 4, p99 dropped from 6.5ms (round-robin) to 337µs. Happy to open a PR with the policy, tests, and Caddyfile docs if maintainers are open to it.
Assistance Disclosure
AI used
If AI was used, describe the extent to which it was used.
AI tooling assisted with research and drafting; the algorithm follows my prior brpc/ClickHouse implementations.