reflector: back off when watch stream closes early to prevent reconnect storms

[kaddynator]

Background

When the Kubernetes API server closes a watch stream shortly after the initial list — typically 2–6 seconds, observed on GKE and behind API gateways — the reflector reconnects immediately and enters a tight loop. With the default timeout_seconds=10 this produces dozens of reconnects per minute, degrading API server performance and triggering rate limiting (related to #436).

This is the companion fix suggested in the inline review of #755, where @juliantaylor noted that "other k8s components typically relist watches in the range of minutes to never, every 10 seconds is a bit much" and @yuvipanda agreed it should be done in a separate PR.

Closes #577, related to #436, follow-on from #755

Reconnect behaviour: before and after

flowchart TD
    A([watch stream exits cleanly]) --> B{watch_duration
≥ 30s?}

    B -- Yes
normal timeout or restart --> C[reset early_close_delay
reconnect immediately]
    B -- No
server closed early --> D[log warning
sleep early_close_delay
double delay ← capped at 30s]

    C --> E([start next watch])
    D --> E

Loading

Before this change the else branch always reconnected immediately, taking the left path for every stream exit — early or normal.

Change

In the else branch of _watch_and_update, detect when watch_duration < 30s on a clean (non-exception) stream exit and apply exponential backoff before reconnecting. The delay doubles on each consecutive early close (starting at 0.1s, capped at 30s). Watches that run ≥30s — normal timeout_seconds or restart_seconds expiry — reconnect immediately as before.

A separate early_close_delay counter is used so this backoff does not interfere with the existing exception backoff tracked by cur_delay.

No breaking changes

Deployments on clusters that close watch streams normally (at or after timeout_seconds) will see no behavioral difference. The backoff only activates on abnormally short stream lifetimes.