How to use /health endpoint and properly deal with the time-out ?

[zejunlitg]

version info:

etcd Version: 3.3.22
Git SHA: 282cce72f
Go Version: go1.12.17
Go OS/Arch: linux/amd64

checking /health endpoint sometimes reports this error:

2024-02-02 02:37:36.064366 W | etcdserver/api/etcdhttp: /health error; QGET failed etcdserver: request timed out (status code 503)

We checked the source code and apparently a 1 second timeout was hardcoded:

ctx, cancel := context.WithTimeout(context.Background(), time.Second)

in

if h.Health == "true" {
		ctx, cancel := context.WithTimeout(context.Background(), time.Second)
		_, err := srv.Do(ctx, etcdserverpb.Request{Method: "QGET"})
		cancel()
		if err != nil {
			h.Health = "false"
			plog.Warningf("/health error; QGET failed %v (status code %d)", err, http.StatusServiceUnavailable)
		}
	}

This is causing random failure in system when the load is getting relatively heavy.
How should we deal with it?
the health status also seems to be very sensitive:

2024-02-02 02:37:36.064366 W | etcdserver/api/etcdhttp: /health error; QGET failed etcdserver: request timed out (status code 503)
2024-02-02 02:37:39.066534 W | etcdserver/api/etcdhttp: /health error; QGET failed etcdserver: request timed out (status code 503)
2024-02-02 02:37:40.953636 I | etcdserver/api/etcdhttp: /health OK (status code 200)
2024-02-02 02:37:42.068253 W | etcdserver/api/etcdhttp: /health error; QGET failed etcdserver: request timed out (status code 503)

as you can see at 40s it succeeded but 2 seconds later it failed.
Any recommendation for a long-term fix for this? should I upgrade to newer versions? If so what is the most not-up-to-date version that has a fix for it?

[jmhbnz]

Hi @zejunlitg - Thanks for your question, etcd version 3.3.22 is four years old and unsupported. There have been countless bugs, security improvements and other improvements over the last four years so please update to a supported release as soon as possible.

[zejunlitg]

@jmhbnz thanks for your suggestion, un upgrade would potentially be what we want to do in the future, but for now, is there a way to alleviate the /health check issue? in short, ETCD is working but just cuz of the load is heavy, it reports that it's unhealthy, we only wanna fail what we do when we're certain that ETCD is down or is and will remain unhealthy, is there a way to do it in 3.3.22? thanks!