New install of ETCD Server version and ETCD Cluster are different on one cluster, but are the same on another.

[tomkcpr]

Hey There!

Hope folks are doing good!

Have two clusters. One 3 node cluster is RHEL / CentOS / Alma / Rocky Linux based system version 8, the other 3 node cluster is identical 3 node with same OS. ETCD version is 3.X.Y on both, brand new install. Plan is to use Patroni but for now it's been disabled and is not running (was momentarily). Cleared ETCD Databases on both clusters and started each 3 node cluster up. Just the ETCD cluster, nothing else. On the first cluster, the etcdserver and etcdcluster match or nearly match:

$ curl http://localhost:2379/version
{"etcdserver":"3.X.Y","etcdcluster":"3.X.Y"}

OR

$ curl http://localhost:2379/version
{"etcdserver":"3.X.Y","etcdcluster":"3.X.0"}

On the second cluster, with apparently the exact same packages:

$ curl http://localhost:2379/version
{"etcdserver":"3.X.Y","etcdcluster":"3.A.B"}

X and Y match the server version installed, yet ETCD chooses a totally different etcdcluster version for 3.A.B or 3.0.0. Clearing the ETCD DB to let it reinitialize had no effect and the issue persists.

Curious, how does ETCD determine the etcdcluster version and why did it choose those above? For example, and only that, is there some other files that are perhaps checked or some package missing that forces ETCD to use a lower etcdcluster version then the etcdserver version? I've searched and read a few somewhat related posts but still can't pin point what ETCD uses to decide to use a specific etcdcluster version. ( Checked out the source as well but saw few references to etcdcluster on a quick check? )

What messages should someone also look for to determine what decision lead to the etcdcluster version selection?

Cheers,

[jmhbnz]

Hey @tomkcpr - Thanks for your question. For a 3.5.x etcd cluster you can take a look at https://github.com/etcd-io/etcd/blob/release-3.5/server/etcdserver/cluster_util.go#L167.

When you say "Clearing the ETCD DB to let it reinitialize had no effect and the issue persists." can you please confirm if the data directory for all three cluster members was wiped?

Please provide startup logs for each cluster member if you would like us to provide any more detail.

[tomkcpr]

Correct, the data directory for ETCD was wiped out on all 3 nodes.

Wouldn't be able to provide the files, sorry. Just looking for general information on the decisions that go into selecting a version to give a clue where to look and what to check.

Thank you!

[tomkcpr]

Could be reading things incorrectly so might need a bit of help. I got so far as tracing it to these sections:

https://github.com/etcd-io/etcd/blob/v3.5.7/server/etcdserver/cluster_util.go#L141

which creates the vers map, then

https://github.com/etcd-io/etcd/blob/v3.5.7/server/etcdserver/cluster_util.go#L275

which asks the member's what their version is? I might be reading something incorrectly here as I'm not a GO Lang expert however that's after the member already determined the version it wants to use it appears. In this case 3.A.B vs the 3.X.Y or 3.X.0 that the member already predetermined apparently at startup.

The part I'm interested in is how each individual member determiner's it's own version or version ranges, when starting up for the first time, before any http requests for other member's version. In other words, if I start the 3 nodes up for the fist time, without any default.etcd DB present, how and where does ETCD determine that it should use 3.A.B even before the other members query that node for the version.

Cheers,

[jmhbnz]

If I understand the question correctly I believe the initial version for an individual server instance comes from: https://github.com/etcd-io/etcd/blob/v3.5.7/api/version/version.go.

We can see that being imported here https://github.com/etcd-io/etcd/blob/v3.5.7/server/embed/etcd.go#L32. To be used when etcd prints it's version: https://github.com/etcd-io/etcd/blob/v3.5.7/server/embed/etcd.go#L308 when etcd starts an individual server as part of func StartEtcd in the same file.