PD crash loops during VolumeReplace due to bad member entry.

[anish-db]

Bug Report

What version of Kubernetes are you using?
v1.30.4

What version of TiDB Operator are you using?
1.5.4

What did you do?
Requested a pd storage class change using VolumeReplace feature in tidb-operator with
.spec.pd.spareVolReplaceReplicas = 3
Cluster normall has 3 replicsa (0,1,2), with spare vol replace, temporarily adds 3,4,5.

What did you expect to see?
Expected to see volume replace to happen successfully

What did you see instead?
Some PD's crash looping due to an invalid member in PD's etcd group.

Logs from crash looping PD:

Here's the output of pd-ctl member when the pd pod was crash looping:

{
  "header": {
    "cluster_id": 7332562777068724797
  },
  "members": [
    {
      "member_id": 805731620209838514,
      "peer_urls": [
        "https://devdev-pd-5.devdev-pd-peer.tidb.svc:2380"
      ]
    },
    {
      "name": "devdev-pd-2",
      "member_id": 3664288931222184044,
      "peer_urls": [
        "https://devdev-pd-2.devdev-pd-peer.tidb.svc:2380"
      ],
      "client_urls": [
        "https://devdev-pd-2.devdev-pd-peer.tidb.svc:2379"
      ],
      "deploy_path": "/",
      "binary_version": "v6.5.11",
      "git_hash": "86ddba17874d642e1ad3ec6cce47462b0e366ced"
    },
    {
      "name": "devdev-pd-1",
      "member_id": 12706059420938024546,
      "peer_urls": [
        "https://devdev-pd-1.devdev-pd-peer.tidb.svc:2380"
      ],
      "client_urls": [
        "https://devdev-pd-1.devdev-pd-peer.tidb.svc:2379"
      ],
      "deploy_path": "/",
      "binary_version": "v6.5.11",
      "git_hash": "86ddba17874d642e1ad3ec6cce47462b0e366ced"
    }
  ],
  "leader": {
    "name": "devdev-pd-2",
    "member_id": 3664288931222184044,
    "peer_urls": [
      "https://devdev-pd-2.devdev-pd-peer.tidb.svc:2380"
    ],
    "client_urls": [
      "https://devdev-pd-2.devdev-pd-peer.tidb.svc:2379"
    ],
    "deploy_path": "/",
    "binary_version": "v6.5.11",
    "git_hash": "86ddba17874d642e1ad3ec6cce47462b0e366ced"
  },
  "etcd_leader": {
    "name": "devdev-pd-2",
    "member_id": 3664288931222184044,
    "peer_urls": [
      "https://devdev-pd-2.devdev-pd-peer.tidb.svc:2380"
    ],
    "client_urls": [
      "https://devdev-pd-2.devdev-pd-peer.tidb.svc:2379"
    ],
    "deploy_path": "/",
    "binary_version": "v6.5.11",
    "git_hash": "86ddba17874d642e1ad3ec6cce47462b0e366ced"
  }
}

The "bad" entry was:

    {
      "member_id": 805731620209838514,
      "peer_urls": [
        "https://devdev-pd-5.devdev-pd-peer.tidb.svc:2380"
      ]
    },

Solution was to manually cleanup with pd-ctl member delete id 805731620209838514
Similar issue to #126

I suspect some pd started for spare replace replicas is being turnedup disk-replaced and deleted for scale down too quickly before it is ready. The repro is not deterministic and only happens some of the times so there is some race condition.

[nolouch]

It is like #126, we have the optimization by tikv/pd#1279. it records the state in the join file. I wonder if the problem is the devdev-pd-5 missing the join file and then pd-5 cannot rejoin or pd-5 is missing abnormally before it join success..