☂️ [DEP-04] Implement EtcdMember Custom Resource

[shreyas-s-rao]

How to categorize this issue?

/area quality robustness usability
/kind enhancement cleanup api-change task

What would you like to be added:
Implement [DEP-04] EtcdMember Custom Resource proposal.

Task List

• Further API changes:
  • [Enhancement] Improve BackupReady conditions #618
• CRDs:
  • Write go structs, generate CRDs for EtcdMember
  • Update Etcd resource status
  • Update helm chart structure (add crds folder in single chart)
• Etcd controller
  • Create/delete EtcdMember resources during reconciliation
  • Stop creating snapshot leases; delete existing snapshot leases
  • Scale-up changes:
    • Check for PeerTLSEnabled from EtcdMember instead of member lease annotation
    • Requeue reconciliation after updating peerTLS and re-check, rather than a blocking wait in the same reconciliation
    • Don't restart pods when peerTLS is enabled/disabled
    • Don't add annotation to sts for scale-up
    • Configmap changes (naming, format)
    • Adapt etcd config generation per member
  • Feature gate to enable/disable EtcdMember functionality
• Custodian controller:
  • Update status.members by checking EtcdMember and member lease resources
  • Check etcd status.ready using pod readiness, and adapt Ready and AllMembersReady checks accordingly
  • Check BackupReady condition using EtcdMember resources instead of snapshot leases
• Compaction controller
  • Watch EtcdMember resources, with predicate to filter on changes to status.snapshots section
  • Introduce new flag for threshold on accumulated delta snapshot size, like CompactionSnapshotSizeThreshold
  • Trigger compaction job by computing difference between status.snapshots.lastDelta.endRevision and status.snapshots.lastFull.endRevision, or checking status.snapshots.accumulatedDeltaSize
• ☂️ Re-write etcd-backup-restore as etcd-steward, with EtcdMember updation capabilities etcd-steward#1
• Add upgrade tests for etcd clusters
• Add chaos tests for etcd clusters
• Document design decisions