Certificate rejected with Etcd 3.2.9 and 3.3.0-rc.2

[novinfor]

Hi,

I'm having the following issues with Etcd certificate generated by Tack (cfssl).

make all fails on ❤ Curling apiserver external elb.

Cluster was started with Etcd version 3.2.9 which reports the following errors:

Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.378578 I | etcdmain: etcd Version: 3.2.9
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.378592 I | etcdmain: Git SHA: f1d7dd8
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.378602 I | etcdmain: Go Version: go1.8.4
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.378611 I | etcdmain: Go OS/Arch: linux/amd64
...
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.378724 I | embed: peerTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = false
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.471926 I | embed: ClientTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = true
...
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.618877 I | etcdmain: rejected connection from "10.0.10.11:54576" (remote error: tls: bad certificate)
Jan 15 12:22:39 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1053]: 2018-01-15 12:22:39.628363 I | etcdmain: rejected connection from "10.0.10.12:50336" (remote error: tls: bad certificate)

After enforcing the Etcd latest version by adding the ETCD_IMAGE_TAG environment variable to modules/etcd/cloud-config.yml and modules/worker/cloud-config.yml:

Environment="ETCD_IMAGE_TAG=v3.3"

The error message changed slightly:

Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.256196 I | etcdmain: etcd Version: 3.3.0-rc.2
Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.256208 I | etcdmain: Git SHA: 9e079d8f0
Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.256217 I | etcdmain: Go Version: go1.9.2
Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.256225 I | etcdmain: Go OS/Arch: linux/amd64
...
Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.256335 I | embed: peerTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = false, crl-file =
Jan 15 12:05:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:10.340613 I | embed: ClientTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = true, crl-file =
...
Jan 15 12:05:12 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:12.656516 I | embed: rejected connection from "10.0.10.11:42408" (error "remote error: tls: bad certificate", ServerName "test.kz8s")
Jan 15 12:05:12 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1045]: 2018-01-15 12:05:12.664169 I | embed: rejected connection from "10.0.10.12:51198" (error "remote error: tls: bad certificate", ServerName "test.kz8s")

but the problem persists :(

Etcd version 3.1.11 (ETCD_IMAGE_TAG=v3.1.11) is working fine and cluster is started successfully.

Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.529288 I | etcdmain: etcd Version: 3.1.11
Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.529299 I | etcdmain: Git SHA: 960f460
Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.529307 I | etcdmain: Go Version: go1.8.5
Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.529315 I | etcdmain: Go OS/Arch: linux/amd64
...
Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.529389 I | embed: peerTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = false
Jan 15 11:40:05 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:05.617281 I | embed: ClientTLS: cert = /etc/ssl/certs/k8s-etcd.pem, key = /etc/ssl/certs/k8s-etcd-key.pem, ca = , trusted-ca = /etc/ssl/certs/ca.pem, client-cert-auth = true
...
Jan 15 11:40:10 ip-10-0-10-10.us-west-2.compute.internal etcd-wrapper[1074]: 2018-01-15 11:40:10.339317 I | embed: ready to serve client requests

Versions used:

❤ checking prerequisities

# aws --version
aws-cli/1.11.13 Python/3.5.2 Linux/4.9.49-moby botocore/1.4.70

# cfssl version
Version: 1.2.0
Revision: dev
Runtime: go1.6.2

# jq --version
jq-1.5-1-a5b5cbe

# kubectl version --client
Client Version: version.Info{Major:"1", Minor:"9", GitVersion:"v1.9.1", GitCommit:"3a1c9449a956b6026f075fa3134ff92f7d55f812", GitTreeState:"clean", BuildDate:"2018-01-04T11:52:23Z", GoVersion:"go1.9.2", Compiler:"gc", Platform:"linux/amd64"}

# terraform --version
Terraform v0.11.2

and Tack 10bc698.

Thanks for checking this issue.

[jacohend]

@novinfor I'm getting this with kube-aws too...I wonder if this is an etcd issue, or at least somewhere upstream of this?