Unable to start ETCD and failing with error 'tls: failed to find any PEM data in key input' #16024
Replies: 4 comments 14 replies
-
|
Hi @ahrtr Thanks and Regards, |
Beta Was this translation helpful? Give feedback.
-
I'd recommend that you check that your PEM files all can be properly loaded. You can also use openssl for that, it should give similar error messages. |
Beta Was this translation helpful? Give feedback.
-
|
Hey @IamSatyaonline - Please start by verifying certificates on disk as mentioned above, something like |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jmhbnz Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hey @IamSatyaonline - Thanks for narrowing it down to the file corruption. There are a ton of reasons why a file on a disk can become corrupted. My next question would be, can you replicate the issue? If you can, then please update this discussion with a detailed breakdown of what steps you took to arrive at a corrupted certificate file. |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jmhbnz Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jmhbnz This issue is coming 2 times out of 10. Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
#Hi @jmhbnz Below is the env variables we are using in our environment. The issue is rarely reproducible. As per our investigation the self sign certificates which is being used for peer communication is having incomplete data in key.pem file. which is causing this issue. So We came to the conclusion that if we use the CA signed peer certificates instead of self sign peer certificates , this issue would get fixed. Could you please confirm if that is the correct and valid fix ? Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
I think you guys have some faulty hardware at play, it seems highly unlikely that 3/10 certs would come out half-way corrupted. Can you run etcd on another server? |
Beta Was this translation helpful? Give feedback.
-
|
Hi @tjungblu Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hi @ahrtr Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Thanks, |
Beta Was this translation helpful? Give feedback.
-
|
Hi, Thanks, |
Beta Was this translation helpful? Give feedback.
-
Bug report criteria
What happened?
We have 3 members cluster of ETCD service. ETCD service is not running and throwing the error 'tls: failed to find any PEM data in key input'.
As per our prilimarliy investigation , Key files which is being used for peer communication which might be corrupted. But we are not sure about the root cause.
We are using the ETCD-3.5.7 and self sign certificates are being used for peer communication. Could you please help us to know the exact root cause of the issue.
Sharing below the log snippet with the error coming in the logs.
2023-06-02T14:58:00.721849815Z {"caller":"embed/etcd.go:484","cipher-suites":[],"message":"starting with peer TLS","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:58:00.721+00:00","tls-info":"cert = /data/fixtures/peer/cert.pem, key = /data/fixtures/peer/key.pem, client-cert=/data/fixtures/peer/cert.pem, client-key=/data/fixtures/peer/key.pem, trusted-ca = , client-cert-auth = false, crl-file = ","version":"1.2.0"}
2023-06-02T14:58:00.721912667Z {"advertise-client-urls":["https://etcd-0.etcd.spider3:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380"],"caller":"embed/etcd.go:373","data-dir":"/data","message":"closing etcd server","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:58:00.721+00:00","version":"1.2.0"}
2023-06-02T14:58:00.721960692Z {"advertise-client-urls":["https://etcd-0.etcd.spider3:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380"],"caller":"embed/etcd.go:375","data-dir":"/data","message":"closed etcd server","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:58:00.721+00:00","version":"1.2.0"}
2023-06-02T14:58:00.722031174Z {"caller":"etcdmain/etcd.go:204","error":"tls: failed to find any PEM data in key input","message":"discovery failed","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"critical","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:32\nruntime.main\n\truntime/proc.go:255","timestamp":"2023-06-02T14:58:00.721+00:00","version":"1.2.0"}
What did you expect to happen?
We want ETCD up and running without any error.
How can we reproduce it (as minimally and precisely as possible)?
It's kind of intermittent and not reproducible. always
Anything else we need to know?
No response
Etcd version (please run commands below)
Etcd configuration (command line flags or environment variables)
paste your configuration here
bash-4.4$ env
ETCD_INITIAL_CLUSTER_TOKEN=etcd
TLS_ENABLED=true
ETCD_MAX_SNAPSHOTS=3
CLIENT_PORTS=2379
TZ=UTC
HOSTNAME=etcd-0
COMPONENT_VERSION=v3.5.7
HTTP_PROBE_CMD_DIR=/usr/local/bin/health
HTTP_PROBE_READINESS_CMD_TIMEOUT_SEC=15
ETCD_LISTEN_CLIENT_URLS=https://0.0.0.0:2379
ETCD_HEARTBEAT_INTERVAL=100
ETCD_AUTO_COMPACTION_RETENTION=100
DISARM_ALARM_PEER_INTERVAL=6
ETCD_TRUSTED_CA_FILE=/data/combinedca/cacertbundle.pem
DB_THRESHOLD_PERCENTAGE=70
MONITOR_ALARM_INTERVAL=5
PEER_CERT_AUTH_ENABLED=true
TRUSTED_CA=/data/combinedca/cacertbundle.pem
PEER_CLIENTS_CERTS=/run/sec/certs/peer/srvcert.pem
FIFO_DIR=/fifo
KUBERNETES_PORT_443_TCP_PROTO=tcp
ENTRYPOINT_RESTART_ETCD=true
HTTP_PROBE_NAMESPACE=ztissan
KUBERNETES_PORT_443_TCP_ADDR=
ETCDCTL_CERT=/run/sec/certs/client/clicert.pem
ENTRYPOINT_DCED_PROCESS_INTERVAL=5
ETCD_LOG_LEVEL=debug
ENTRYPOINT_CHECKSNUMBER=60
8889
KUBERNETES_PORT=tcp://:443
POD_NAME=etcd-0
ERIC_PM_SERVER_PORT_9089_TCP_PORT=9089
ERIC_ETCD_SERVICE_PORT=2379
PWD=/
ETCD_LISTEN_PEER_URLS=https://0.0.0.0:2380
HOME=/home/dced
ERIC_ETCD_SERVICE_PORT_CLIENT_PORT_TLS=2379
ETCD_AUTO_COMPACTION_MODE=revision
KUBERNETES_SERVICE_PORT_HTTPS=443
ERIC_ETCD_PORT_2379_TCP_ADDR=
KUBERNETES_PORT_443_TCP_PORT=443
ETCD_LOGGER=zap
PEER_AUTO_TLS_ENABLED=true
ETCD_CERT_FILE=/run/sec/certs/server/srvcert.pem
ETCD_PEER_AUTO_TLS=true
ERIC_ETCD_PORT_2379_TCP_PORT=2379
KUBERNETES_PORT_443_TCP=tcp://:443
ERIC_ETCD_PORT_2379_TCP=tcp://:2379
LISTEN_PEER_URLS=https://0.0.0.0:2380
DEFRAGMENT_PERIODIC_INTERVAL=60
CONTAINER_NAME=etcd
COMPONENT=etcd
ETCD_DATA_DIR=/data
ETCD_CLIENT_CERT_AUTH=true
TERM=xterm
ETCDCTL_ENDPOINTS=etcd.ztissan:2379
HTTP_PROBE_LIVENESS_CMD_TIMEOUT_SEC=15
ETCD_METRICS=basic
PEER_CLIENT_KEY_FILE=/run/sec/certs/peer/srvprivkey.pem
HTTP_PROBE_CONTAINER_NAME=etcd
GODEBUG=tls13=1
ETCDCTL_API=3
ERIC_ETCD_PORT=tcp://:2379
ETCD_SNAPSHOT_COUNT=5000
ETCD_MAX_WALS=3
SHLVL=1
HTTP_PROBE_POD_NAME=etcd-0
KUBERNETES_SERVICE_PORT=443
ETCD_INITIAL_ADVERTISE_PEER_URLS=https://etcd-0.etcd-peer.ztissan.svc.cluster.local:2380
HTTP_PROBE_STARTUP_CMD_TIMEOUT_SEC=15
ETCD_KEY_FILE=/run/sec/certs/server/srvprivkey.pem
ETCD_ELECTION_TIMEOUT=1000
HTTP_PROBE_SERVICE_NAME=etcd
ETCDCTL_CACERT=/data/combinedca/cacertbundle.pem
ETCD_NAME=etcd-0
ETCD_QUOTA_BACKEND_BYTES=268435456
ENTRYPOINT_PIPE_TIMEOUT=5
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin
ETCD_ADVERTISE_CLIENT_URLS=https://etcd-0.etcd.ztissan:2379
DCED_PORT=2379
KUBERNETES_SERVICE_HOST=
FLAVOUR=etcd-v3.5.7-linux-amd64
ETCDCTL_KEY=/run/sec/certs/client/cliprivkey.pem
_=/usr/bin/env
Etcd debug information (please run commands below, feel free to obfuscate the IP address or FQDN in the output)
Relevant log output
2023-06-02T14:57:40.311883089Z {"message":"The current timezone is UTC","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.311+00:00","version":"1.2.0"} 2023-06-02T14:57:40.319907939Z {"message":"Startup command: /usr/local/bin/health/StartupProbe.sh","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.319+00:00","version":"1.2.0"} 2023-06-02T14:57:40.319935154Z {"message":"Startup commmand timeout value 15 seconds","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.319+00:00","version":"1.2.0"} 2023-06-02T14:57:40.320022235Z {"message":"Readiness command: /usr/local/bin/health/ReadinessProbe.sh","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.319+00:00","version":"1.2.0"} 2023-06-02T14:57:40.320095449Z {"message":"Readiness commmand timeout value 15 seconds","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.320+00:00","version":"1.2.0"} 2023-06-02T14:57:40.320247443Z {"message":"Liveness command: /usr/local/bin/health/LivenessProbe.sh","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.320+00:00","version":"1.2.0"} 2023-06-02T14:57:40.320326277Z {"message":"Liveness commmand timeout value 15 seconds","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.320+00:00","version":"1.2.0"} 2023-06-02T14:57:40.320415316Z {"message":"The http probe service 1.5.0 is ready to listen and serve on :9000","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.320+00:00","version":"1.2.0"} 2023-06-02T14:57:40.346234288Z {"message":"+ getopts cs option","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"} 2023-06-02T14:57:40.346314039Z {"message":"+ case \"${option}\" in","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"} 2023-06-02T14:57:40.346522082Z {"message":"+ CHECK=true","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"} 2023-06-02T14:57:40.346568194Z {"message":"+ getopts cs option","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"} 2023-06-02T14:57:40.346611005Z {"message":"+ case \"${option}\" in","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"} 2023-06-02T14:57:40.346647224Z {"message":"+ SUICIDE=true","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"} 2023-06-02T14:57:40.346688012Z {"message":"+ getopts cs option","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"} 2023-06-02T14:57:40.346723100Z {"message":"+ mkdir -p /data/combinedca/","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.346+00:00","version":"1.2.0"} 2023-06-02T14:57:40.425138435Z {"message":"+ [[ -v SUICIDE ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.424+00:00","version":"1.2.0"} 2023-06-02T14:57:40.425604259Z {"message":"++ ps -ef","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.425+00:00","version":"1.2.0"} 2023-06-02T14:57:40.425653943Z {"message":"++ grep watch_cert","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.425+00:00","version":"1.2.0"} 2023-06-02T14:57:40.425805950Z {"message":"++ grep -v grep","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.425+00:00","version":"1.2.0"} 2023-06-02T14:57:40.429301486Z {"message":"+ [[ -z '' ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.429+00:00","version":"1.2.0"} 2023-06-02T14:57:40.429517679Z {"message":"+ rm -f /data/combinedca/cacertbundle.pem","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.429+00:00","version":"1.2.0"} 2023-06-02T14:57:40.429618524Z {"message":"+ nohup /usr/local/bin/scripts/watch_cert.sh /run/sec/cas/clientca /run/sec/cas/pmca /run/sec/cas/siptlsca","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.429+00:00","version":"1.2.0"} 2023-06-02T14:57:40.430713026Z {"message":"+ [[ ! -s /data/combinedca/cacertbundle.pem ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.430+00:00","version":"1.2.0"} 2023-06-02T14:57:40.430801734Z {"message":"+ awk 1 /run/sec/cas/clientca/client-cacertbundle.pem /run/sec/cas/siptlsca/cacertbundle.pem","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.430+00:00","version":"1.2.0"} 2023-06-02T14:57:40.432971688Z {"message":"+ [[ ! -s /data/combinedca/cacertbundle.pem ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.432+00:00","version":"1.2.0"} 2023-06-02T14:57:40.433048788Z {"message":"+ [[ ! -s /data/combinedca/cacertbundle.pem ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.432+00:00","version":"1.2.0"} 2023-06-02T14:57:40.433118466Z {"message":"+ [[ -v CHECK ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.433+00:00","version":"1.2.0"} 2023-06-02T14:57:40.433222071Z {"message":"+ [[ ! -f /data/combinedca/cacertbundle.pem ]]","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.433+00:00","version":"1.2.0"} 2023-06-02T14:57:40.433295675Z {"message":"Setup peer certs for etcd-0 ","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.433+00:00","version":"1.2.0"} 2023-06-02T14:57:40.433391709Z {"message":"Auto TLS enable , using ETCD auto generated certs for peer communication","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.433+00:00","version":"1.2.0"} 2023-06-02T14:57:40.532124693Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.531+00:00","variable-name":"ETCD_ADVERTISE_CLIENT_URLS","variable-value":"https://etcd-0.etcd.spider3:2379","version":"1.2.0"} 2023-06-02T14:57:40.532173064Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_AUTO_COMPACTION_MODE","variable-value":"revision","version":"1.2.0"} 2023-06-02T14:57:40.532230541Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_AUTO_COMPACTION_RETENTION","variable-value":"100","version":"1.2.0"} 2023-06-02T14:57:40.532290393Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_CERT_FILE","variable-value":"/run/sec/certs/server/srvcert.pem","version":"1.2.0"} 2023-06-02T14:57:40.532348883Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_CLIENT_CERT_AUTH","variable-value":"true","version":"1.2.0"} 2023-06-02T14:57:40.532423774Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_DATA_DIR","variable-value":"/data","version":"1.2.0"} 2023-06-02T14:57:40.532461823Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_ELECTION_TIMEOUT","variable-value":"1000","version":"1.2.0"} 2023-06-02T14:57:40.532524552Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_HEARTBEAT_INTERVAL","variable-value":"100","version":"1.2.0"} 2023-06-02T14:57:40.532579293Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_INITIAL_ADVERTISE_PEER_URLS","variable-value":"https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380","version":"1.2.0"} 2023-06-02T14:57:40.532623744Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_INITIAL_CLUSTER","variable-value":"etcd-0=https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380","version":"1.2.0"} 2023-06-02T14:57:40.532681218Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_INITIAL_CLUSTER_STATE","variable-value":"new","version":"1.2.0"} 2023-06-02T14:57:40.532731491Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_INITIAL_CLUSTER_TOKEN","variable-value":"etcd","version":"1.2.0"} 2023-06-02T14:57:40.532780027Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_KEY_FILE","variable-value":"/run/sec/certs/server/srvprivkey.pem","version":"1.2.0"} 2023-06-02T14:57:40.532836995Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_LISTEN_CLIENT_URLS","variable-value":"https://0.0.0.0:2379","version":"1.2.0"} 2023-06-02T14:57:40.532902896Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_LISTEN_PEER_URLS","variable-value":"https://0.0.0.0:2380","version":"1.2.0"} 2023-06-02T14:57:40.532956146Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_LOG_LEVEL","variable-value":"info","version":"1.2.0"} 2023-06-02T14:57:40.533002550Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.532+00:00","variable-name":"ETCD_LOGGER","variable-value":"zap","version":"1.2.0"} 2023-06-02T14:57:40.533065561Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_MAX_SNAPSHOTS","variable-value":"3","version":"1.2.0"} 2023-06-02T14:57:40.533109094Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_MAX_WALS","variable-value":"3","version":"1.2.0"} 2023-06-02T14:57:40.533156961Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_METRICS","variable-value":"basic","version":"1.2.0"} 2023-06-02T14:57:40.533206629Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_NAME","variable-value":"etcd-0","version":"1.2.0"} 2023-06-02T14:57:40.533252146Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_PEER_AUTO_TLS","variable-value":"true","version":"1.2.0"} 2023-06-02T14:57:40.533316465Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_QUOTA_BACKEND_BYTES","variable-value":"268435456","version":"1.2.0"} 2023-06-02T14:57:40.533377769Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_SNAPSHOT_COUNT","variable-value":"5000","version":"1.2.0"} 2023-06-02T14:57:40.533414350Z {"caller":"flags/flag.go:113","message":"recognized and used environment variable","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","variable-name":"ETCD_TRUSTED_CA_FILE","variable-value":"/data/combinedca/cacertbundle.pem","version":"1.2.0"} 2023-06-02T14:57:40.533524832Z {"args":["/usr/local/bin/etcd"],"caller":"etcdmain/etcd.go:73","message":"Running: ","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"} 2023-06-02T14:57:40.533607997Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"auth_successful","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"} 2023-06-02T14:57:40.533677042Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"cert_watcher.txt","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"} 2023-06-02T14:57:40.533825720Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"combinedca","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"} 2023-06-02T14:57:40.533895896Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"etcd.liveness","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"} 2023-06-02T14:57:40.534042861Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"fixtures","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.533+00:00","version":"1.2.0"} 2023-06-02T14:57:40.534098248Z {"caller":"etcdmain/etcd.go:446","data-dir":"/data","filename":"lost+found","message":"found invalid file under data directory","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"warning","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"} 2023-06-02T14:57:40.534163014Z {"caller":"etcdmain/etcd.go:116","data-dir":"/data","dir-type":"member","message":"server has been already initialized","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"} 2023-06-02T14:57:40.534209953Z {"caller":"embed/etcd.go:124","listen-peer-urls":["https://0.0.0.0:2380"],"message":"configuring peer listeners","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"} 2023-06-02T14:57:40.534339542Z {"caller":"embed/etcd.go:484","cipher-suites":[],"message":"starting with peer TLS","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.534+00:00","tls-info":"cert = /data/fixtures/peer/cert.pem, key = /data/fixtures/peer/key.pem, client-cert=/data/fixtures/peer/cert.pem, client-key=/data/fixtures/peer/key.pem, trusted-ca = , client-cert-auth = false, crl-file = ","version":"1.2.0"} 2023-06-02T14:57:40.534429623Z {"advertise-client-urls":["https://etcd-0.etcd.spider3:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380"],"caller":"embed/etcd.go:373","data-dir":"/data","message":"closing etcd server","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"} 2023-06-02T14:57:40.534508748Z {"advertise-client-urls":["https://etcd-0.etcd.spider3:2379"],"advertise-peer-urls":["https://etcd-0.etcd-peer.spider3.svc.cluster.local:2380"],"caller":"embed/etcd.go:375","data-dir":"/data","message":"closed etcd server","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"name":"etcd-0","service_id":"etcd","severity":"info","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"} 2023-06-02T14:57:40.534567414Z {"caller":"etcdmain/etcd.go:204","error":"tls: failed to find any PEM data in key input","message":"discovery failed","metadata":{"container_name":"etcd","namespace":"spider3","pod_name":"etcd-0"},"service_id":"etcd","severity":"critical","stacktrace":"go.etcd.io/etcd/server/v3/etcdmain.startEtcdOrProxyV2\n\tgo.etcd.io/etcd/server/v3/etcdmain/etcd.go:204\ngo.etcd.io/etcd/server/v3/etcdmain.Main\n\tgo.etcd.io/etcd/server/v3/etcdmain/main.go:40\nmain.main\n\tgo.etcd.io/etcd/server/v3/main.go:32\nruntime.main\n\truntime/proc.go:255","timestamp":"2023-06-02T14:57:40.534+00:00","version":"1.2.0"}Beta Was this translation helpful? Give feedback.
All reactions