Web Hook in cert-manager is not working properly. Can anyone please me out.

[minigamkreddy]

I am getting below issue while creating the self-signed issuer and a certificate resource in a test namespace.

Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Service Unavailable
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Service Unavailable

Describe the bug:
we will create a self-signed issuer and a certificate resource in a test namespace

apiVersion: v1
kind: Namespace
metadata:
  name: cert-manager-test
---
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: test-selfsigned
  namespace: cert-manager-test
spec:
  selfSigned: {}
---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: selfsigned-cert
  namespace: cert-manager-test
spec:
  dnsNames:
    - example.com
  secretName: selfsigned-cert-tls
  issuerRef:
    name: test-selfsigned**

Expected behaviour:
Need to communicate with webhook and create the self assigned certificate.

Steps to reproduce the bug:
I have followed the steps from the given below link
https://cert-manager.io/docs/installation/kubectl/

Anything else we need to know?:

Environment details::

• Kubernetes version:
  kubectl version
  Client Version: v1.28.1
  Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
  Server Version: v1.28.1

• Cloud-provider/provisioner:

• cert-manager version: ert-manager-controller:
  Image: quay.io/jetstack/cert-manager-controller:v1.14.5

• Install method:
  https://cert-manager.io/docs/installation/kubectl/

/kind bug

[hawksight]

Hey @minigamkreddy thanks for raising. This is a fairly common error to see and usually it is networking, DNS or cloud provider specific as the issue. We have a guide to help debug if you could try that first?

https://cert-manager.io/docs/troubleshooting/webhook/

Failing that, can you please share your k8s environment details.

In general every cert-manager resource if sent to the cert-manager-webhook deployment to validate the resource before it is saved to k8s to be actioned. It appears k8s cannot find that service, so please check that component is running.

[minigamkreddy]

Thanks For replying back

Yes I will Follow the link which you have provided me.

CERT MANAGER DETAILS
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get pods -n cert-manager
NAME READY STATUS RESTARTS AGE
cert-manager-7ddd8cdb9f-c7kwx 1/1 Running 1 38h
cert-manager-cainjector-57cd76c845-fk77m 1/1 Running 1 38h
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 38h

Environment Details
Kubernetes version:
kubectl version
Client Version: v1.28.1
Kustomize Version: v5.0.4-0.20230601165947-6ce0bf390ce3
Server Version: v1.28.1

OS Details
Description: Ubuntu 20.04.6 LTS

cert-manager version: ert-manager-controller:
Image: quay.io/jetstack/cert-manager-controller:v1.14.5

Note : For more Details related to Environment please reply back.

MORE DEATAILS
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get endpoints -n cert-manager cert-manager-webhook
NAME ENDPOINTS AGE
cert-manager-webhook 192.168.224.104:10250 38h

kubectl get pod -n cert-manager -l app.kubernetes.io/name=webhook
NAME READY STATUS RESTARTS AGE
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 39h

root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get pod -n cert-manager -l app.kubernetes.io/name=webhook
NAME READY STATUS RESTARTS AGE
cert-manager-webhook-cf8f9f895-n6n6q 1/1 Running 1 39h

root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl logs -n cert-manager -l app.kubernetes.io/name=webhook | head -10
W0508 09:55:18.187126 1 client_config.go:618] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0508 09:55:18.273616 1 webhook.go:129] "using dynamic certificate generating using CA stored in Secret resource" logger="cert-manager.webhook" secret_namespace="cert-manager" secret_name="cert-manager-webhook-ca"
I0508 09:55:18.273781 1 server.go:146] "listening for insecure healthz connections" logger="cert-manager" address=":6080"
I0508 09:55:18.274173 1 server.go:206] "listening for secure connections" logger="cert-manager" address=":10250"
I0508 09:55:18.303989 1 reflector.go:351] Caches populated for *v1.Secret from k8s.io/[email protected]/tools/cache/reflector.go:229
I0508 09:55:19.280437 1 dynamic_source.go:255] "Updated cert-manager TLS certificate" logger="cert-manager" DNSNames=["cert-manager-webhook","cert-manager-webhook.cert-manager","cert-manager-webhook.cert-manager.svc"]
root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms#

root@KmasterVM:/home/manoj/VM1_E810/vcsr-orch/helms# kubectl get deploy -n cert-manager cert-manager-webhook -oyaml | grep -A3 ports:
ports:
- containerPort: 10250
name: https
protocol: TCP

[minigamkreddy]

resource mapping not found for name: "cert-manager" namespace: "" from "test-resources.yaml": no matches for kind "Issuer" in version "v1"
ensure CRDs are installed first
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": proxyconnect tcp: dial tcp 10.10.224.60:3128: connect: connection refused
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": proxyconnect tcp: dial tcp 10.10.224.60:3128: connect: connection
refused
When I do the kubectl apply -f test-resources.yaml cert-manager is excepted the response from the proxy server.
10.10.224.60:3128: these should not happen cert-manager should except the response from the current cluster.

Below commands are not working
kubectl -n cert-manager port-forward deploy/cert-manager-webhook 6080

curl -sS --dump-header - 127.0.0.1:6080/healthz => These command except the response from there 10.10.224.60:3128. These is not communicated with interval cluster.

How to reslove these issue.

[hawksight]

If you run kubectl get crd do you see the cert-manager CRD resources in the cluster?
Just looking at this error:

resource mapping not found for name: "cert-manager" namespace: "" from "test-resources.yaml": no matches for kind "Issuer" in version "v1"
ensure CRDs are installed first

Perhaps the CRDs were not installed, and that could explain it. But i expect they are there otherwise there should be more failing components.

[minigamkreddy]

root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl get crd
NAME CREATED AT
bgpconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
bgppeers.crd.projectcalico.org 2023-09-05T04:25:42Z
blockaffinities.crd.projectcalico.org 2023-09-05T04:25:42Z
caliconodestatuses.crd.projectcalico.org 2023-09-05T04:25:42Z
certificaterequests.cert-manager.io 2024-05-07T14:58:36Z
certificates.cert-manager.io 2024-05-07T14:58:36Z
challenges.acme.cert-manager.io 2024-05-07T14:58:36Z
clusterinformations.crd.projectcalico.org 2023-09-05T04:25:42Z
clusterissuers.cert-manager.io 2024-05-07T14:58:36Z
felixconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
globalnetworkpolicies.crd.projectcalico.org 2023-09-05T04:25:42Z
globalnetworksets.crd.projectcalico.org 2023-09-05T04:25:42Z
hostendpoints.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamblocks.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamconfigs.crd.projectcalico.org 2023-09-05T04:25:42Z
ipamhandles.crd.projectcalico.org 2023-09-05T04:25:42Z
ippools.crd.projectcalico.org 2023-09-05T04:25:42Z
ipreservations.crd.projectcalico.org 2023-09-05T04:25:42Z
issuers.cert-manager.io 2024-05-07T14:58:36Z
kubecontrollersconfigurations.crd.projectcalico.org 2023-09-05T04:25:42Z
networkpolicies.crd.projectcalico.org 2023-09-05T04:25:42Z
networksets.crd.projectcalico.org 2023-09-05T04:25:42Z
orders.acme.cert-manager.io 2024-05-07T14:58:36Z
root@KmasterVM:/home/manoj/VM1_E810/cert-manager#

root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl apply -f test-resources.yaml
root@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl apply -f test-resources.yaml
namespace/cert-manager unchanged
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Service Unavailable
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": Service Unavailable

[minigamkreddy]

above issue got resloved. But I am seeing another issue

oot@KmasterVM:/home/manoj/VM1_E810/cert-manager# kubectl apply -f test-resources.yaml
namespace/cert-manager-test unchanged
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z
Error from server (InternalError): error when creating "test-resources.yaml": Internal error occurred: failed calling webhook "webhook.cert-manager.io": failed to call webhook: Post "https://cert-manager-webhook.cert-manager.svc:443/validate?timeout=30s": tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z

I have change the time to IST but still it doesn't work

Can you suggest me any idea how to reslove these issue.

[hawksight]

Ok so you have CRDs and you now have a new error. Previously it was connection refused but now you have "tls: failed to verify certificate: x509: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z"

Can you get the cert-manager-webhook-ca certificate out of the secret it is stored in?

k get secret -n cert-manager cert-manager-webhook-ca -o json | jq -r '.data["tls.crt"]' | base64 --decode | openssl x509 -text -noout

Assuming that cert is active (which it should be). I would validate that your server time is synced properly. This seems like your master node you are running on. Maybe try checking your server time compared to actual time to see if there is any drift?

sntp -d pool.ntp.org

For example my output seems to be:

..
+0.022888 +/- 0.025390 pool.ntp.org 162.159.200.1

Which seems very much in sync.

The concerning bit in your error is that the cert seems to be issued for the future in your case. So some time setting appears to be off: certificate has expired or is not yet valid: current time 2024-05-15T17:19:50Z is before 2024-05-15T21:01:12Z

[minigamkreddy]

Thanks,
For replying back.
The issue got resolved I am able to create the certificate.

But the Certificate which was created not able to access in different namespaces using clusterissuer as Kind.

Do you have any reference yaml file. So that I can create the certificate which can be accessible across namespaces .