Kubernetes - Cert-manager challenges failed to get let'sencrypt signed certificates, propagation check failed (wrong status code 400, expected 200)

[Maximus-IT]

Hi,

I'm using cert-manager(helm deployment) in Kubernetes cluster context to get certificates signed automatically by letsencrypt, for my wikijs application behind an ingress-nginx service. I can't join my service wikijs in HTTP or HTTPS anymore.

First step

After an automatic update of ingress-nginx, my service wikijs did'nt work anymore with https. My cert-manager service was stucked in "CrashLoopBackOff" statement.

First error in cert-manager log (after Ingress-nginx update to 1.8.1 version) :

Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource

Second step

So i performed an update of each service to fix this issue and followed prerequisites here for compatibility : https://github.com/kubernetes/ingress-nginx

Cert-manager started correctly and "CrashLoopBackOff" statement disappeared. So first error was fixed.

Old versions:
K8s : 1.19.0
Cert-manager : 1.3.0
ingress-Nginx : 0.7.0 (not sure about that because i used https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/static/provider/cloud/deploy.yaml)

New Versions:
K8s : 1.24.13-2
Cert-manager : 1.12.0
Ingress-Nginx : 1.8.1 (auto, i did nothing. Maybe the hoster did an update of K8s or something without warn me)
But i had new errors after updating K8s and Cert-Manager.

Cert-manager logs :

"cert-manager/challenges: propagation check failed" err="wrong status code '400', expected '200'"
"cert-manager/challenges/http01/selfCheck/http01/ensurePod: found one existing HTTP01 solver pod [...]"
It seems like my previous certificates and settings was not deleted. So i deleted all and verify with : kubectl get Issuers,ClusterIssuers,Certificates,CertificateRequests,Orders,Challenges --all-namespaces

Third step

I decided to reinstall from scratch my cert-manager service with these command lines :

Delete my issuer and my ingress :

kubectl delete -f wikijs-issuer-prod.yml
kubectl delete -f wikijs-ingress-prod.yml

Uninstall cert-manager :

helm --namespace cert-manager delete cert-manager
kubectl delete namespace cert-manager

Install cert-manager :

helm repo add jetstack https://charts.jetstack.io
helm repo update
kubectl apply -f https://github.com/cert-manager/cert-manager/releases/download/v1.12.0/cert-manager.crds.yaml
helm install \
  cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.12.0

Apply again my Issuer :

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-issuer
  namespace: wikijs-prod
spec:
  acme:
    email: xxxxxxxx@xxxxxx
    server: https://acme-v02.api.letsencrypt.org/directory
    privateKeySecretRef:
      name: key-cert-cm-issuer-prod
    solvers:
    - http01:
        ingress:
          class: nginx

Apply again my ingress :

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: wikijs
  labels:
    app.kubernetes.io/name: wikijs
    app.kubernetes.io/instance: wikijs
  annotations:
    app.kubernetes.io/name: wikijs
    app.kubernetes.io/instance: wikijs
    app.kubernetes.io/app: wikijs
    kubernetes.io/ingress.class: nginx
    cert-manager.io/issuer: letsencrypt-issuer
    acme.cert-manager.io/http01-edit-in-place: "true"
    nginx.ingress.kubernetes.io/from-to-www-redirect: "true"
spec:
  tls:
    - hosts:
        - domain.tld
      secretName: cert-tls-prod-domain.tld
  rules:
    - host: domain.tld
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: wikijs
                port: 
                  number: 80

Restart my ingress-nginx :
kubectl rollout restart deploy/ingress-nginx-controller -n ingress-nginx

And errors still subsists :

Cert-manager logs:

"cert-manager/challenges: propagation check failed" err="wrong status code '400', expected '200'"
"cert-manager/challenges/http01/selfCheck/http01/ensurePod: found one existing HTTP01 solver pod [...]"

Error in browser : ERR_SSL_PROTOCOL_ERROR

What i'm doing wrong ?

Thanks for your help !