Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'

[hahahannes]

Did someone have the problem where the ACME challenge URL is accessible from outside (resulting in a 200 in the web server logs) but the self check shows 404 (resulting in a 404 in the web serer logs for the exact same path)?

[thurianknight]

I am running into this issue (or something very close) today. I deployed 3 new sets of resources, with their own ingresses and certificates. 2 of them worked perfectly, but 1 if them is hung up on that 404 error from the solver.

The solver pod is running of course. I can access it via curl and reproduce the 404 error like this:
curl -v http://172.20.0.xxx:8089/.well-known/acme-challenge/THIS_IS_MY_TOKEN

And the results:

*   Trying 172.20.0.xxx:8089...
* TCP_NODELAY set
* Connected to 172.20.0.xxx (172.20.0.xxx) port 8089 (#0)
> GET /.well-known/acme-challenge/THIS_IS_MY_TOKEN
> Host: 172.20.0.xxx:8089
> User-Agent: curl/7.68.0
> Accept: */*
> 
* Mark bundle as not supporting multiuse
< HTTP/1.1 404 Not Found
< Content-Type: text/plain; charset=utf-8
< X-Content-Type-Options: nosniff
< Date: Tue, 07 May 2024 15:38:27 GMT
< Content-Length: 19
< 
404 page not found

Going through the public ingress gives the same 404 result.

Describing the challenge gives this status and events:

Status:
  Presented:   true
  Processing:  true
  Reason:      Waiting for HTTP-01 challenge propagation: wrong status code '404', expected '200'
  State:       pending
Events:
  Type    Reason     Age   From                     Message
  ----    ------     ----  ----                     -------
  Normal  Started    50m   cert-manager-challenges  Challenge scheduled for processing
  Normal  Presented  50m   cert-manager-challenges  Presented challenge using HTTP-01 challenge mechanism

As mentioned, I deployed 3 sites within moments of each other, and 2 of them worked perfectly. And this system overall has been working perfectly for over 6 months. This is the first time I have encountered this issue.

I have tried deleting the order, challenge, solver pod and ingress and recreating the ingress. This recreated all the other resources in turn, as expected... with the same 404 error coming from the solver pod.

The solver pod has this in the log:
I0507 15:47:42.695262 1 solver.go:87] "cert-manager/acmesolver: got successful challenge request, writing key" host="my.domain.com" path="/.well-known/acme-challenge/THIS_IS_MY_TOKEN" base_path="/.well-known/acme-challenge" token="THIS_IS_MY_TOKEN"

I realize that the solver pods are running a very stripped down Linux "non-distro", so I'm not able to run any commands against it like env or cat /var/log or anything else that I have tried.

Anyone have advice for how to troubleshoot/resolve this?

[thurianknight]

And after re-reading more closely, I realize that the original poster's issue was different than mine. I will post mine as a new issue.