Using certbot to renew certificates using dns-01 challenge. The machine that’s running certbot client is using cloudflared as a dns-proxy for DoH.

In the process of renewing a certificate, certbot inserts a TXT record _acme-challenge.host.domain using the Cloudflare API. This record is correctly created.

Certbot then proceeds to poll for this value to ensure it’s queryable before asking the certificate provider to generate a certificate. The first query for this record results in an NXDOMAIN result since the record has not yet propagated to queryable DNS servers.

Subsequent polls, are getting a cached negative result. You can see the TTL decreasing on manual queries. The initial TTL on the negative result appears to be 30 minutes.
Certbot times-out after 5 minutes of polling for the record, so certificate renewal never succeeds. Restarting the cloudflared process clears the cache and later queries successfully get the correct TXT record result.

I don’t know whether this is a change to cloudflared or whether upstream authoritative servers are now sending SOA TTLs for negative responses resulting in cloudflared caching the negative result. The effect is that using certbot with dns-01 challenge, via Cloudflare + cloudflared proxy-dns does not work.

This machine had been renewing certificates happily. The most recent renewal failed and the certificate expired. Suggesting that something changed within the last 3 months. The version of cloudflared being used is from Aug 22.

Note: I’m actually using the built-in certificate issuance from Caddy, but used certbot here as a more familiar term.
Also posted here: https://community.cloudflare.com/t/cloudflared-proxy-dns-is-caching-negative-responses/414454