Unable to update one of my subdomains since 2.11.4

[inkanyamba]

Issue Details

I wasn't able to access one of my subdomains after updating to 2.11.4.
Debugged with Claude so not sure how accurate the issue is but debugged for a longer than I'd care to admit before finally going back to 2.11.3.

Perhaps some literal embodied quote characters causing some issues. e.g. "OQJskOaeWRBc82svCMEYT99uN0gYhadxU5vscYMBibw" instead of OQJskOaeWRBc82svCMEYT99uN0gYhadxU5vscYMBibw

Hope I don't get crucified for opening up this issue but just in case anyone else faces the same issue hopefully this helps. Below is a dump of the write up Claude did regarding the issue... I went through it's basically the same as everything I'd done with it during my attempt to debug. Hope it helps with getting a patch for the issue.

Summary

When Caddy creates the _acme-challenge TXT record for a DNS-01 challenge via this module, the record's content field on Cloudflare ends up with literal double-quote characters wrapped around the actual token value, e.g.:

"OQJskOaeWRBc82svCMEYT99uN0gYhadxU5vscYMBibw"

instead of the expected:

OQJskOaeWRBc82svCMEYT99uN0gYhadxU5vscYMBibw

Because Caddy's own propagation check (certmagic.solverWrapper) compares the literal expected token against what it resolves over DNS, this mismatch means the check can never succeed, regardless of resolver used or how long propagation_timeout is set. Every obtain/renew attempt fails after the timeout with:

solving challenges: waiting for solver certmagic.solverWrapper to be ready:
timed out waiting for record to fully propagate; verify DNS provider
configuration is correct

Environment

• Caddy version: 2.11.4 (image: ghcr.io/caddybuilds/caddy-cloudflare:2.11.4)
• Module version: dns.providers.cloudflare v0.2.4 (per caddy list-modules --versions)
• DNS provider: Cloudflare (single API token, Zone:DNS:Edit + Zone:Read scoped)
• Caddyfile relevant snippet:

caddyfile

  (cf_dns) {
      tls {
          dns cloudflare {env.CLOUDFLARE_API_TOKEN}
          resolvers 1.1.1.1 1.0.0.1
      }
  }

  example.sub.yourdomain.com {
      import cf_dns
      reverse_proxy 127.0.0.1:1234
  }

Steps to reproduce

1. Configure a site with the Cloudflare DNS-01 challenge as above.
2. Trigger a certificate obtain (e.g. fresh container start for a domain with no existing cert).
3. While the attempt is in progress, query Cloudflare's API directly for the created record:

   curl -s "https://api.cloudflare.com/client/v4/zones/<zone_id>/dns_records?type=TXT&name=_acme-challenge.example.sub.yourdomain.com" \
     -H "Authorization: Bearer <token>"

4. Observe the content field includes literal \" characters wrapping the value.

Evidence

For comparison, here are three TXT records that existed simultaneously at the same name (_acme-challenge.<subdomain>.yourdomain.com) during debugging:

Source | content field (raw, as returned by Cloudflare API) | ttl -- | -- | -- Manual curl -X POST (plain content, no quotes in payload) | manual-test-value-12345 | 120 Caddy ACME attempt #1 | "OQJskOaeWRBc82svCMEYT99uN0gYhadxU5vscYMBibw" | 1 Caddy ACME attempt #2 | "F3v_kA6KMzm88n5gooov9aH26bzKwhVu7q7SepNQQY0" | 1

The manually-created record (content submitted with no surrounding quotes) was stored and returned by Cloudflare exactly as submitted — no quotes added. Both Caddy-created records, from the same token and same zone, came back with literal quote characters baked into the content.

This was not a propagation/caching/resolver issue — ruled out by:

• Setting explicit resolvers 1.1.1.1 1.0.0.1 in the tls block (no change in behavior)
• Querying the zone's authoritative nameserver directly (jake.ns.cloudflare.com) during an active attempt — record never matched because of the quote mismatch, not because it was absent
• Confirming via the Cloudflare API directly that the record existed in the zone the entire time, just with malformed content

Expected behavior

The TXT record content written to Cloudflare should be the raw challenge token, with no additional quote characters, matching what a manual API call with the same payload produces.

Actual behavior

The module (or an underlying dependency, possibly related to the recent libdns 1.0 API migration) appears to be double-wrapping the content value in quotes before submitting it to Cloudflare's API, resulting in a permanently broken DNS-01 challenge.

Regression window / workaround

This appears to be a regression introduced between the module version bundled with Caddy 2.11.3 and 2.11.4. Pinning to the 2.11.3-based image resolves the issue completely with an otherwise identical Caddyfile and Cloudflare token. I have not yet identified the exact caddy-dns/cloudflare (or libdns/cloudflare) version bundled with the 2.11.3 build for direct comparison — happy to pull that if useful for triage.

Additional context

Cloudflare's own API/dashboard does have a known general quirk around auto-wrapping TXT content in quotes in some circumstances (acknowledged in their own UI warning, and reported against several other tools: terraform-provider-cloudflare#4650, certbot#10048). That said, the fact that a manually-submitted unquoted payload was stored unquoted, while the module's own writes were quoted, suggests the quoting is happening on the request-construction side here rather than purely a Cloudflare-side inconsistency — but I'm not certain of the exact mechanism without digging into the module's source.

Summary

When Caddy creates the _acme-challenge TXT record for a DNS-01 challenge via this module, the record's content field on Cloudflare ends up with literal double-quote characters wrapped around the actual token value, e.g.:

"OQJskOaeWRBc82svCMEYT99uN0gYhadxU5vscYMBibw"

instead of the expected:

OQJskOaeWRBc82svCMEYT99uN0gYhadxU5vscYMBibw

Because Caddy's own propagation check (certmagic.solverWrapper) compares the literal expected token against what it resolves over DNS, this mismatch means the check can never succeed, regardless of resolver used or how long propagation_timeout is set. Every obtain/renew attempt fails after the timeout with:

solving challenges: waiting for solver certmagic.solverWrapper to be ready:
timed out waiting for record to fully propagate; verify DNS provider
configuration is correct

Environment

Caddy version: 2.11.4 (image: ghcr.io/caddybuilds/caddy-cloudflare:2.11.4)
Module version: dns.providers.cloudflare v0.2.4 (per caddy list-modules --versions)
DNS provider: Cloudflare (single API token, Zone:DNS:Edit + Zone:Read scoped)
Caddyfile relevant snippet:

caddyfile (cf_dns) {
tls {
dns cloudflare {env.CLOUDFLARE_API_TOKEN}
resolvers 1.1.1.1 1.0.0.1
}
}

example.sub.yourdomain.com {
import cf_dns
reverse_proxy 127.0.0.1:1234
}

Steps to reproduce

Configure a site with the Cloudflare DNS-01 challenge as above.
Trigger a certificate obtain (e.g. fresh container start for a domain with no existing cert).
While the attempt is in progress, query Cloudflare's API directly for the created record:

curl -s "https://api.cloudflare.com/client/v4/zones/<zone_id>/dns_records?type=TXT&name=_acme-challenge.example.sub.yourdomain.com"
-H "Authorization: Bearer "

Observe the content field includes literal " characters wrapping the value.

Evidence

For comparison, here are three TXT records that existed simultaneously at the same name (_acme-challenge..yourdomain.com) during debugging:

Sourcecontent field (raw, as returned by Cloudflare API)ttlManual curl -X POST (plain content, no quotes in payload)manual-test-value-12345120Caddy ACME attempt #1"OQJskOaeWRBc82svCMEYT99uN0gYhadxU5vscYMBibw"1Caddy ACME attempt #2"F3v_kA6KMzm88n5gooov9aH26bzKwhVu7q7SepNQQY0"1

The manually-created record (content submitted with no surrounding quotes) was stored and returned by Cloudflare exactly as submitted — no quotes added. Both Caddy-created records, from the same token and same zone, came back with literal quote characters baked into the content.

This was not a propagation/caching/resolver issue — ruled out by:

Setting explicit resolvers 1.1.1.1 1.0.0.1 in the tls block (no change in behavior)
Querying the zone's authoritative nameserver directly (jake.ns.cloudflare.com) during an active attempt — record never matched because of the quote mismatch, not because it was absent
Confirming via the Cloudflare API directly that the record existed in the zone the entire time, just with malformed content

Expected behavior

The TXT record content written to Cloudflare should be the raw challenge token, with no additional quote characters, matching what a manual API call with the same payload produces.

Actual behavior

The module (or an underlying dependency, possibly related to the recent libdns 1.0 API migration) appears to be double-wrapping the content value in quotes before submitting it to Cloudflare's API, resulting in a permanently broken DNS-01 challenge.

Regression window / workaround

This appears to be a regression introduced between the module version bundled with Caddy 2.11.3 and 2.11.4. Pinning to the 2.11.3-based image resolves the issue completely with an otherwise identical Caddyfile and Cloudflare token. I have not yet identified the exact caddy-dns/cloudflare (or libdns/cloudflare) version bundled with the 2.11.3 build for direct comparison — happy to pull that if useful for triage.

Additional context

Cloudflare's own API/dashboard does have a known general quirk around auto-wrapping TXT content in quotes in some circumstances (acknowledged in their own UI warning, and reported against several other tools: terraform-provider-cloudflare#4650, certbot#10048). That said, the fact that a manually-submitted unquoted payload was stored unquoted, while the module's own writes were quoted, suggests the quoting is happening on the request-construction side here rather than purely a Cloudflare-side inconsistency — but I'm not certain of the exact mechanism without digging into the module's source.

Assistance Disclosure

AI used

If AI was used, describe the extent to which it was used.

used Claude to debug the issue and write up parts of the issue

[mohammed90]

Ideally, we need to be able to reproduce the bug in the most minimal way possible. This allows us to write regression tests to verify the fix is working. If we can't reproduce it, then you'll have to test our changes for us until it's fixed -- and then we can't add test cases, either.

I've attached a template below that will help make this easier and faster! This will require some effort on your part -- please understand that we will be dedicating time to fix the bug you are reporting if you can just help us understand it and reproduce it easily.

This template will ask for some information you've already provided; that's OK, just fill it out the best you can. 👍 I've also included some helpful tips below the template. Feel free to let me know if you have any questions!

Thank you again for your report, we look forward to resolving it!

Template

## 1. Environment

### 1a. Operating system and version

```
paste here
```


### 1b. Caddy version (run `caddy version` or paste commit SHA)

```
paste here
```


### 1c. Go version (if building Caddy from source; run `go version`)

```
paste here
```


## 2. Description

### 2a. What happens (briefly explain what is wrong)




### 2b. Why it's a bug (if it's not obvious)




### 2c. Log output

```
paste terminal output or logs here
```



### 2d. Workaround(s)




### 2e. Relevant links




## 3. Tutorial (minimal steps to reproduce the bug)

Helpful tips

1. Environment: Please fill out your OS and Caddy versions, even if you don't think they are relevant. (They are always relevant.) If you built Caddy from source, provide the commit SHA and specify your exact Go version.

2. Description: Describe at a high level what the bug is. What happens? Why is it a bug? Not all bugs are obvious, so convince readers that it's actually a bug.

   • 2c) Log output: Paste terminal output and/or complete logs in a code block. DO NOT REDACT INFORMATION except for credentials.
   • 2d) Workaround: What are you doing to work around the problem in the meantime? This can help others who encounter the same problem, until we implement a fix.
   • 2e) Relevant links: Please link to any related issues, pull requests, docs, and/or discussion. This can add crucial context to your report.

3. Tutorial: What are the minimum required specific steps someone needs to take in order to experience the same bug? Your goal here is to make sure that anyone else can have the same experience with the bug as you do. You are writing a tutorial, so make sure to carry it out yourself before posting it. Please:

   • Start with an empty config. Add only the lines/parameters that are absolutely required to reproduce the bug.
   • Do not run Caddy inside containers.
   • Run Caddy manually in your terminal; do not use systemd or other init systems.
   • If making HTTP requests, avoid web browsers. Use a simpler HTTP client instead, like curl.
   • Do not redact any information from your config (except credentials). Domain names are public knowledge and often necessary for quick resolution of an issue!
   • Note that ignoring this advice may result in delays, or even in your issue being closed. 😞 Only actionable issues are kept open, and if there is not enough information or clarity to reproduce the bug, then the report is not actionable.

Example of a tutorial:

Create a config file:

{ ... }

Open terminal and run Caddy:

$ caddy ...

Make an HTTP request:

$ curl ...

Notice that the result is ___ but it should be ___.