Challenge Fails for Letsencrypt/Cloudflare DNS01 #4995

rwlove · 2022-03-28T20:22:58Z

rwlove
Mar 28, 2022

Hi community. I’ve been struggling with cert-manager/letsencrypt/cloudflare for about a week now and I cannot seem to figure out what is going on. I had this working, but I must have had some transient setting as when I rebuilt my cluster it stopped working.

I know this post is very verbose, but I’m not sure where the problem is.

From what I can tell I end up with two challenges and they don’t complete because they cannot verify my domain with cloudflare.

I’m fairly certain my token and token settings are valid because I run the suggested curl command to validate the token successfully and I use the same token for external-dns and I can see entries in cloudflare for my selected ingresses/services.

➜  ~ curl -X GET "https://api.cloudflare.com/client/v4/user/tokens/verify" \
     -H "Authorization: Bearer <redacted>" \
     -H "Content-Type:application/json"
{"result":{"id":"f989f91a5644a18ce3cc92e325805714","status":"active"},"success":true,"errors":[],"messages":[{"code":10000,"message":"This API Token is valid and active","type":null}]}%

Verbose cert-manager logs show:

I0328 20:28:03.462870       1 reflector.go:382] external/io_k8s_client_go/tools/cache/reflector.go:167: forcing resync
I0328 20:28:03.949337       1 scheduler.go:132] cert-manager/controller/challenges/challenge-scheduler "msg"="there is already a challenge processing with this domain"  "domain"="foo.com" "type"="DNS-01"

Any suggestions on how to debug this would be greatly appreciated!!!

Following this flow:

https://cert-manager.io/docs/faq/troubleshooting/

Certificate -> Certificate Request -> Order -> Challenge

Certificate

➜  fleet-infra git:(main) ✗ kubectl -n network get certificate 
NAME                 READY   SECRET                   AGE
foo-com   False   foo-com-tls   4h43m

Spec:
  Common Name:  foo.com
  Dns Names:
    foo.com
    *.foo.com
  Issuer Ref:
    Kind:       ClusterIssuer
    Name:       letsencrypt-production
  Secret Name:  foo-com-tls
  Secret Template:
    Annotations:
      reflector.v1.k8s.emberstack.com/reflection-allowed:             true
      reflector.v1.k8s.emberstack.com/reflection-allowed-namespaces:  home,media,monitoring,storage,flux-system,kube-system
      reflector.v1.k8s.emberstack.com/reflection-auto-enabled:        true
      reflector.v1.k8s.emberstack.com/reflection-auto-namespaces:     home,media,monitoring,storage,flux-system,kube-system
Status:
  Conditions:
    Last Transition Time:        2022-03-28T15:07:28Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      False
    Type:                        Ready
    Last Transition Time:        2022-03-28T15:07:28Z
    Message:                     Issuing certificate as Secret does not exist
    Observed Generation:         1
    Reason:                      DoesNotExist
    Status:                      True
    Type:                        Issuing
  Next Private Key Secret Name:  foo-com-dhv77
Events:                          <none>

CertificateRequest

➜  fleet-infra git:(main) ✗ kubectl -n network get certificaterequests.cert-manager.io
NAME                       APPROVED   DENIED   READY   ISSUER                   REQUESTOR                                         AGE
foo-com-4kn9t   True                False   letsencrypt-production   system:serviceaccount:cert-manager:cert-manager   4h49m

Spec:
  Extra:
    authentication.kubernetes.io/pod-name:
      cert-manager-64f4cd78bf-skjg7
    authentication.kubernetes.io/pod-uid:
      8389639a-ea42-4519-b650-6a26be7460d3
  Groups:
    system:serviceaccounts
    system:serviceaccounts:cert-manager
    system:authenticated
  Issuer Ref:
    Kind:    ClusterIssuer
    Name:    letsencrypt-production
  Request:   LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ3RUQ0NBWjBDQVFBd0hURWJNQmtHQTFVRUF4TVNkR2hsYzNSbFlXMWxaR055WVdJdVkyOXRNSUlCSWpBTgpCZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6dVFzN3dQK0taNXV2VGpmbDlaajBhU2l5WlJaCld5VmRKK1lPQzNxT21vSU1VTzgvV0VZNFV3cGVlSmE4WmRScGNhcHVieXNlVnFNdXhQd0dQaVdqcFBFMHhMQjgKd1hxZXAzbFRSR1hUYzVFRGJCU0lkSHZIZEM1cmNDQkVIellZNzlEVTc4djFyRDhlT2dyWTl4WE5PTEkzYVJDNApUV2c0am1aUHVzeVFEc0RXU0pCQ0ZMaDQ0Snp4UUNyWHNJU2pPWWlVUXMwOHNRVVdsRzJZeEhnbE9YODB1TjUzCjQ3Q3VuT1ZiYmh5SGRxV1hGMzdxRTVma3FYM0RHdHZiVk1hTHJYTDFCOEtyV3Jic1d6RmF6KzJWTFUxRklWdWQKQ1owdTJtYUlLdDE5cEtYNEtsZU1PSXNFeGFvTERNSk5DU2tqY1Y4MFJ4dXJTMWFyQSs5NHJrdkZBd0lEQVFBQgpvRk13VVFZSktvWklodmNOQVFrT01VUXdRakF6QmdOVkhSRUVMREFxZ2hKMGFHVnpkR1ZoYldWa1kzSmhZaTVqCmIyMkNGQ291ZEdobGMzUmxZVzFsWkdOeVlXSXVZMjl0TUFzR0ExVWREd1FFQXdJRm9EQU5CZ2txaGtpRzl3MEIKQVFzRkFBT0NBUUVBV3FqSE94Tm9CR2lNM3YvanpHU3M3U2s1RmhOcExYRk9sRWNwUnRMTjg0ZTBZc3kyRE9EQgpXMDJKN1hEMkRpZXF0RDZycHpyZCtraVRud01KaWZjNmJxNUFxN0szdjdsQ25HaDY1NHppdFptNXlzWjFwdy9uCjRNK2Ywdy9TR0FVWm90a0orOTFOcktLcHE1bDVMU1FXT0tjL0Y4M1kyM3RkcnFzMnlVeWxmWEFRYVVNRzhWYlUKTUtQSkZLMCtSbzVnSFlMOEY1ZkVWSU10MUx0V3RIbG1NNE51TXpIZno3TkVRYUZtZVdCV0U3K0svdGttV1QrVQplcUw0dmd6K1BweVVndjRHZHB4Z0V4TEgwTEVZcVdLc2NCZ3ZkbEU5Y2dvT29xajcrSC9uWjczdUdxM21XUE5KCk40THdjMVdaREZDa0lwekZ2Vm10MEsxN1NLZHBUalZTd0E9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
  UID:       b31e617b-58fe-4fca-bb9b-672b1f847498
  Username:  system:serviceaccount:cert-manager:cert-manager
Status:
  Conditions:
    Last Transition Time:  2022-03-28T15:07:28Z
    Message:               Certificate request has been approved by cert-manager.io
    Reason:                cert-manager.io
    Status:                True
    Type:                  Approved
    Last Transition Time:  2022-03-28T15:07:28Z
    Message:               Waiting on certificate issuance from order network/foo-com-4kn9t-2137103152: "pending"
    Reason:                Pending
    Status:                False
    Type:                  Ready
Events:                    <none>

ClusterIssuer

➜  fleet-infra git:(main) ✗ kubectl -n network get clusterissuers.cert-manager.io     
NAME                     READY   AGE
letsencrypt-production   True    4h53m
letsencrypt-staging      True    4h53m

Spec:
  Acme:
    Email:            [email protected]
    Preferred Chain:  
    Private Key Secret Ref:
      Name:  letsencrypt-production
    Server:  https://acme-v02.api.letsencrypt.org/directory
    Solvers:
      dns01:
        Cloudflare:
          API Key Secret Ref:
            Key:   cloudflare-token
            Name:  cloudflare-token-secret
          Email:   [email protected]
      Selector:
        Dns Zones:
          foo.com
Status:
  Acme:
    Last Registered Email:  [email protected]
    Uri:                    https://acme-v02.api.letsencrypt.org/acme/acct/472337820
  Conditions:
    Last Transition Time:  2022-03-28T15:07:10Z
    Message:               The ACME account was registered with the ACME server
    Observed Generation:   2
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

➜  fleet-infra git:(main) ✗ kubectl -n network describe secrets cloudflare-token-secret 
Name:         cloudflare-token-secret
Namespace:    network
Labels:       kustomize.toolkit.fluxcd.io/name=apps
              kustomize.toolkit.fluxcd.io/namespace=flux-system
Annotations:  <none>

Type:  Opaque

Data
====
cloudflare-token:  40 bytes

Order

➜  fleet-infra git:(main) ✗ kubectl -n network get orders.acme.cert-manager.io
NAME                                  STATE     AGE
foo-com-4kn9t-2137103152   pending   4h55m

 Owner References:
    API Version:           cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  CertificateRequest
    Name:                  foo-com-4kn9t
    UID:                   2a195769-a1a1-430a-a935-16f1ae533448
  Resource Version:        13409
  UID:                     f88e6f79-dcce-4b0f-8293-4afa24060b02
Spec:
  Common Name:  foo.com
  Dns Names:
    foo.com
    *.foo.com
  Issuer Ref:
    Kind:   ClusterIssuer
    Name:   letsencrypt-production
  Request:  LS0tLS1CRUdJTiBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0KTUlJQ3RUQ0NBWjBDQVFBd0hURWJNQmtHQTFVRUF4TVNkR2hsYzNSbFlXMWxaR055WVdJdVkyOXRNSUlCSWpBTgpCZ2txaGtpRzl3MEJBUUVGQUFPQ0FROEFNSUlCQ2dLQ0FRRUF6dVFzN3dQK0taNXV2VGpmbDlaajBhU2l5WlJaCld5VmRKK1lPQzNxT21vSU1VTzgvV0VZNFV3cGVlSmE4WmRScGNhcHVieXNlVnFNdXhQd0dQaVdqcFBFMHhMQjgKd1hxZXAzbFRSR1hUYzVFRGJCU0lkSHZIZEM1cmNDQkVIellZNzlEVTc4djFyRDhlT2dyWTl4WE5PTEkzYVJDNApUV2c0am1aUHVzeVFEc0RXU0pCQ0ZMaDQ0Snp4UUNyWHNJU2pPWWlVUXMwOHNRVVdsRzJZeEhnbE9YODB1TjUzCjQ3Q3VuT1ZiYmh5SGRxV1hGMzdxRTVma3FYM0RHdHZiVk1hTHJYTDFCOEtyV3Jic1d6RmF6KzJWTFUxRklWdWQKQ1owdTJtYUlLdDE5cEtYNEtsZU1PSXNFeGFvTERNSk5DU2tqY1Y4MFJ4dXJTMWFyQSs5NHJrdkZBd0lEQVFBQgpvRk13VVFZSktvWklodmNOQVFrT01VUXdRakF6QmdOVkhSRUVMREFxZ2hKMGFHVnpkR1ZoYldWa1kzSmhZaTVqCmIyMkNGQ291ZEdobGMzUmxZVzFsWkdOeVlXSXVZMjl0TUFzR0ExVWREd1FFQXdJRm9EQU5CZ2txaGtpRzl3MEIKQVFzRkFBT0NBUUVBV3FqSE94Tm9CR2lNM3YvanpHU3M3U2s1RmhOcExYRk9sRWNwUnRMTjg0ZTBZc3kyRE9EQgpXMDJKN1hEMkRpZXF0RDZycHpyZCtraVRud01KaWZjNmJxNUFxN0szdjdsQ25HaDY1NHppdFptNXlzWjFwdy9uCjRNK2Ywdy9TR0FVWm90a0orOTFOcktLcHE1bDVMU1FXT0tjL0Y4M1kyM3RkcnFzMnlVeWxmWEFRYVVNRzhWYlUKTUtQSkZLMCtSbzVnSFlMOEY1ZkVWSU10MUx0V3RIbG1NNE51TXpIZno3TkVRYUZtZVdCV0U3K0svdGttV1QrVQplcUw0dmd6K1BweVVndjRHZHB4Z0V4TEgwTEVZcVdLc2NCZ3ZkbEU5Y2dvT29xajcrSC9uWjczdUdxM21XUE5KCk40THdjMVdaREZDa0lwekZ2Vm10MEsxN1NLZHBUalZTd0E9PQotLS0tLUVORCBDRVJUSUZJQ0FURSBSRVFVRVNULS0tLS0K
Status:
  Authorizations:
    Challenges:
      Token:        cPjsa3x8__zlvW4HMbzg-XuOzPnPENtJq_R7HZQhCo8
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/92349546120/DmulLg
    Identifier:     foo.com
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/92349546120
    Wildcard:       true
    Challenges:
      Token:        89C_aS-mr3AyOKKPIIKKPwZoskM0wfKCDH5FTSW8CiQ
      Type:         http-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/92349546130/ZwKjkA
      Token:        89C_aS-mr3AyOKKPIIKKPwZoskM0wfKCDH5FTSW8CiQ
      Type:         dns-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/92349546130/42LDXw
      Token:        89C_aS-mr3AyOKKPIIKKPwZoskM0wfKCDH5FTSW8CiQ
      Type:         tls-alpn-01
      URL:          https://acme-v02.api.letsencrypt.org/acme/chall-v3/92349546130/7Zisxg
    Identifier:     foo.com
    Initial State:  pending
    URL:            https://acme-v02.api.letsencrypt.org/acme/authz-v3/92349546130
    Wildcard:       false
  Finalize URL:     https://acme-v02.api.letsencrypt.org/acme/finalize/472337820/75311576070
  State:            pending
  URL:              https://acme-v02.api.letsencrypt.org/acme/order/472337820/75311576070
Events:
  Type    Reason   Age   From          Message
  ----    ------   ----  ----          -------
  Normal  Created  60m   cert-manager  Created Challenge resource "foo-com-4kn9t-2137103152-3623191017" for domain "foo.com"
  Normal  Created  60m   cert-manager  Created Challenge resource "foo-com-4kn9t-2137103152-1962712046" for domain "foo.com"
  Normal  Created  54m   cert-manager  Created Challenge resource "foo-com-4kn9t-2137103152-3623191017" for domain "foo.com"
  Normal  Created  54m   cert-manager  Created Challenge resource "foo-com-4kn9t-2137103152-1962712046" for domain "foo.com"

Challenges

➜  fleet-infra git:(main) ✗ kubectl get challenges.acme.cert-manager.io -o wide -A
NAMESPACE   NAME                                             STATE     DOMAIN               REASON                                                                             AGE
network     foo-com-4kn9t-2137103152-1962712046             foo.com                                                                                      56m
network     foo-com-4kn9t-2137103152-3623191017   pending   foo.com   while attempting to find Zones for domain _acme-challenge.foo.com....   56m


Name:         foo-com-4kn9t-2137103152-1962712046
Namespace:    network
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
<snip>
 Owner References:
    API Version:           acme.cert-manager.io/v1
    Block Owner Deletion:  true
    Controller:            true
    Kind:                  Order
    Name:                  foo-com-4kn9t-2137103152
    UID:                   f88e6f79-dcce-4b0f-8293-4afa24060b02
  Resource Version:        196432
  UID:                     bcc69569-8c5e-42c6-bd36-105ca6db79c9
Spec:
  Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/92349546130
  Dns Name:           foo.com
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-production
  Key:     -AbXqHcDS4qGuynrKlOHzUcZ94cpziDLPxzXciqfngA
  Solver:
    dns01:
      Cloudflare:
        API Key Secret Ref:
          Key:   cloudflare-token
          Name:  cloudflare-token-secret
        Email:   [email protected]
    Selector:
      Dns Zones:
        foo.com
  Token:     89C_aS-mr3AyOKKPIIKKPwZoskM0wfKCDH5FTSW8CiQ
  Type:      DNS-01
  URL:       https://acme-v02.api.letsencrypt.org/acme/chall-v3/92349546130/42LDXw
  Wildcard:  false
Events:      <none>

Name:         foo-com-4kn9t-2137103152-3623191017
Namespace:    network
Labels:       <none>
Annotations:  <none>
API Version:  acme.cert-manager.io/v1
Kind:         Challenge
Metadata:
<snip>
Spec:
  Authorization URL:  https://acme-v02.api.letsencrypt.org/acme/authz-v3/92349546120
  Dns Name:           foo.com
  Issuer Ref:
    Kind:  ClusterIssuer
    Name:  letsencrypt-production
  Key:     H8ZkL1xoHoaLsgW8kFsve53dhesobidOhrFlWfCB1Yo
  Solver:
    dns01:
      Cloudflare:
        API Key Secret Ref:
          Key:   cloudflare-token
          Name:  cloudflare-token-secret
        Email:   [email protected]
    Selector:
      Dns Zones:
        foo.com
  Token:     cPjsa3x8__zlvW4HMbzg-XuOzPnPENtJq_R7HZQhCo8
  Type:      DNS-01
  URL:       https://acme-v02.api.letsencrypt.org/acme/chall-v3/92349546120/DmulLg
  Wildcard:  true
Status:
  Presented:   false
  Processing:  true
  Reason:      while attempting to find Zones for domain _acme-challenge.foo.com.
while querying the Cloudflare API for GET "/zones?name=_acme-challenge.foo.com" 
           Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header
  State:  pending
Events:
  Type     Reason        Age                From          Message
  ----     ------        ----               ----          -------
  Normal   Started       57m                cert-manager  Challenge scheduled for processing
  Warning  PresentError  56m (x5 over 57m)  cert-manager  Error presenting challenge: while attempting to find Zones for domain _acme-challenge.foo.com.
while querying the Cloudflare API for GET "/zones?name=_acme-challenge.foo.com" 
            Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header
  Warning  PresentError  12m (x10 over 55m)  cert-manager  Error presenting challenge: while attempting to find Zones for domain _acme-challenge.foo.com.
while querying the Cloudflare API for GET "/zones?name=_acme-challenge.foo.com" 
   Error: 6003: Invalid request headers<- 6103: Invalid format for X-Auth-Key header

Answered by rwlove

Mar 29, 2022

...and just like that I found the problem.

I was using a Cloudflare token, but I was using apiKeySecretRef and not apiTokenSecretRef.

View full answer

rwlove · 2022-03-29T01:25:33Z

rwlove
Mar 29, 2022
Author

...and just like that I found the problem.

I was using a Cloudflare token, but I was using apiKeySecretRef and not apiTokenSecretRef.

0 replies

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Challenge Fails for Letsencrypt/Cloudflare DNS01 #4995

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 1 comment

{{title}}

Select a reply

Challenge Fails for Letsencrypt/Cloudflare DNS01 #4995

rwlove Mar 28, 2022

Certificate

CertificateRequest

ClusterIssuer

Order

Challenges

Replies: 1 comment

rwlove Mar 29, 2022 Author

rwlove
Mar 28, 2022

rwlove
Mar 29, 2022
Author