Expired Token on refreshed AWS keys

[lee5i3]

I am using Hashicorp Vault to rotate my AWS keys automatically, however after they are rotated, I was able to verify the keys were updated but external-dns is still showing they were expired, requiring me to restart my pod in order to get external-dns to access AWS again

I'm using helm to install external-dns from here https://kubernetes-sigs.github.io/external-dns v1.13.0

My values look like this

  env:
    - name: AWS_SHARED_CREDENTIALS_FILE
      value: "/.aws/credentials"
 
  domainFilters: 
    - <my domain>.com
  
  podAnnotations:
    vault.hashicorp.com/agent-inject: 'true'
    vault.hashicorp.com/role: 'vault'
    vault.hashicorp.com/agent-inject-secret-credentials: '/aws/creds/external-dns'
    vault.hashicorp.com/secret-volume-path: '/.aws'
    vault.hashicorp.com/agent-inject-template-credentials: |
      {{ with secret "/aws/creds/external-dns" }}
      [default]
      aws_access_key_id = {{ .Data.access_key }}
      aws_secret_access_key = {{ .Data.secret_key }}
      aws_session_token = {{ .Data.security_token }}
      {{ end }}

vault is refreshing them in the .aws/credentials, im not sure why external-dns is unable to see the new keys
my workaround was to add this which tells vault to kill the process every time vault refreshes the keys

  shareProcessNamespace: true
  podAnnotations:
    vault.hashicorp.com/agent-inject-command-credentials: |
          kill -TERM $(pidof external-dns) || true

[Nexarian]

This is a known bug in the V1 AWS Golang SDK:

• SharedCredsProvider: Expire using aws_session_expiration aws/aws-sdk-go#3167
• aws/credentials: SharedCredentialsProvider does not refresh on expired credentials. aws/aws-sdk-go#3163

We may need to think about updating external-dns to https://github.com/aws/aws-sdk-go-v2. Is there a plan for that?

[Nexarian]

More links:

• [Feature request] Refresh credentials from file aws/aws-sdk-go#1993
• feat[chart]: Add shareProcessNamespace field to deployment #2715

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle rotten
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle rotten

[k8s-triage-robot]

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

[k8s-ci-robot]

@k8s-triage-robot: Closing this issue, marking it as "Not Planned".

In response to this:

The Kubernetes project currently lacks enough active contributors to adequately respond to all issues and PRs.

This bot triages issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Reopen this issue with /reopen
• Mark this issue as fresh with /remove-lifecycle rotten
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/close not-planned

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rajeshkio]

@lee5i3 I am facing the same issue. How did you fix it? Still restarting the container? kill -TERM $(pidof external-dns) || true