Service account token request from synced service account does not work with AWS IRSA

[giepa]

Is your feature request related to a problem?

In our setup, to facilitate AWS IRSA, we configured service accounts to be synced to the host cluster:

sync:
  serviceaccounts:
    enabled: true

This works well when service accounts are mounted on pods. However when a service account token is requested in vcluster, for example:

kubectl create token mysa

Then the token is rejected as invalid by the AWS sdk:

WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://kubernetes.default.svc.cluster.local status code: 400,

I suspect it is because the token is issued by the vcluster and not the host cluster. Is there a way around this, how can we request a valid token?

Which solution do you suggest?

Vcluster should forward service account token requests to the host cluster when service account sync is enabled.

Which alternative solutions exist?

No response

Additional context

No response

[FabianKramm]

@giepa thanks for creating this issue! That is expected, since vcluster has its own service accounts and certificates. You will need to add the annotation to the actual workload service account and then AWS should inject a token automatically.

[giepa]

@FabianKramm thanks for your response, could you clarify:

You will need to add the annotation to the actual workload service account

Which annotation are you referring to?

In our case, the workload service account is not being used, instead a token is requested for a given service account, in a given namespace.

To give more context, we are running external-secrets-operator in vcluster. The operator has the concept of secret store, where a service account reference is provided, for example:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: my-store
  namespace: my-ns
spec:
  provider:
    aws:
      auth:
        jwt:
          serviceAccountRef:
            name: my-sa

For the above example, external secrets operator, will request a token for service account my-sa in my-ns and use it to authenticate with the AWS-sdk. It is this token that is rejected as invalid:

WebIdentityErr: failed to retrieve credentials caused by: InvalidIdentityToken: No OpenIDConnect provider found in your account for https://kubernetes.default.svc.cluster.local status code: 400,

Is there a way around this? Would vcluster be able to request tokens from the host cluster instead, when service account sync is enabled?

[joaocc]

@giepa did you manage to get an answer to this? we have exactly the same need. It would be wonderful if we could do it via IRSA. If that is not possible, an alternative (which you prob already considered) would be to configure the provider to use explicit IAM account and credentials, but this kind of negates the advantages of IRSA/AWS and is less secure.

[giepa]

@joaocc as an interim, we did configure the provider to use explicit IAM account and credentials, as you said its not secure, we are getting away with it only because its for dev environments, for production such an approach would be a no go. If you do come up with something better, please let us know.

[joaocc]

Thanks. As a quick clarification, I didn't meant it was not secure, but simply less secure as IRSA, as we have secrets being stored there (and, FWIW, In our context, we still consider a valid approach even for production).
Having said this, of course we would prefer to be able to use IRSA, as it is much more convenient (gently nodding at @FabianKramm for some additional insight)

[joaocc]

Update from recent testing:

When we create the following manifest inside the vcluster, we get no errors and we get store validated

apiVersion: external-secrets.io/v1beta1
kind: ClusterSecretStore
metadata:
  name: aws-ssm-ps--0
spec:
  provider:
    aws:
      auth:
        jwt:
          serviceAccountRef:
            name: ext-secrets-sa-x-ext-secrets-x-vcluster
      region: eu-west-1
      service: ParameterStore

However, if we then add an ExternalSecret with a reference to that cluster secret store, it won't synchronize, and I get the following error
the desired SecretStore aws-ssm-ps--0 is not ready even though the store appears as "validated"

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: my-ext-secr-1
  namespace: flux-system
spec:
  secretStoreRef:
    kind: ClusterSecretStore
    name: aws-ssm-ps--0
  data:
    - remoteRef:
        key: /path/to/ssm-ps-key/k1
      secretKey: my_k1
    - remoteRef:
        key: /path/to/ssm-ps-key/k2
      secretKey: my_k2
  dataFrom: []
  refreshInterval: 4h
  target:
    ...

Additionally:

• using vcluster-eks 0.16.4 on EKS 1.27
• "sync.serviceaccounts" is also set to true
• the AWS role is created correctly and annotated with "ext-secrets-sa-x-ext-secrets-x-vcluster" at the correct host namespace (not vcluster namespace) and has permissions on the SSM-PS
• the service account is annotated with the ARN of the role

@FabianKramm, would you have any suggestion on this? Thanks

[joaocc]

After some more digging, I found this info on external-secrets docs (https://external-secrets.io/latest/provider/aws-secrets-manager/#eks-service-account-credentials), where serviceAccountRef must also include namespace even on ClusterSecretStores so that secrets can be created in any namespace.

However, once we change the relevant definition to include a namespace, the ClusterSecretStore no longer gets validated. This happens if we include the namespace as seen by the host (which would make sense since the serviceAccount is also as seen by the host), and also if we use the vcluster namespace (had to try it)

          serviceAccountRef:
            name: ext-secrets-sa-x-ext-secrets-x-vcluster
            namespace = vcluster-ns-on-host

          serviceAccountRef:
            name: ext-secrets-sa-x-ext-secrets-x-vcluster
            namespace = ext-secrets

In either case, we get the following error message:
unable to create session: ServiceAccount "ext-secrets-sa-x-ext-secrets-x-vcluster" not found

[iasikou]

We are facing the exact same issue with external-secrets and vcluster. What you are seeing though is expected since vcluster cluster does not have any visibility on the ext-secrets-sa-x-ext-secrets-x-vcluster service accounts since those sa are created in the host cluster if serviceaccounts sync is enabled.

I believe that for serviceAccountRef the vcluster service account should be used, but that seems not to work either.

[twoTimesAgnew]

Facing the same issue here with vCluster and vault-secrets-operator using GCP. vso attempts to request a token from sts.googleapis.com, but we keep getting an error stating that Issuer https://kubernetes.default.svc.cluster.local does not match https://{google_cluster_url}.

This makes total sense because when we decode the JWT created by kubectl create token {service-account-within-vcluster}, we see that the local issuer is https://kubernetes.default.svc.cluster.local and the expected issuer is the url GCP has for the cluster.

I've tried messing with the --service-account-issuer values for k3s, but to no avail. I don't think this could even work theoretically since the vCluster has no notion of the host cluster. Kind of at a dead end with this one.