external auth (exec) not working (azure/kubelogin and aws-iam-authenticator)

[hofq]

I tried connecting to our prod clusters using "azure/kubelogin" and aws-iam-authenticator.

None of these Auth Methods worked.

Is there an eta we can expect exec auth to work? We want to switch from the closed source software lens to headlamp, but this is currently blocking.

[illume]

Hi.

Thanks for writing.

Yeah, we use the k8s APIs rather than kubectl and haven't implemented support for the "client-go credential plugins".

As a not-great work around authentication for both AKS and EKS can be done with OIDC. I realize this is probably not suitable however. Because the point of the aws-iam-authenticator is to avoid having to manage separate services. (please correct me if I'm wrong?)

Supporting external auth commands isn't currently on the short term roadmap. But I personally think it's something we should support.

---

External auth related docs

• client-go credential plugins
• aws-iam-authenticator
• azure/kubelogin

exec:
    apiVersion: ...
    command: ...
    args: ...

Other related issues:

A number of others have reported here and elsewhere wanting support for external auth commands.

Here are some of the github issues:

• Flatpak: cannot execute aws eks get-token #1582
• bad gateway while accessing AKS cluster #1341

[gaeljw]

I agree this will be a must have for most companies.

[strowi]

Would also like for this to work. I'm guessing there are a lot of people using some form of client-side addon (e.g. i am using passman to store the credentials in keychain):

- name: rancher
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - keychain
      - user@cluster
      command: kubectl-passman
      env: null
      interactiveMode: IfAvailable
      provideClusterInfo: false

[lindblombr]

From #1716 I've been able to get this working like so

Can confirm what @kdeyko says. I have a similar issue with a custom OIDC binary we invoke from exec. If we open the app through point-and-click (macOS), we just get bad gateway and lots of errors in the dev console connecting to localhost. If I open up in a terminal via /Applications/Headlamp.app/Contents/MacOS/Headlamp, everything works perfectly.

Could be a work-around until the underlying connectivity issue is resolved?

[lindblombr]

On macos, I've used the following work-around to make this work via native Finder/Dock/Spotlight launch

$ sudo launchctl config user path /bin:/usr/bin:/sbin:/usr/sbin:/usr/local/bin:/usr/local/sbin:/opt/homebrew/bin

This ensures that the paths where my exec plugins are located are in the environment of every application I launch via Finder et al. Unfortunately, the change requires root and a reboot, but successfully unblocks Headlamp in local GUI mode to work properly with exec configuration.

Incorporating something similar to shell-path in the electron startup should allow Headlamp to inherit user-configured PATH environment so that Headlamp works as expected: https://github.com/sindresorhus/shell-path (hopefully there is a newer, maintained dependency). Headlamp appears to implement kubeconfig handling in a way that already supports exec so long as any executable referenced by exec is either absolute path or IN the search PATH. TL;DR, enabling this feature seems to only require fixing the way Headlamp launches with respect to incorporating/honoring user-configured PATH that tools like kubectl already depend on.

[illume]

Thanks @lindblombr

[pg185210]

Hi.

Thanks for writing.

Yeah, we use the k8s APIs rather than kubectl and haven't implemented support for the "client-go credential plugins".

As a not-great work around authentication for both AKS and EKS can be done with OIDC. I realize this is probably not suitable however. Because the point of the aws-iam-authenticator is to avoid having to manage separate services. (please correct me if I'm wrong?)

Supporting external auth commands isn't currently on the short term roadmap. But I personally think it's something we should support.

External auth related docs

• client-go credential plugins
• aws-iam-authenticator
• azure/kubelogin

exec:
    apiVersion: ...
    command: ...
    args: ...

Other related issues:

A number of others have reported here and elsewhere wanting support for external auth commands.

Here are some of the github issues:

• Flatpak: cannot execute aws eks get-token #1582
• bad gateway while accessing AKS cluster #1341

Yep, I'd love that. OpenLens picks up my AWS EKS context and connects to the clusters out of the box. Headlamp recognizes the contexts, but just shows "Bad Gateway" status for the clusters. I guess I'll stay with OpenLens for now :(

edit2:
Oh wait, yes, looks like it's exactly the problem:

16:18:10.725 › server process stderr: 2024/05/08 16:18:10 http: proxy error: getting credentials: exec: executable aws not found

It looks like you are trying to use a client-go credential plugin that is not installed.

To learn more about this feature, consult the documentation available at:
      https://kubernetes.io/docs/reference/access-authn-authz/authentication/#client-go-credential-plugins

thats what my kube config looks like:

- name: arn:aws:eks:us-west-2:011111111111:cluster/mycluster1
  user:
    exec:
      apiVersion: client.authentication.k8s.io/v1beta1
      args:
      - --region
      - us-west-2
      - eks
      - get-token
      - --cluster-name
      - mycluster1
      - --output
      - json
      command: aws