Kubernetes kubeconfig setting being set in stack from kubeconfig json as secret

[sheldonhull]

Using Go SDK.

I tried to store a kubeconfig (in json format) as a secret value in a stack.
It didn't seem to be able to reach the server.
When I provide the path to the kubeconfig it works.

Should I be able to store the kubeconfig directly in the stack and provider use it without more configuration or not?

Why? I have a development stack that requires service account token + cert to authenticate and deploy. I want to allow running the plan for any team member without needing them to deal with a 5-6 command setup to generate a local kubeconfig as this stack only goes to this one target location.

Since each stack's secrets are encrypted per stack I thought it might be a good use case for this, rather than expecting a local kubeconfig to be generated each time.

[rawkode]

You can definitely store a kubeconfig in a Pulumi secret. I do this with:

pulumi config set --secret kubernetes:kubeconfig < /path/to/kubeconfig

If this doesn't work, do let me know and I'll put together a small example repository.

[sheldonhull]

Will do. Thank you for this. I will give it another shot and see if I can simplify the rubegoldberg machine a little 😁.

[sheldonhull]

@rawkode I am doing basic provider setup via sdk, so:

pargs := &kubernetes.ProviderArgs{}

I was trying to trim this abomination out:

if configData.Onlyyaml {
			_ = ctx.Log.Info("Onlyyaml set to true, so this won't deploy to k8 directly", nil)
			renderProviderName = "k8s-yaml-renderer"
			pargs.RenderYamlToDirectory = pulumi.String("yaml")
		} else {
			_ = ctx.Log.Info(fmt.Sprintf("homedir: %q", homeDir), nil)
			kubeConfigOverride, isSet := os.LookupEnv("KUBECONFIG")
			if isSet {
				_ = ctx.Log.Info(fmt.Sprintf("KUBECONFIG override: %q", kubeConfigOverride), nil)
				if _, err := os.Stat(kubeConfigOverride); err != nil {
					b, err := os.ReadFile(kubeConfigOverride)
					if err != nil {
						_ = ctx.Log.Error(fmt.Sprintf("Unable to read Kubeconfig override: %q, %v", kubeConfigOverride, err), nil)
					}
					pargs.Kubeconfig = pulumi.String(b)
				}
			} else {
				_ = ctx.Log.Info(fmt.Sprintf("KUBECONFIG using config in pulumi: %q", kubeConfigOverride), nil)
				pargs.Kubeconfig = pulumi.StringPtr("kubernetes:kubeconfig")
			}
		}

I want to instead bring it down to:

pargs.Namespace = pulumi.String(configData.Namespace)
renderProviderName := "k8"
if configData.Onlyyaml {
	renderProviderName = "k8s-yaml-renderer"
	pargs.RenderYamlToDirectory = pulumi.String("yaml")
}
// if not overriding to yaml renderer, then I'm not going to even set kubeconfig in the provider, and let it pull from config automatically (i assume)
renderProvider, err := kubernetes.NewProvider(ctx, renderProviderName, pargs)

Am I on the right track?

[sheldonhull]

@rawkode yeah I just tried this and without any overrides and using json kubeconfig compressed to a single line = configured Kubernetes cluster is unreachable: unable to load Kubernetes client configuration from kubeconfig file: invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable.

EDIT: No difference with json kubeconfig or yaml.

[sheldonhull]

This actually seems very relevant to challenges I had with this. Might be an issue with the provider in state misaligned with what I expect #7741

[sheldonhull]

Ok I got it to work, but only when I stopped trying to use kubernetes:kubeconfig. Instead I have it working as a normal config value.

		if configData.Onlyyaml {
			_ = ctx.Log.Info("Onlyyaml set to true, so this won't deploy to k8 directly", nil)
			renderProviderName = "k8s-yaml-renderer"
			pargs.RenderYamlToDirectory = pulumi.String("yaml")
		} else {
			pargs.Kubeconfig = cfg.GetSecret("serviceaccount")
		}
		type kuby struct {
			Kubeconfig string `json:"kubeconfig"`
		}
		kb := kuby{}
		cfg.RequireSecretObject("serviceaccount", &kb)
		pargs.Kubeconfig = pulumi.String(kb.Kubeconfig)
		renderProvider, err := kubernetes.NewProvider(ctx, renderProviderName, pargs)

[giovannicandido]

For me it did'nt work, I'm using pulumi deployments and a have to work around this by setting a environment variable with a base64 encoded kubeconfig then running a script to decode and generate the ~/.kube/config file before the pulumi run its deployment.