[Go] kubens not resolving file paths relative to config file

[nbusseneau]

Hi,

Just switched from bash version to Go version on a Windows environment (Git Bash).

It looks like kubens does not treat relative file paths in .kube/config as relative to the config file, but rather as relative to current working directory:

λ kubens
error: could not list namespaces (is the cluster accessible?): failed to initialize config: invalid configuration: [
unable to read client-cert REDACTED_PATH_CRT.crt for REDACTED_USER due to open REDACTED_PATH_CRT.crt: The system cannot find the file specified.,
unable to read client-key REDACTED_PATH_KEY.key for REDACTED_USER due to open REDACTED_PATH_KEY.key: The system cannot find the file specified.,
unable to read certificate-authority REDACTED_PATH_CRT_2.crt for REDACTED_CLUSTER due to open REDACTED_PATH_CRT_2.crt: The system cannot find the file specified.]

λ cd ~/.kube/
λ kubens
{works as expected}

λ cd ..
error: could not list namespaces (is the cluster accessible?): failed to initialize config: invalid configuration: [
unable to read client-cert REDACTED_PATH_CRT.crt for REDACTED_USER due to open REDACTED_PATH_CRT.crt: The system cannot find the file specified.,
unable to read client-key REDACTED_PATH_KEY.key for REDACTED_USER due to open REDACTED_PATH_KEY.key: The system cannot find the file specified.,
unable to read certificate-authority REDACTED_PATH_CRT_2.crt for REDACTED_CLUSTER due to open REDACTED_PATH_CRT_2.crt: The system cannot find the file specified.]

Current workaround is to use absolute file paths in config file by prefixing all paths with C:\Users\USERNAME\.kube\, but that kinda defeats the purpose of having a .kube directory :D

I did not have a chance to play with Go yet, but if nobody picks this up and I have some time this weekend, I'll have a try at it.

Thanks for the awesome work by the way! 👍

[nbusseneau]

Preliminary investigation indicates the errors occurs when calling clientcmd here:

kubectx/cmd/kubens/list.go

Line 61 in 401188f

cfg, err := clientcmd.RESTConfigFromKubeConfig(b)

So perhaps this actually is a bug for https://github.com/kubernetes/client-go/?

[ahmetb]

Does the same kubeconfig file work with kubectl?

My guess also would be that this is a client-go issue. That said I have never seen file paths in kubeconfig file. So I am not sure what you’re trying to do is valid.

[nbusseneau]

Yes, works fine with kubectl no matter the current working dir. Here are examples from the documentation where you can see how a config can include client certificates and/or CA certificates: https://kubernetes.io/docs/tasks/access-application-cluster/configure-access-multiple-clusters/

apiVersion: v1
clusters:
- cluster:
    certificate-authority: fake-ca-file
    server: https://1.2.3.4
  name: development
contexts:
- context:
    cluster: development
    namespace: frontend
    user: developer
  name: dev-frontend
current-context: dev-frontend
kind: Config
preferences: {}
users:
- name: developer
  user:
    client-certificate: fake-cert-file
    client-key: fake-key-file

[ahmetb]

Yeah I am suspecting client-go must have a knob for this. But I have not seen it.

[eshepelyuk]

Opened an duplicate since experiencing the same issue, but question is why kubectl works in this case ? Does it use other GO libraries ?

[ahmetb]

This was reported by couple more folks using mingw/msys2 systems (similar to git bash) in #233. I'm now trying to figure out if this reproduces outside these shells (natively on windows).

@Skymirrh @eshepelyuk Is it possible you move these kubeconfig file + cert files to %USERPROFILE%.kube\ and run kubens.exe directly from cmd.exe or powershell?

why kubectl works in this case

I'm guessing because kubectl determines which kubeconfig files to parse itself, so it can save where it has read the file from. in our case, we do custom yaml parsing to prevent the node structure in the document, so we pass client-go library a "parsed YAML", and we need to investigate if client-go has a way of specifying "relative directory" of where the file was read from separately.

I personally don't have any files referenced from my kubeconfig file so I don’t run into this, but I can see many others do have this.

[eshepelyuk]

@ahmetb
for now - there's a simple workaround - just to fix manually ~/.kube/config file and put absolute path there.

Well, what you suggested to use %USERPROFILE%\.kube for cmd.exe - is the exact location that is used by msys2 bash \ git for windows bash - so no need to move files.

But I did run cmd.exe and executed kubens - and I received exactly the same error

D:\projects>kubens
error: could not list namespaces (is the cluster accessible?): failed to initialize config: invalid configuration: unable to read certificate-authority testcluster.nas.local.crt for testcluster.nas.local due to open testcluster.nas.local.crt: The system cannot find the file specified.

[ahmetb]

just to fix manually ~/.kube/config file and put absolute path there.

May I ask everyone in this issue: Where do these manifests with relative file paths come from? Is it from a particular cloud-provider, or is it hand-crafted? (The result won't change anything, I’m just curious.)

[eshepelyuk]

In my case - those configs are created by the scripts thaat are provided by our infrastructure engineers.

I've looked into the script they gave and this config is created by pure kubectl call

kubectl config set-cluster "$CLUSTER_NAME" \
  --server https://api.${CLUSTER_NAME} \
  --certificate-authority "$HOME/.kube/$CLUSTER_NAME.crt"

So, it seems this format is created by kubectl itself - you may notice that .crt file is provided with absolute path, but eventually config do not contain full path.

[nbusseneau]

Is it possible you move these kubeconfig file + cert files to %USERPROFILE%.kube\ and run kubens.exe directly from cmd.exe or powershell?

I'm not sure I understand the question, but if the question is "what happens if you run kubens from cmd or Powershell rather than from Git Bash?", then there is no difference.

Git Bash's ~ refers to Windows' %USERPROFILE%, so my .kube and its config/certificates are already stored in the right place. I can use kubectl from Git Bash or Cmd/PowerShell interchangeably, the only advantage is that on Git Bash I can source completion scripts to alleviate resource discovery 😁

in our case, we do custom yaml parsing to prevent the node structure in the document, so we pass client-go library a "parsed YAML"

Ow, I missed that during preliminary investigation. My bad, would have obviously been the primary suspect had I seen it!

Where do these manifests with relative file paths come from? Is it from a particular cloud-provider, or is it hand-crafted? (The result won't change anything, I’m just curious.)

"Hand-crafted" by replicating examples from the Kubernetes documentation itself, as already linked above. The setup process for the config file includes calls to kubectl config using relative paths, thus why the quotes around "hand-crafted" (because it actually is kubectl manipulating the file format).

You may note this interesting bit from the documentation:

File and path references in a kubeconfig file are relative to the location of the kubeconfig file. File references on the command line are relative to the current working directory. In $HOME/.kube/config, relative paths are stored relatively, and absolute paths are stored absolutely.
-- Source

[ahmetb]

Yeah the last bit’s certainly valuable. Now we need to go investigate how we can make kubectl use a particular base path when we use the RESTConfigFromKubeConfig method with an in-memory byte array of the kubeconfig file contents.

[nbusseneau]

I have taken a quick look at clientcmd and it looks like BuildConfigFromFlags is a good candidate for loading config files.

There actually is an example provided by Go client:

var kubeconfig *string
if home := homeDir(); home != "" {
	kubeconfig = flag.String("kubeconfig", filepath.Join(home, ".kube", "config"), "(optional) absolute path to the kubeconfig file")
} else {
	kubeconfig = flag.String("kubeconfig", "", "absolute path to the kubeconfig file")
}
flag.Parse()

// use the current context in kubeconfig
config, err := clientcmd.BuildConfigFromFlags("", *kubeconfig)
if err != nil {
	panic(err.Error())
}

Source: https://github.com/kubernetes/client-go/blob/f099a72e140ab46a70ceb9337c75802c3a2be045/examples/out-of-cluster-client-configuration/main.go

[ahmetb]

BuildConfigFromFlags is a good candidate for loading config files.

It is ––except it double-reads the file. We already read it in-memory once for detailed YAML doc tree parsing, if we use that it’ll parse the file yet again. (Since we open the file for RW, it might cause issues e.g. on windows.)

Furthermore we need to be able to work with multiple kubeconfig files, and it's likely we'll implement some of that logic ourselves. I’ll take a closer look later on. Thanks for flagging.

[nbusseneau]

It is ––except it double-reads the file. We already read it in-memory once for detailed YAML doc tree parsing, if we use that it’ll parse the file yet again. (Since we open the file for RW, it might cause issues e.g. on windows.)

Note: I am not an expert at all with clientcmd, however from my quick looks at the documentation and code, I don't think clientcmd is intended to be used for loading the config file from memory. I think it would probably be more fitting to the intended use case if, instead of reading the config file to memory to get a YAML tree, you'd load the config file using clientcmd and then use that as the basis for the rest of your program (RW access is supported by clientcmd configs).

Furthermore we need to be able to work with multiple kubeconfig files, and it's likely we'll implement some of that logic ourselves. I’ll take a closer look later on. Thanks for flagging.

FYI, it seems clientcmd can handle that as well, using EnvVar from PathOptions which are deferred to ConfigAccess via GetLoadingPrecedence() and then used in loaders for the Load() method.

[reitzig]

May I ask everyone in this issue: Where do these manifests with relative file paths come from? Is it from a particular cloud-provider, or is it hand-crafted? (The result won't change anything, I’m just curious.)

FWIW, I use direnv to cleanly separate k8s configs between projects:

.envrc

export KUBECONFIG="/home/raphael/.kube/config:kube_config.yaml"

Here, using a relative path for the project config seems prudent: it's automatically correctly scoped and will not break when moving the project folder around. Continuing from there, storing certificates in the project folder, for instance, seems a natural choice, so kube_config.yaml will have relative paths to those as well.