Adding K8s with juju_kubernetes_cloud resource doesn't inherit config from the default controller

[Gmerold]

Description

Hello Team,

This bug report is a result of this thread on Matrix.

I have Canonical K8s deployed via Juju on a machine managed by MAAS. Now I need to add Canonical K8s to Juju (I already have a Juju controller created on MAAS). For that I'm using the following:

resource "terraform_data" "fetch_kube_config" {
  depends_on = [module.k8s]
  provisioner "local-exec" {
    command = <<-EOT
      juju wait-for application k8s -m $MODEL --timeout 3600s --query='name=="k8s" && (status=="active" || status=="idle")'
      juju exec -m $MODEL -u k8s/0 -- sudo k8s config | tee $KUBE
    EOT
    environment = {
      MODEL = juju_model.k8s-cluster.name
      KUBE  = "${path.module}/${juju_model.k8s-cluster.name}_kubeconfig.yaml"
    }
  }
}

data "local_file" "kubeconfig" {
  depends_on = [terraform_data.fetch_kube_config]
  filename   = "${path.module}/${juju_model.k8s-cluster.name}_kubeconfig.yaml"
}

resource "juju_kubernetes_cloud" "k8s-cluster" {
  name              = juju_model.k8s-cluster.name
  kubernetes_config = data.local_file.kubeconfig.content
}

Everything seems to be fine, new cloud is added to Juju, but when I try to deploy any K8s charm to a model created on Canonical K8s it gets crazy about the StorageClass:

controller-0: 16:57:10 ERROR juju.worker.caasapplicationprovisioner.runner exited "ausf": ensuring application "ausf": StorageClass.storage.k8s.io "sdcore-" is invalid: [metadata.name: Invalid value: "sdcore-": a lowercase RFC 1123 subdomain must consist of lower case alphanumeric characters, '-' or '.', and must start and end with an alphanumeric character (e.g. 'example.com', regex used for validation is '[a-z0-9]([-a-z0-9]*[a-z0-9])?(\.[a-z0-9]([-a-z0-9]*[a-z0-9])?)*'), provisioner: Required value]

When I add K8s to Juju manually with juju add-k8s whatever --controller my_existing_controller everything works fine.

juju_kubernetes_cloud resource doesn't seem to allow specifying the controller, but seems to inherit the default one (?). The only problem is that, that the models created on the K8s cloud added with Juju TF provider do not inherit configuration from this controller.

In my case the problem was with the workload-storage config option. For the model created on K8s added via Juju TF provider the value was default "". For the the model created on K8s added manually with explicitly specified controller the value is controller csi-rawfile-default.

The expected behavior would be that there's not difference in Juju model config.

Cheers,
Bart

Urgency

Annoying bug in our test suite

Terraform Juju Provider version

0.14.0, 0.15.0, 0.16.0

Terraform version

v1.10.5

Juju version

3.6.2

Terraform Configuration(s)

Reproduce / Test

Steps to reproduce described above.

Debug/Panic Output

Notes & References

No response