Env vs config file error when I tried to use both of them

[ksemele]

What steps did you take and what happened?

I followed this manual from capv for simple test vSphere cluster.

So I skipped Set required environment variables: part because of I have an existing config file here: ~/.cluster-api/clusterctl.yaml

I added 2 extra envs to my env and checked that they are exist:

env | grep EXP_
EXP_CLUSTER_RESOURCE_SET=true
EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION=true

When I tried to apply a cluster.yaml file I got an validation error:

kubectl apply -f cluster.yaml
cluster.cluster.x-k8s.io/my-cluster created
vspherecluster.infrastructure.cluster.x-k8s.io/my-cluster created
vspheremachinetemplate.infrastructure.cluster.x-k8s.io/my-cluster created
machinedeployment.cluster.x-k8s.io/my-cluster-md-0 created
clusterresourceset.addons.cluster.x-k8s.io/my-cluster-crs-0 created
secret/my-cluster created
secret/vsphere-config-secret created
configmap/csi-manifests created
secret/cloud-provider-vsphere-credentials created
configmap/cpi-manifests created
Error from server (Invalid): error when creating "cluster.yaml": admission webhook "validation.kubeadmcontrolplane.controlplane.cluster.x-k8s.io" denied the request: KubeadmControlPlane.cluster.x-k8s.io "my-cluster" is invalid: [spec.kubeadmConfigSpec.format: Forbidden: can be set only if the KubeadmBootstrapFormatIgnition feature gate is enabled, spec.kubeadmConfigSpec.ignition: Forbidden: can be set only if the KubeadmBootstrapFormatIgnition feature gate is enabled]
Error from server (Invalid): error when creating "cluster.yaml": admission webhook "validation.kubeadmconfigtemplate.bootstrap.cluster.x-k8s.io" denied the request: KubeadmConfigTemplate.bootstrap.cluster.x-k8s.io "my-cluster-md-0" is invalid: [spec.template.spec.format: Forbidden: can be set only if the KubeadmBootstrapFormatIgnition feature gate is enabled, spec.template.spec.ignition: Forbidden: can be set only if the KubeadmBootstrapFormatIgnition feature gate is enabled]

What did you expect to happen?

Cluster is created.

Cluster API version

1.8.4

Kubernetes version

1.31.1

Anything else you would like to add?

I think there are some errors with envs vs configfile...
Because when I added EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION: true to my existing ~/.cluster-api/clusterctl.yaml
This error is gone.

I also created the task first in the capv repo, but they said that this is probably a problem in clusterctl: kubernetes-sigs/cluster-api-provider-vsphere#3245

Label(s) to be applied

/kind bug
One or more /area label. See https://github.com/kubernetes-sigs/cluster-api/labels?q=area for the list of labels.

[k8s-ci-robot]

This issue is currently awaiting triage.

If CAPI contributors determine this is a relevant issue, they will accept it by applying the triage/accepted label and provide further guidance.

The triage/accepted label can be added by org members by writing /triage accepted in a comment.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

[neolit123]

clusterctl uses the viper library by default

cluster-api/cmd/clusterctl/client/config/reader_viper.go

Line 78 in f810589

func (v *viperReader) Init(ctx context.Context, path string) error {

i tried adding a new line in this unit test:

cluster-api/cmd/clusterctl/client/config/reader_viper_test.go

Line 44 in f810589

t.Setenv("EXP_CLUSTER_RESOURCE_SET", "true")

and reading it later:

t.Log(v.Get("EXP_CLUSTER_RESOURCE_SET"))

it works as expected, i.e. both config from file and env vars are read.

the error comes from

cluster-api/bootstrap/kubeadm/api/v1beta1/kubeadmconfig_types.go

Line 287 in f810589

if !feature.Gates.Enabled(feature.KubeadmBootstrapFormatIgnition) {

so seems the feature gate is not read correctly.

but checking from where the FG comes from in this case, it is applied as a flag value by using kustomize:

cluster-api/bootstrap/kubeadm/config/manager/manager.yaml

Line 25 in f810589

- "--feature-gates=MachinePool=${EXP_MACHINE_POOL:=true},KubeadmBootstrapFormatIgnition=${EXP_KUBEADM_BOOTSTRAP_FORMAT_IGNITION:=false}"

and that only works from env vars i.e. viper and config is not in the picture, i would think.

are you sure adding these EXP_ vars in the clusterctl config works?
because i cannot confirm any GO code reading them i.e. clusterctl doens't.

[sbueringer]

@ksemele Can you please share at which time you ran which clusterctl commands with which content of the clusterctl.yaml file?