Rancher helm-controller vanished!

[xanalogica]

I have a frustrating problem, where rke2 for v1.23 was working for a week, and then it stopped processing HelmChart manifests, and I believe it is because the helm-controller is nowhere to be found. Odd thing is, it installed rke2-canal, rke2-coredns and rke2-metrics server fine, but then it won't install cert-manager or Hashicorp vault.

The deploy controller successfully notices the manifests for those two in /var/lib/rancher/rke2/server/manifests and loads them into a namespace, but then no helm-controller comes along to turn them into a helm-install-XXXX job. And the apiVersion header is correct and matches what is used for rke2 internal manifests.:

apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: cert-manager
  namespace: kube-system
spec:
  repo: https://charts.jetstack.io
  chart: cert-manager
  version: v1.9.1
  set:
    installCRDs: true
  targetNamespace: cert-manager

I don't see any error logs from rke2 about shutting down helm-controller. Does anyone have pointer to where to look next? I've spent two days on this. Any tips much appreciated!

[brandond]

The helm controller does not run in a pod; it is embedded in the RKE2 server itself. I would check the rke2-server journald logs for any errors when you apply the updated manifest.

[xanalogica]

Thanks Brandond for this information that the helm controller is embedded in the RKE2 server. I thought it had crashed/vanished.

Checking the logs, I saw that it was balking at parsing a boolean into a int32, for cert-manager installCRDs: true. I replaced true with 1 and it worked. It was passing thru the HelmChart CRD using the set: tag but perhaps that CRD is forcing numbers.

The odd part is that this parsing error jammed the helm controller -- by having one HelmChart YAML with an error, the controller would not process any other HelmChart YAMLs but gave no error. Apparently it maintains a queue of work and when one fails, the queue just stops. I didn't expect that. Thanks again.

[brandond]

That should not be the case. Are you sure that the error wasn't related to parsing the manifest? If you had both HelmChart resources in the same file, and that file contained a syntax error, then it is expected that none of the resources in that file would be processed.