helm: Re-factor the way we handle CRDs in fluent-operator's helm chart(s)

[joshuabaird]

Is your feature request related to a problem? Please describe.

How we currently handle CRDs:

• We currently distribute fluent-operator's CRDs in two separate "sub-charts" which are dependencies of the "main" chart.
• This makes it difficult to upgrade Helm releases when there are CRD changes, because CRDs (in crds/ directories) are not upgraded by default in Helm. Users have to manually fetch CRDs, run kubectl replace etc
• The usage of file:// references in dependencies also causes duplicate CRDs in the release as reported here.

Describe the solution you'd like

I'd like to propose that we implement something similar to how Karpenter handles their Helm releases:

• Include the fluentd and fluent-bit CRDs in the fluent-operator's "main" chart's crds/ folder.
  This will ensure all CRDs get installed during an initial helm install
• Create a new "top-level" chart named fluent-operator-crds
• Put CRDs in the templates/ directory of the fluent-operator-crds chart
• Placing them in the templates/ directory will allow a user to helm upgrade the fluent-operator-crds chart to upgrade our CRDs instead instead of having to fetch them manually and kubectl replace etc
• In this setup, we'll need to publish updated CRDs to two locations -- charts/fluent-operator/crds and charts/fluent-operator-crds/templates -- but this should be fairly simple with CI. (edited)

Additional context

I don't think this would introduce any breaking changes:

• Currently, if a user wants to install fluent-operator without any CRDs, they can set fluentd.crdsEnable or fluentbit.crdsEnable to false
• Moving forward, we would deprecate these two values, and a user could simply use helm install --skip-crds` if they want to prevent CRD installation

As part of this, we can also clarify and update our documentation for "installing" and "upgrading."

[marcofranssen]

I don't think we can use the --skip-crds.

1. It would not allow to selectively install which crds you want in the cluster (FluentBit vs Fluentd).
2. Furthermore many helm charts e.g. Spire which I also maintain use separate crd charts to give users more control. (E.g. in ArgoCD I install CRDs in a separate sync wave in a separate ArgoCD application.)

My proposal would be to make the CRDs top level charts with their own release cadence (Similar like Prometheus, Spire). Then the other chart can depend on the CRD charts. This will allow following:

• Install only the fluentbit or fluentd crds or both
• allow advanced usecases using argocd (replace, syncwaves etc)
• Maintain crds in one place
• Make it possible to release them in their own cadence (only when operator changes result in CRD changes)
• Allow to release the main chart more often without crd changes (e.g. to improve the values.yaml UX and all other kind of deployment optimizations).

Example

Make crds installation dependent on in we enable fluentbit/fluentd. still allow to not install them via one global flag installCrds.enabled (so users like me can install the Chart myself in separate ArgoCD app)

Chart.yaml

apiVersion: v2
name: fluent-operator
description: Fluent Operator provides great flexibility in building a logging layer based on Fluent Bit and Fluentd.
keywords:
  - logging
  - fluent-bit
  - fluentd
  - operator
version: 3.10.22
# renovate: datasource=docker depName=ghcr.io/fluent/fluent-operator/fluent-operator
appVersion: "3.9.0"
icon: https://raw.githubusercontent.com/fluent/fluent-operator/master/docs/images/fluent-operator-icon.svg
home: https://www.fluentd.org/
sources:
  - https://github.com/fluent/fluent-operator
dependencies:
  - name: fluent-bit-crds
    repository: https://fluent.github.io/helm-charts/fluent-bit-crds
    version: 3.9.0
    condition: fluentbit.enabled && installCrds.enabled
  - name: fluentd-crds
    repository: https://fluent.github.io/helm-chartsfluentd-crds
    version: 3.5.0
    condition: fluentd.enabled && installCrds.enabled

[joshuabaird]

You could selectively install CRDs in this scenario:

• Install the main fluent-operator chart but use --skip-crds (this can be done in Argo as well)
• Install the fluent-operator-crds chart with the following values
  *fluentd.installCrds: true
  • fluentbit.installCrdsL: false

This proposal does break the CRDs out into their own top-level chart (eg, fluent-operator-crds). I don't really see much value in separating the fluentbit and fluentd CRD charts; we can selectively control which CRDs get installed using values in the CRD chart and I believe it adds a bit more operational overhead in the release process.

I took a look at the Spire chart(s), but I don't see that the spire-crds chart is a dependency of the spire chart. It looks like users need to a) install the spire CRD chart and then b) install the spire main chart.

In my proposal, we could just leave the CRDs out of the fluent-operator (main) chart all together (like Spire) if we wanted. The Karpenter project puts them there to simplify the initial installation for users -- they can just install a single chart to get an operational controller with CRDs. In addition, it provides them in a separate top-level chart so that they can be easily upgraded and/or managed separately like you do.

[marcofranssen]

Still I would make 2 separate CRDs charts as those evolve also each in their own pace.

I would also skip duplicating them in a crds folder.

[joshuabaird]

Still I would make 2 separate CRDs charts as those evolve also each in their own pace.

I think we should consider operational overhead here. I'll think about this and post my thoughts here in a bit!

I would also skip duplicating them in a crds folder.

This is fine if we're ok with forcing our users to install two helm charts to get an operational controller. I'll take a look around at some other popular Helm charts.