How to properly override helmChart values using overlays? #4658
Comments
k8s-ci-robot
added
needs-kind
|
@paul-bormans: This issue is currently awaiting triage. SIG CLI takes a lead on issue triage for this repo, but any Kubernetes member can accept issues by applying the The Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
I override Helm charts at the later patches stage using standard Kustomize patching. You can see examples of this throughout my main public Kubernetes repo, starting at the Kustomization.yaml: |
|
To do this, you would currently need to use the "transformed transformers" technique: https://kubectl.docs.kubernetes.io/references/kustomize/kustomization/#transformed-transformers This is not particularly elegant, and we have a new Kustomize kind called "Composition" that is intended to improve workflows like these. You can read more about it in this KEP: kubernetes/enhancements#2299 This PR details the work left to do to get Composition merged: #4323 /kind support |
k8s-ci-robot
added
kind/support
|
@KnVerey: Closing this issue. In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
|
Why close ? Proposed solution is not enough, can we have a sample please ? |
|
I closed it because there is an entire KEP tracking the problem and solution for the "transformed transformers" technique being too complicated. I don't think there is value in having separate issues open about it for individual transformers. The docs I linked to do already provide examples of the current and future techniques. |
|
If I understand the proposed composition solution correctly, you are redefining the generated manifests after the Helm build. While it would be much simpler and also safer to be able to patch the values.yaml before the Helm build. So when the Helm chart comes to evolve, nothing is to be done on the kustomize side except modify the version and possibly update the values.yaml if necessary. In the case of composition, the patched manifest may have been modified, even deleted, etc... Maintenance therefore becomes a real problem with this solution. |
|
I would like this too. I have a base which includes a helm chart with sane values. But I would like the environment overlays to be able to adjust (like enable debugging) rather than making small changes in kustomize patches (which is actually tedious since in my current case, I have to rewrite the whole command/args in the pod specs just to enable debug logging - not to mention making some values ineffective since the args are overwritten and related values have no effect anymore). I would much prefer to override some chart values, and then have kustomize render the chart manifests. |
|
@NickLarsenNZ you can actually add command args without having to rewrite the whole command args array. Simple example: More complicated example with positional-based replacement of an existing arg: I do a lot of Kustomize installations and patching, so you may be interested in seeing the master https://github.com/HariSekhon/Kubernetes-configs I agree it would be nice to support some additional Helm overrides in |
This is already possible without Kustomize (if I understand what you're trying to do). Helm supports multiple values files, and although it seems to be black magic that almost no one uses or even knows about, when you specify multiple values files it automatically does shallow merges of those values files. This means you don't need Kustomize at all for this use case. I've been consulting and teaching this method at various companies and conferences over the last few years, this is a path to simplicity and success. Here's an example. First part of the trick is to use sub-charts... # requirements.yaml
dependencies:
- name: volume-autoscaler
version: 1.0.5
repository: "https://devops-nirvana.s3.amazonaws.com/helm-charts/"Then specify your "global" settings applied to all your envs... # values.yaml
volume-autoscaler:
slack_webhook_url: "https://hooks.slack.com/services/1234/123/1234"
slack_channel: "INTENTIONALLY-INVALID-WILL-OVERRIDE-IN-ENV-VALUES-FILES"
resources:
limits:
cpu: 20m
memory: 75Mi
requests:
cpu: 20m
memory: 75MiThen you do overrides in an values-env file... # values-stage.yaml
volume-autoscaler:
slack_channel: "devops-stage"Then you use helm with two values files. And ideally, you bake this logic into your CI/CD framework, and a README in your Helm Chart's folder so people know to use a specific command that includes both values files. The command examples are... # First download dependency (subchart)
helm dependencies update volume-autoscaler
# Then helm diff with multiple values to check what will change (YOU should ALWAYS diff before applying)
helm diff upgrade --namespace infrastructure --allow-unreleased volume-autoscaler ./volume-autoscaler -f volume-autoscaler/values.yaml -f volume-autoscaler/values-stage.yaml
# And finally helm upgrade
helm upgrade --install --namespace infrastructure volume-autoscaler ./volume-autoscaler -f volume-autoscaler/values.yaml -f volume-autoscaler/values-stage.yamlThe way "shallow merge" works, is that if you have an map (dict) it'll merge the values from both the values.yaml and any subsequent values. However, if you have a list, it will override that list, there is no way to merge a list. Also, this is not limited to two values files, it's as many as you want. I have customers that setup multiple "sub-environments" on one kubernetes cluster, eg: one per-namespace. In this type of setup, we'd have three values files. A # this one everything is values-stage will override anything in values, which is the desired outcome
-f volume-autoscaler/values.yaml -f volume-autoscaler/values-stage.yaml
# is not the same as this, which everything in values will override everything in staging config, which
# is generally the exact opposite of what you want
-f volume-autoscaler/values-stage.yaml -f volume-autoscaler/values.yaml |
|
And @paul-bormans as well might be relevant (see above message). As long as your upstream chart has the features in it, you don't need Kustomize. Just use helm only, keep things simpler. I only (personally, opinionated) recommend using Kustomize when your upstream chart doesn't support customizing something. |
|
Thanks @AndrewFarley
That's exactly why I'm using Kustomize with the chart. Unfortunately, Helm is closed for extension, while Kustomize is open for extension. So I'm having to add a few additional resources along with it. |
|
@paul-bormans, if I understand what you meant correctly, I just add a second values file in the base that is generically for any environment overrides, but then I have to place that file (say with a CI/CD step). If I understood correctly, I have thought of doing that, but I really want the environment overlays to represent the modifications as they are - so we have a clearer picture of environmental disparity (one of Kustomizes strengths). Having to do some background file placement feels a bit too hacky (though it works) and interrupts my Kustomize development workflow (where I render various environments locally and pipe the output through some utilities that help me see the bits I'm interested in). Perhaps I misunderstood, so apologies if that's the case. |
|
Personally I changed my approach, where before I asked kustomize to build my chart helm. Now I ask Helm to build with the Each tool retains its scope of operation and I do not distort the kustomize system by the helm chart build. To automate this need I only created a helm binary wrapper (used by my argocd) :
Next, simply use @AndrewFarley solution to override helm chart values :
|
|
@KnVerey Can you elaborate how "transformed transformers" would solve this issue? As far as I understand everything you suggested works after Helm chart is build because transformers, transformed transformers and composition works after generators. This is very error prone and subject to unexpected behavior. Can't we have something like |
|
can we just overwrite the helmCharts by name if it is defined in overlays? |
How do you mean? AFAIK, the overwriting (patching) occurs on the resulting manifests, which neither the helm values, nor the helmChart configuration are any part of. |
|
@NickLarsenNZ sorry for the confuse. I mean if I define a helmCharts with the save name as the charts from base in the overlays, can we merge the properties of charts? for example I have such config in base helmCharts:
- name: minecraft
includeCRDs: false
valuesInline:
minecraftServer:
eula: true
difficulty: hard
rcon:
enabled: true
then in overlay/dev, I have namePrefix: dev-
resources:
- ../../base
helmCharts:
- name: minecraft
valuesInline:
minecraftServer:
eula: false
difficulty: easythen we can merge the helmCharts part before helm expand the template |
@xujihui1985, I wish it could overlay helm values (or properties as you mention), but it can only operate on the resulting manifests that helm spits out. So Kustomize only passes the values to helm to use. You don't see the values in overlays or other generators/transforms. That means you have to apply patches as you would if it was just plain manifests from a base kustomization, which is sometimes awkward when it would be more ideal to patch the helm values yaml. I should mention, the helm template is rendered where it is defined, so overlays without a helmCharts config never see anything about Helm. I hope one day it works in a way that you and I would find useful. |
|
@NickLarsenNZ Understood. Thanks for clarify. |
|
Why is it closed? There still isn't an option to do it |
|
This issue should remain open as it's highly relevant. |
|
Can we reopen it? The above mentioned composition API is closed, we need some good practice with Helm! |
|
We still need this. Especially now that we have multiple sources for ArgoCD. That would be awesome to be able to merge valuesObjects |
|
Composition API is not abandoned - see #5198 |
|
But I wonder how the Composition API would help with Helm charts? For example, if I want to override just few fields, like the version (to have different version on staging and production environments) and only few values (like some base/redirection URIs)? |
|
I found a way to do values overlay by using Here's an example that I used for grafana-agent agent:
resources:
requests:
cpu: "150m"
memory: "256Mi"
limits:
cpu: "200m"
memory: "1Gi"
helmCharts:
- name: grafana-agent
version: 0.27.2
repo: https://grafana.github.io/helm-charts
releaseName: grafana-agent
namespace: grafana
includeCRDs: false
valuesFile: ../../base/values.yaml
valuesMerge: override # how to treat valuesInline with respect to Values. Legal values: ‘merge’, ‘override’, ‘replace’. Defaults to ‘override’.
valuesInline:
agent:
resources:
requests:
# cpu: "150m" # Will inherit from base
memory: "2Gi" # Will override
limits:
# cpu: "200m" # Will inherit from base
memory: "3Gi" # Will overridenow you can build but make sure to disable load restrictor
|
|
The other venue I'm thinking is just use configmap generator which can merge yaml properties I think. Then use replacements feature and somehow make helm call that, maybe use the helm generator as a file and try to transform the transformers as mentioned here. |
|
kustomize v5 added the field |
this is awesome |
|
Hello guys, any news on this one? Still not possible to do the override like suggested by [xujihui1985]? Thanks. Br, |
As far as I understand it this is not how it works now. You can override values by specifying the base values first with |
|
Just be sure to disable the load restrictor |
|
I am managing multiple clusters (e.g. cluster-base/
└── devops-tools/
└── kube-prometheus/
├── values.yml
├── namespace.yml
└── kustomization.yml
cluster-overlay/
└── development/
└── components/
└── kube-prometheus/
├── values-overwrite.yml
└── kustomization.yml
└── production/
└── components/
└── kube-prometheus/
├── values-overwrite.yml
└── kustomization.ymlThe ---
apiVersion: kustomize.config.k8s.io/v1alpha1
kind: Component
namespace: monitoring
resources:
- namespace.ymlThe ---
components:
- ../../../../cluster-base/devops-tools/kube-prometheus/
helmCharts:
- name: kube-prometheus-stack
repo: https://prometheus-community.github.io/helm-charts
namespace: monitoring
releaseName: monitoring
version: 57.2.1
includeCRDs: true
valuesFile: ../../../../cluster-base/devops-tools/kube-prometheus/values.yml
additionalValuesFiles:
- values-overwrite.ymlAnd in ---
prometheus:
ingress:
hosts:
- prometheus.caas.my-url.com
tls:
- secretName: prometheus-tls
hosts:
- prometheus.caas.my-url.com
grafana:
env:
GF_SERVER_ROOT_URL: "https://monitoring.caas.my-url.com/"
ingress:
hosts:
- monitoring.caas.my-url.com
tls:
- secretName: monitoring-tls
hosts:
- monitoring.caas.my-url.com
|
|
@uluzox this definitely works? Does it require That said, using |
|
Hi @seanturner026, |
|
@uluzox Is it possible to merge instead of overwrite the Expected Result: Current Result: |
|
Perhaps try swapping what you pass in for |
Hi, I don't think this is possible since merging array members is not possible the way you would like to have it. |
you can't merge command array, but it's total valid to merge other fields under container |
|
@KnVerey This has been closed way to early, and the community is clearly still in need for a solution. |
|
There is a way to do it more generally, as shown here https://github.com/wmiller112/kustomize-helm-values/tree/main The helm generator config is put in an external resource, rather than inline in the Then the It's a bit brain-bending and I won't say I like it, but it does work, and can be used to patch the helm chart inflator configuration, inline values, etc. |
and others
Consider following "base":
Then on a "develop" overlay i need to pass in additional overrides compared to the baseline (provided by the values.yml o/t chart + some defaults in the kustomized/base layer.
I know this isn't working :-)....:
I'm puzzled here how to bring in this seem to be trivial need:
The only way seems to be a full copy of the base helmChart entry; but that defeats the purpose of having overlays (what it comes to the charts).
Any advise?
Paul
The text was updated successfully, but these errors were encountered: