Yaml parse error when volume are mounted via configmap or secret on kubernetes

[jerome-karabenli]

Describe the bug

When I add env GATUS_CONFIG_PATH=/config as described in doc, this env point on directory, gatus tries to recursively read all yaml or yml files but in kubernetes when we mount a volume inside a container and this volume is provided by configMap or Secret Kubernetes mount a kind of symlink on the mountPath, the real data is on the same folder but in hidden folder with date as name. This is probably to have latest date of change of configMap or Secret, because if we change/update the values from configMap or Secret theses data are updated after a few seconds inside a container in pod.

What do you see?

2024/03/04 14:58:22 [config][LoadConfiguration] Reading configuration from /config/..2024_03_04_14_58_14.4259295514/config.yaml
2024/03/04 14:58:22 [config][LoadConfiguration] Reading configuration from /config/config.yaml
panic: error reading configuration from directory /config: error due to parameter with value of primitive type: only maps and slices/arrays can be merged, which means you cannot have define the same key twice for parameters that are not maps or slices/arrays

goroutine 1 [running]:
main.main()
/app/main.go:19 +0x1b1

What do you expect to see?

2024/03/04 15:00:04 [config][LoadConfiguration] Reading configuration from configFile=/config/config.yaml
2024/03/04 15:00:04 [config][validateAlertingConfig] Alerting is not configured
2024/03/04 15:00:04 [config][validateEndpointsConfig] Validated 7 endpoints
2024/03/04 15:00:04 [controller][Handle] Listening on 0.0.0.0:8080

List the steps that must be taken to reproduce this issue

add env in kubernetes deployment GATUS_CONFIG_PATH=/config

Version

twinproduction/gatus:v5.7.0

Additional information

if I could I would have proposed a PR but I don't know how to code in Go.
The problem is probably because gatus tries to read all yaml files recursively in specified folder by env.

Some improvement may be to not just read all yaml files inside specified folder but keep the recursive read because there is a security improvement to be achieved by splitting security configs, alerting configs and endpoints configs.

For exemple in Kubernetes we can imagine to mount a secret as volume in /config/security/security.yaml and an alerting config which contains credentials from another secret on /config/alerting/alerting.yaml and developers can add monitoring endpoints from configMap on /config/endpoints/endpoints.yaml

In this scenario it would be great to read recursively folders in /config path but do not read file in folder starting by dot or a regex which excludes folders looking like "..2024_03_04_14_58_14"

[x45dev]

Thanks for reporting this. I am experiencing the same problem. It means that Gatus configuration is Kubernetes is essentially broken. :(

[x45dev]

PS: Issue persists in v5.10.0
A very ugly temporary workaround is to manually build the container image with the configfile, but that's hardly a solution!

[TwiN]

Could either of you show me what your YAML looks like? I've been running Gatus on Kubernetes and I've never had this kind of issue.

I'm going to be honest here, if I saw a path that looked like /config/..2024_03_04_14_58_14.4259295514/config.yaml, I would have assumed that there's a bug somewhere as that folder name doesn't look very natural 😅

This is probably to have latest date of change of configMap or Secret, because if we change/update the values from configMap or Secret theses data are updated after a few seconds inside a container in pod.

What do you mean by "probably"? I had assumed the mechanism that generated a folder with the current timestamp would be something implemented on your end, because I've never seen Kubernetes do something like this.

There's an example here for how to deploy Gatus on Kubernetes: https://github.com/TwiN/gatus/blob/master/.examples/kubernetes/gatus.yaml

[x45dev]

Could either of you show me what your YAML looks like? I've been running Gatus on Kubernetes and I've never had this kind of issue.

I'm going to be honest here, if I saw a path that looked like /config/..2024_03_04_14_58_14.4259295514/config.yaml, I would have assumed that there's a bug somewhere as that folder name doesn't look very natural 😅

This is probably to have latest date of change of configMap or Secret, because if we change/update the values from configMap or Secret theses data are updated after a few seconds inside a container in pod.

What do you mean by "probably"? I had assumed the mechanism that generated a folder with the current timestamp would be something implemented on your end, because I've never seen Kubernetes do something like this.

There's an example here for how to deploy Gatus on Kubernetes: https://github.com/TwiN/gatus/blob/master/.examples/kubernetes/gatus.yaml

Hello @TwiN, thank you for your quick response.

I've replicated the error with the example configuration that you've provided in your link.

I did some testing and have determined that this error actually seems to relate to circumstances where the GATUS_CONFIG_PATH env is used. If it is not set, and the configmap is mounted to the default location (/config, as defined by the commandline configFile= default values), then everything works as expected.

If you set GATUS_CONFIG_PATH, then it will break, and you will see weird paths like the one mentioned before (/config/..2024_03_04_14_58_14.4259295514/config.yaml). Note - yes, it looks bizarre, but mine look like that too.

So, the current fix is to just mount configfiles at /config. (Incidentally, my previous configmaps used the global.yaml and endpoints.yaml config files, but those didn't work either. I had to merge it all into a single config.yaml file.)

[TwiN]

Alright, I've been able to replicate it & I see the following error message:

2024/05/10 03:23:56 [config][LoadConfiguration] Reading configuration from /config/..2024_05_10_03_23_55.1182937433/config.yaml
2024/05/10 03:23:56 [config][LoadConfiguration] Reading configuration from /config/config.yaml
panic: error reading configuration from directory /config: error due to parameter with value of primitive type: only maps and slices/arrays can be merged, which means you cannot have define the same key twice for parameters that are not maps or slices/arrays

I don't have the bandwidth to test this right now, but I think that a solution could be to skip symbolic links?

Something like modifying

gatus/config/config.go

Lines 172 to 173 in 9d151fc

	if fileInfo.IsDir() {
	err := walkConfigDir(configPath, func(path string, d fs.DirEntry, err error) error {

to also check if the file is a symbolic link. I'm not sure how this will impact Gatus' ability to hot reload though.

Relevant post: https://stackoverflow.com/questions/50685385/kubernetes-config-map-symlinks-data-is-there-a-way-to-avoid-them

It may be possible to replicate this outside of Kubernetes by just creating symbolic links and replicating what Kubernetes does by copying this folder structure:

drwxrwxrwx 3 root root 4096 May 10 03:22 .
drwxr-xr-x 3 root root 4096 May 10 03:22 ..
drwxr-xr-x 2 root root 4096 May 10 03:22 ..2024_05_10_03_23_55.1182937433
lrwxrwxrwx 1 root root   16 May 10 03:22 config.yaml-> ..2024_05_10_03_23_55.1182937433/config.yaml

Reference: https://stackoverflow.com/a/50687707/4603538

---

Just to clarify, the error message

error reading configuration from directory /config: error due to parameter with value of primitive type: only maps and slices/arrays can be merged, which means you cannot have define the same key twice for parameters that are not maps or slices/arrays

likely just means that the same configuration file is read twice, once through the real folder (..2024_05_10_03_23_55.1182937433 folder) that contains the file, and once through the symbolic link (config.yaml) to the file.

An extremely lazy way to "fixing" the issue could be to checksum each file, and if the file has already been loaded, then skip it. This would ensure that the same config file isn't read twice, which should prevent the error from happening.