invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

[firatalkis]

Hello,

When I have started icinga-kubernetes service on rhel8 then getting this error:

icinga-kubernetes[102140]: F0325 14:10:55.837316 102140 main.go:32] invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable

Could you please help me for what did I do wrong?
Thanks.

[razvanmac95]

Hi @firatalkis,

I had this issue too, and it was because I didn't mount the volume with :/.kube/config properly.
Can you check that?

Thanks,
Razvan

[firatalkis]

Hi @razvanmac95,

I have added oc package and ran oc login command for create /.kube/config file. Then file was created you can see below but I am getting the same error:

[lippserd]

Hi @firatalkis,

The KUBERNETES_MASTER environment variable expects the API endpoint of a Kubernetes master.

Please try setting KUBECONFIG instead, i.e.

export KUBECONFIG=/var/spool/icinga2/.kube/config

Best regards,
Eric

[firatalkis]

Hi @lippserd,

yes its worked for this issue: "try setting KUBERNETES_MASTER environment variable"

but afret that I got this error:

Actually I also added those commands to mysql configuration file

I0404 08:56:35.828377 72217 database.go:285] "Connecting to database" logger="database"
I0404 08:56:35.831957 72217 main.go:75] "Importing schema" logger="database"
F0404 08:56:35.841707 72217 main.go:80] Error 1709 (HY000): Index column size too large. The maximum column size is 767 bytes.

[lippserd]

Hi @firatalkis,

You also have to add innodb_default_row_format=dynamic to the mysqld configuration.

Best regards,
Eric

[firatalkis]

Hi @lippserd,

thanks for your help but the problems do not end for me :)

when I run "systemctl start icinga-kubernetes" then getting error you can see below:

icinga-kubernetes[127617]: F0404 11:43:43.675125 127617 main.go:32] invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
systemd[1]: Received SIGCHLD from PID 127617 (icinga-kubernet).
systemd[1]: Child 127617 (icinga-kubernet) died (code=exited, status=255/n/a)
systemd[1]: Child 127617 belongs to icinga-kubernetes.service
systemd[1]: Unwatching 127617.
systemd[1]: icinga-kubernetes.service: main process exited, code=exited, status=255/n/a
systemd[1]: icinga-kubernetes.service changed start -> failed
systemd[1]: Job icinga-kubernetes.service/start finished, result=failed
systemd[1]: Failed to start Icinga for Kubernetes.
-- Subject: Unit icinga-kubernetes.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel

-- Unit icinga-kubernetes.service has failed.

-- The result is failed.

but I run "/usr/sbin/icinga-kubernetes --config /etc/icinga-kubernetes/config.yml" this command then daemon is starting but why icinga-kubernetes want to reaching to github? our servers does not access to internet.

goroutine 332 [running]:
k8s.io/apimachinery/pkg/util/runtime.logPanic({0x18ed6c0, 0xc001902dc8})
k8s.io/[email protected]/pkg/util/runtime/runtime.go:75 +0x85
k8s.io/apimachinery/pkg/util/runtime.HandleCrash({0x0, 0x0, 0xc000102540?})
k8s.io/[email protected]/pkg/util/runtime/runtime.go:49 +0x6b
panic({0x18ed6c0?, 0xc001902dc8?})
runtime/panic.go:770 +0x132
github.com/icinga/icinga-kubernetes/pkg/schema/v1.(*Node).Obtain(0xc00179b440, {0x1d41d38, 0xc00190a008})
github.com/icinga/icinga-kubernetes/pkg/schema/v1/node.go:67 +0x11f9
github.com/icinga/icinga-kubernetes/pkg/sync/v1.(*Sync).sync.func1(0xc00045e6c0)
github.com/icinga/icinga-kubernetes/pkg/sync/v1/sync.go:85 +0x4f
github.com/icinga/icinga-kubernetes/pkg/sync.(*Sink).Upsert(0x2a443d0?, {0x1d2cc10, 0xc000391270}, 0xc0004f6e18?)
github.com/icinga/icinga-kubernetes/pkg/sync/sink.go:59 +0x38
github.com/icinga/icinga-kubernetes/pkg/sync.(*Controller).stream(0xc00043d5c0, {0x1d2cc10, 0xc000391270}, 0xc0001215f0)
github.com/icinga/icinga-kubernetes/pkg/sync/controller.go:93 +0x468
github.com/icinga/icinga-kubernetes/pkg/sync.(*Controller).Stream(0xc00043d5c0, {0x1d2cc10, 0xc000391270}, 0xc0001215f0)
github.com/icinga/icinga-kubernetes/pkg/sync/controller.go:55 +0x369
github.com/icinga/icinga-kubernetes/pkg/sync/v1.(*Sync).sync.func3()
github.com/icinga/icinga-kubernetes/pkg/sync/v1/sync.go:98 +0x46
golang.org/x/sync/errgroup.(*Group).Go.func1()
golang.org/x/[email protected]/errgroup/errgroup.go:78 +0x56
created by golang.org/x/sync/errgroup.(*Group).Go in goroutine 36
golang.org/x/[email protected]/errgroup/errgroup.go:75 +0x96
panic: failed to parse CIDR : invalid CIDR address: [recovered]
panic: failed to parse CIDR : invalid CIDR address:

Apart from this I have one more question.

icigna-kubernetes using the /.kube/config for authentication to kubernetes but the duration of the token in the config is 24 hours. It seems like the token here needs to be constantly updated.

[lippserd]

Hi @firatalkis,

Sorry for replying a bit late. Could you please pull the image and try again? The issue should be fixed.

but I run "/usr/sbin/icinga-kubernetes --config /etc/icinga-kubernetes/config.yml" this command then daemon is starting but why icinga-kubernetes want to reaching to github? our servers does not access to internet.

Icinga for Kubernetes is not trying to reach the internet. This is a stack trace which lists all open functions involved that lead to the error. The URLs here just tell from which Go module the code is coming from.

icigna-kubernetes using the /.kube/config for authentication to kubernetes but the duration of the token in the config is 24 hours. It seems like the token here needs to be constantly updated.

There are a lot of authentication and authorization methods in K8s. I think you could just configure and use something else for Icinga for Kubernetes.

Best regards,
Eric

[firatalkis]

Hi @lippserd,

I am sorry for the late response because our icinga test envoriment included very old components, for example mysql 5.5 version.

so I have refresh install icinga on new virtual server as rhel8 then when I try install icinga-kubernetes got error in the below:

when I run "systemctl enable --now icinga-kubernetes" command getting same error "

then I try "/usr/sbin/icinga-kubernetes --config /etc/icinga-kubernetes/config.yml" then I see error in mariadb log file :

2024-04-25 8:41:34 587 [Warning] Aborted connection 587 to db: 'kubernetes' user: 'kubernetes' host: 'localhost' (Got an error reading communication packets)
2024-04-25 8:41:34 583 [Warning] Aborted connection 583 to db: 'kubernetes' user: 'kubernetes' host: 'localhost' (Got an error reading communication packets)
2024-04-25 8:41:34 589 [Warning] Aborted connection 589 to db: 'kubernetes' user: 'kubernetes' host: 'localhost' (Got an error reading communication packets)
2024-04-25 8:41:34 586 [Warning] Aborted connection 586 to db: 'kubernetes' user: 'kubernetes' host: 'localhost' (Got an error reading communication packets)
2024-04-25 8:41:34 584 [Warning] Aborted connection 584 to db: 'kubernetes' user: 'kubernetes' host: 'localhost' (Got an error reading communication packets)

[tectumopticum]

Similar here:
Required settings in /etc/icinga-kubernetes/config.yml have been made (and validated in the icingaweb2-resources-settings).
But starting the local icinga-kubernetes.service fails:

systemctl start icinga-kubernetes.service
Job for icinga-kubernetes.service failed because the control process exited with error code.
See "systemctl status icinga-kubernetes.service" and "journalctl -xeu icinga-kubernetes.service" for details.

░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit icinga-kubernetes.service has begun execution.
░░
░░ The job identifier is 4574268.
Apr 25 09:45:17 <icingahost> icinga-kubernetes[24056]: F0425 09:45:17.036354   24056 main.go:32] invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Apr 25 09:45:17 <icingahost> systemd[1]: icinga-kubernetes.service: Main process exited, code=exited, status=255/EXCEPTION
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ An ExecStart= process belonging to unit icinga-kubernetes.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 255.
Apr 25 09:45:17 <icingahost> systemd[1]: icinga-kubernetes.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit icinga-kubernetes.service has entered the 'failed' state with result 'exit-code'.
Apr 25 09:45:17 <icingahost> systemd[1]: Failed to start Icinga for Kubernetes.
░░ Subject: A start job for unit icinga-kubernetes.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit icinga-kubernetes.service has finished with a failure.
░░
░░ The job identifier is 4574268 and the job result is failed.
Apr 25 10:08:41 <icingahost> systemd[1]: Starting Icinga for Kubernetes...
░░ Subject: A start job for unit icinga-kubernetes.service has begun execution
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit icinga-kubernetes.service has begun execution.
░░
░░ The job identifier is 4577646.
Apr 25 10:08:41 <icingahost> icinga-kubernetes[5773]: F0425 10:08:41.708220    5773 main.go:32] invalid configuration: no configuration has been provided, try setting KUBERNETES_MASTER environment variable
Apr 25 10:08:41 <icingahost> systemd[1]: icinga-kubernetes.service: Main process exited, code=exited, status=255/EXCEPTION
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ An ExecStart= process belonging to unit icinga-kubernetes.service has exited.
░░
░░ The process' exit code is 'exited' and its exit status is 255.
Apr 25 10:08:41 <icingahost> systemd[1]: icinga-kubernetes.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ The unit icinga-kubernetes.service has entered the 'failed' state with result 'exit-code'.
Apr 25 10:08:41 <icingahost> systemd[1]: Failed to start Icinga for Kubernetes.
░░ Subject: A start job for unit icinga-kubernetes.service has failed
░░ Defined-By: systemd
░░ Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
░░
░░ A start job for unit icinga-kubernetes.service has finished with a failure.
░░
░░ The job identifier is 4577646 and the job result is failed.


systemctl status icinga-kubernetes.service
× icinga-kubernetes.service - Icinga for Kubernetes
     Loaded: loaded (/usr/lib/systemd/system/icinga-kubernetes.service; enabled; vendor preset: disabled)
     Active: failed (Result: exit-code) since Thu 2024-04-25 10:08:41 CEST; 26min ago
    Process: 5773 ExecStart=/usr/sbin/icinga-kubernetes --config /etc/icinga-kubernetes/config.yml (code=exited, status=255/EXCEPTION)
   Main PID: 5773 (code=exited, status=255/EXCEPTION)

The required file is available/readable:
-rw-r----- 1 icinga-kubernetes root 611 Apr 25 09:47 /etc/icinga-kubernetes/config.yml

Any additional file(s) required? Unfortunately the documentation is missing further steps how to configure a connection to kubernetes located outside the icinga-master and what to do inside kubernetes.

OS: SuSE Linux Enterprise Server SLES 15 SP5
Service-Version: 0.1.0
Mariadb-version (located on separate server): 10.6.15

[lippserd]

@tectumopticum You need a kubeconfig file that defines how and which Kubernetes cluster to access.

@firatalkis I'm sorry but we only support MySQL from version 8 and MariaDB from version 10.4. However, the documentation states something else that needs to be fixed. Maybe it's also worth a try to increase max_allowed_packet.

[lippserd]

@Jan-Schuppik Please fix the documentation regarding the supported database versions and create a section explaining how to use the kubeconfig file and what it's for. You can link to this section where we write

With locally accessible kubeconfig ...

[firatalkis]

Hi @lippserd,

our mariadb versiyon:

Do I have upgrade to 10.4 version for fixing below issue?

I have changed max_allowed_packet value from 512 MB to 2G. After that I fresh install icinga-kubernetes then when I run service with systemctl got permission error as "status=255" and I changed permission this folder "/etc/icinga-kubernetes" to 755 than this issue is fixed.

May 28 17:30:21 host systemd[1]: icinga-kubernetes.service: Main process exited, code=exited, status=255/n/a
May 28 17:30:21 host systemd[1]: icinga-kubernetes.service: Failed with result 'exit-code'.
May 28 17:30:21 host systemd[1]: Failed to start Icinga for Kubernetes.

I used "xx" service account for create "/var/spool/icinga2/.kube/config" file with "oc login" command but icinga-kubernetes trying to collect data from kubernetes with "system:anonymous" user and that user getting permission error:

May 29 11:55:52 host icinga-kubernetes[813204]: E0529 11:55:52.185682 813204 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Ingress: failed to list *v1.Ingress: ingresses.networking.k8s.io is forbidden: User "system:anonymous" cannot list resource "ingresses" in API group "networking.k8s.io" at the cluster scope
May 29 11:55:57 host icinga-kubernetes[813204]: W0529 11:55:57.188885 813204 reflector.go:539] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Job: jobs.batch is forbidden: User "system:anonymous" cannot list resource "jobs" in API group "batch" at the cluster scope
May 29 11:55:57 host icinga-kubernetes[813204]: E0529 11:55:57.188927 813204 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Job: failed to list *v1.Job: jobs.batch is forbidden: User "system:anonymous" cannot list resource "jobs" in API group "batch" at the cluster scope
May 29 11:55:58 host icinga-kubernetes[813204]: W0529 11:55:58.399065 813204 reflector.go:539] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Node: nodes is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "" at the cluster scope
May 29 11:55:58 host icinga-kubernetes[813204]: E0529 11:55:58.399106 813204 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Node: failed to list *v1.Node: nodes is forbidden: User "system:anonymous" cannot list resource "nodes" in API group "" at the cluster scope
May 29 11:55:58 host icinga-kubernetes[813204]: W0529 11:55:58.845428 813204 reflector.go:539] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:anonymous" cannot list resource "deployments" in API group "apps" at the cluster scope
May 29 11:55:58 host icinga-kubernetes[813204]: E0529 11:55:58.845475 813204 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Deployment: failed to list *v1.Deployment: deployments.apps is forbidden: User "system:anonymous" cannot list resource "deployments" in API group "apps" at the cluster scope
May 29 11:55:58 host icinga-kubernetes[813204]: W0529 11:55:58.881140 813204 reflector.go:539] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:anonymous" cannot list resource "statefulsets" in API group "apps" at the cluster scope
May 29 11:55:58 host icinga-kubernetes[813204]: E0529 11:55:58.881180 813204 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.StatefulSet: failed to list *v1.StatefulSet: statefulsets.apps is forbidden: User "system:anonymous" cannot list resource "statefulsets" in API group "apps" at the cluster scope
May 29 11:55:59 host icinga-kubernetes[813204]: W0529 11:55:59.647303 813204 reflector.go:539] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.Event: events.events.k8s.io is forbidden: User "system:anonymous" cannot list resource "events" in API group "events.k8s.io" at the cluster scope
May 29 11:55:59 host icinga-kubernetes[813204]: E0529 11:55:59.647349 813204 reflector.go:147] k8s.io/[email protected]/tools/cache/reflector.go:229: Failed to watch *v1.Event: failed to list *v1.Event: events.events.k8s.io is forbidden: User "system:anonymous" cannot list resource "events" in API group "events.k8s.io" at the cluster scope
May 29 11:56:00 host icinga-kubernetes[813204]: W0529 11:56:00.667714 813204 reflector.go:539] k8s.io/[email protected]/tools/cache/reflector.go:229: failed to list *v1.PersistentVolumeClaim: persistentvolumeclaims is forbidden: User "system:anonymous" cannot list resource "persistentvolumeclaims" in API group "" at the cluster scope