Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix monitoring of K3s control plane components: kube-scheduler, kube-proxy, and kube-controller-manager #401

Merged
merged 1 commit into from
Apr 30, 2024

Conversation

matofeder
Copy link
Contributor

@matofeder matofeder commented Apr 30, 2024

PR#392 added Kubernetes monitoring deployment play. Kubernetes monitoring is intended to monitor the k3s instance control plane components out of the box.
PR#392 observes several issues that are mentioned in the PR's description, which are related to the k3s distribution.

This PR addresses the 1. issue where the monitoring solution could not monitor K3s control plane components kube-scheduler, kube-proxy, and kube-controller-manager. The second part of the fix has already been merged osism/defaults#177

Tesbed test

  1. Update kubernetes-monitoring play
$ docker exec -it -u 0 osism-ansible bash
$ vim /ansible/infrastructure/kubernetes-monitoring.yml
  1. Execute play
$ osism apply kubernetes-monitoring
2024-04-30 13:48:53 | INFO     | Task was prepared for execution.
2024-04-30 13:48:53 | INFO     | It takes a moment until the task has been started and output is visible here.

PLAY [Apply kubernetes monitoring role] ****************************************

TASK [Deploy kubernetes-monitoring helm chart] *********************************
Tuesday 30 April 2024  13:49:02 +0000 (0:00:03.935)       0:00:03.935 ********* 
changed: [localhost]

PLAY RECAP *********************************************************************
2024-04-30 13:50:26 | INFO     | Play has been completed. There may now be a delay until all logs have been written.
2024-04-30 13:50:26 | INFO     | Please wait and do not abort execution.
localhost                  : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Tuesday 30 April 2024  13:50:26 +0000 (0:01:23.529)       0:01:27.464 ********* 
=============================================================================== 
Deploy kubernetes-monitoring helm chart -------------------------------- 83.53s
  1. Validate deployment
$ kubectl get po -n kubernetes-monitoring 
NAME                                                        READY   STATUS    RESTARTS   AGE
alertmanager-kube-prometheus-alertmanager-0                 2/2     Running   0          2m46s
dnation-kubernetes-jsonnet-translator-797fbf8ddb-msnqs      1/1     Running   0          2m49s
kube-prometheus-operator-85cbd9ffcd-dxmh5                   1/1     Running   0          2m49s
kubernetes-monitoring-grafana-566d9d4f65-gf2gw              3/3     Running   0          2m49s
kubernetes-monitoring-kube-state-metrics-79b6bfbc46-6mjcm   1/1     Running   0          2m49s
kubernetes-monitoring-prometheus-node-exporter-bdp9v        1/1     Running   0          2m49s
kubernetes-monitoring-prometheus-node-exporter-g4l4s        1/1     Running   0          2m49s
kubernetes-monitoring-prometheus-node-exporter-kgnb4        1/1     Running   0          2m49s
kubernetes-monitoring-prometheus-node-exporter-x5jc7        1/1     Running   0          2m49s
kubernetes-monitoring-thanos-query-6c979c7cc7-hhb6x         1/1     Running   0          2m49s
prometheus-kube-prometheus-prometheus-0                     3/3     Running   0          2m46s
  1. Port forward Grafana UI, and access it (username: admin, password: pass)
$ export POD_NAME=$(kubectl get pods --namespace kubernetes-monitoring -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=kubernetes-monitoring" -o jsonpath="{.items[0].metadata.name}")
$ kubectl --namespace kubernetes-monitoring port-forward --address 0.0.0.0 $POD_NAME 3000

Screenshot from 2024-04-30 16-11-02

See that all K3s control plane components kube-scheduler, kube-proxy, and kube-controller-manager are in OK state

Related to: SovereignCloudStack/issues#576

@matofeder matofeder force-pushed the fix_k8s_monitorig_known_issues branch from 608fc22 to 457f590 Compare April 30, 2024 14:19
@matofeder matofeder requested a review from berendt April 30, 2024 14:19
@matofeder matofeder force-pushed the fix_k8s_monitorig_known_issues branch from 457f590 to 3363394 Compare April 30, 2024 14:20
[PR#392](#392) added
Kubernetes monitoring deployment play. Kubernetes monitoring is
intended to monitor the k3s instance control plane components out of
the box.
PR#392 observes several issues that are mentioned in the
PR's descriptioni, which are related to the k3s distribution.

This commit  addresses the 1. issue where the monitoring solution
could not monitor K3s control plane components kube-scheduler,
kube-proxy, and kube-controller-manager.

The second part of the fix: osism/defaults#177

Signed-off-by: Matej Feder <[email protected]>
@matofeder matofeder force-pushed the fix_k8s_monitorig_known_issues branch from 3363394 to f9c830f Compare April 30, 2024 14:30
@berendt berendt merged commit fe39126 into main Apr 30, 2024
2 checks passed
@berendt berendt deleted the fix_k8s_monitorig_known_issues branch April 30, 2024 15:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants