Fix prometheus alerts

[emosbaugh]

What type of PR is this?

type::feature

What this PR does / why we need it:

Fixes alerts

"KubeScheduler has disappeared from Prometheus target discovery."
"KubeControllerManager has disappeared from Prometheus target discovery."
"KubeProxy has disappeared from Prometheus target discovery."

Which issue(s) this PR fixes:

Fixes NONE

Special notes for your reviewer:

Does this PR introduce a user-facing change?

TODO

Does this PR require documentation?

NONE

[emosbaugh]

I need to test kube-bench to see if this violates cis benchmark

[github-actions]

Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-2843-34abec7-prometheus-0.55.0-34.10.0-k8s-docker

[github-actions]

Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-2843-ca770c1-prometheus-0.55.0-34.10.0-k8s-docker

[github-actions]

Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-2843-42f10c3-prometheus-0.55.0-34.10.0-k8s-docker

[github-actions]

Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-2843-a9482e8-prometheus-0.55.0-34.10.0-k8s-docker

[github-actions]

Testgrid Run(s) Executing @ https://testgrid.kurl.sh/run/pr-2843-859b9b4-prometheus-0.55.0-34.10.0-k8s-docker

[xavpaice]

rebased against current main branch

[xavpaice]

IIUC, this change is effective for k8s 1.21? Wonder if we can split out the content of addons/prometheus/0.55.0-34.10.0 from the change to templates & patches, and have a separate commit to apply that to the various addon versions?

[emosbaugh]

IIUC, this change is effective for k8s 1.21? Wonder if we can split out the content of addons/prometheus/0.55.0-34.10.0 from the change to templates & patches, and have a separate commit to apply that to the various addon versions?

This was a work in progress. I made some progress on fixing a few false positive alerts but not all. I can't remember if this was only for <= 1.21 or all versions of k8s. Yes I would suggest splitting out template code. This PR is not ready and needs more work.