Q4 Reduced workload goal - Nov 29 Sprint 4 tracking issue

[GeorgianaElena]

Follow-up to #3467

I. Wrap-ups:

Tasks

Preview Give feedback

1. Wrap up PR about event issue template update with follow-ups tasks listed in the PR description
2. Wrap up PR about central grafana doubled 2i2c datasource. Now, there's a phantom "default" datasource that appears in the dashboads, but NOT in the data source list, which is confusing. Investigate what that's about.
3. Document a k8s upgrade policy: ensure to always use one of the latest 5 minor versions #3248
   nominated-to-be-resolved-during-q4-2023 +1
   

II. Grafana & monitoring:

1. We have two strategies documented and we still need to decide what is "the right one" and document it for the community on docs.2i2c.org website:

• https://infrastructure.2i2c.org/howto/grafana-github-auth/
• https://infrastructure.2i2c.org/sre-guide/support/grafana-account/

Tasks

Preview Give feedback

1. Decide on a path for enabling GitHub auth to grafana for Community Reps of hubs on dedicated clusters #1850
   nominated-to-be-resolved-during-q4-2023 tech:grafana +2
   
2. [docs] Clarify infrastructure documentation around Grafana access #3505
   +0
   
3. Document Grafana access for communities docs#198
   +0
   

2. Very important maintenance work related to Grafana & prometheus atm IMO is:

Tasks

Preview Give feedback

1. Move from grafonnet-lib to grafonnet jupyterhub/grafana-dashboards#89 (Helping Yuvi push forward on: Upgrade from grafonnet-lib to grafonnet 10.4.0 jupyterhub/grafana-dashboards#90)
2. Grafanas show empty due to how variables are refreshed #3237
   nominated-to-be-resolved-during-q4-2023 +1
   

   
3. Document the steps needed to perform Migrate all GCP hubs prometheus disks to pd-balanced from pd-standard #3119 with the help of Yuvi and start doing it as this will also solve Correctly size prometheus disks on all clusters and Make sure prometheus PVs of all our clusters have deletionPolicy set to retain #2717

III. Support process

Tasks

Preview Give feedback

1. Open an issue about creating a very simple issue template for issues that get created as part of a ticket not being solved during the first 30m time frame: [maint-support] Create an issue template for support tickets #3477
2. [maint-support] Create an issue template for support tickets #3477
   +0
   
3. Open an issue about creating an automation to add this tickets to the relevant project boards with the right properties for proper planning in a timely manner (they should now happen manually and I don't think we're very consistent about doing it) -> Automate triager project board tasks #3586