2i2c-aws-us: investigate and fix failure to update grafana dashboards

[consideRatio]

@yuvipanda recently updated grafana dashboards in jupyterhub/grafana-dashboards#100, so I triggered a github workflow to re-deploy them - but it failed for 2i2c-aws-us. It took a long time, and then failed.

I've seen this happen before when the requests became larger than 1MB or so, which was then fixed by an annotation on the Ingress resource for grafana - is it something like that happening again? It succeeds for all other hubs though, and isn't the content the same? Is the content depending on whats presented currently in the dashboards? Hmmm...

$ deployer grafana deploy-dashboards 2i2c-aws-us

Cloning jupyterhub/grafana-dashboards...
Cloning into 'grafana-dashboards'...
remote: Enumerating objects: 855, done.
remote: Counting objects: 100% (330/330), done.
remote: Compressing objects: 100% (142/142), done.
remote: Total 855 (delta 195), reused 280 (delta 179), pack-reused 525
Receiving objects: 100% (855/855), 51.32 MiB | 23.97 MiB/s, done.
Resolving deltas: 100% (495/495), done.
Deploying grafana dashboards to 2i2c-aws-us...
Deployed dashboards/user.jsonnet
Traceback (most recent call last):
  File "/home/erik/dev/2i2c-org/infrastructure/grafana-dashboards/./deploy.py", line 210, in <module>
    main()
  File "/home/erik/dev/2i2c-org/infrastructure/grafana-dashboards/./deploy.py", line 205, in main
    deploy_dashboard(dashboard, folder['id'], api)
  File "/home/erik/dev/2i2c-org/infrastructure/grafana-dashboards/./deploy.py", line 109, in deploy_dashboard
    db = populate_template_variables(api, db)
  File "/home/erik/dev/2i2c-org/infrastructure/grafana-dashboards/./deploy.py", line 172, in populate_template_variables
    labels = get_label_values(api, prom_id, template_query)
  File "/home/erik/dev/2i2c-org/infrastructure/grafana-dashboards/./deploy.py", line 135, in get_label_values
    metrics = api(proxy_url)['data']
  File "/home/erik/dev/2i2c-org/infrastructure/grafana-dashboards/./deploy.py", line 37, in grafana_request
    with urlopen(req, data, context=ctx) as resp:
  File "/home/erik/miniforge3/lib/python3.10/urllib/request.py", line 216, in urlopen
    return opener.open(url, data, timeout)
  File "/home/erik/miniforge3/lib/python3.10/urllib/request.py", line 519, in open
    response = self._open(req, data)
  File "/home/erik/miniforge3/lib/python3.10/urllib/request.py", line 536, in _open
    result = self._call_chain(self.handle_open, protocol, protocol +
  File "/home/erik/miniforge3/lib/python3.10/urllib/request.py", line 496, in _call_chain
    result = func(*args)
  File "/home/erik/miniforge3/lib/python3.10/urllib/request.py", line 1391, in https_open
    return self.do_open(http.client.HTTPSConnection, req,
  File "/home/erik/miniforge3/lib/python3.10/urllib/request.py", line 1352, in do_open
    r = h.getresponse()
  File "/home/erik/miniforge3/lib/python3.10/http/client.py", line 1375, in getresponse
    response.begin()
  File "/home/erik/miniforge3/lib/python3.10/http/client.py", line 318, in begin
    version, status, reason = self._read_status()
  File "/home/erik/miniforge3/lib/python3.10/http/client.py", line 287, in _read_status
    raise RemoteDisconnected("Remote end closed connection without"
http.client.RemoteDisconnected: Remote end closed connection without response
╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /home/erik/dev/2i2c-org/infrastructure/deployer/commands/grafana/deploy_dashboards.py:44 in      │
│ deploy_dashboards                                                                                │
│                                                                                                  │
│   41 │   deploy_script_env.update({"GRAFANA_TOKEN": grafana_token})                              │
│   42 │   try:                                                                                    │
│   43 │   │   print_colour(f"Deploying grafana dashboards to {cluster_name}...")                  │
│ ❱ 44 │   │   subprocess.check_call(                                                              │
│   45 │   │   │   ["./deploy.py", grafana_url],                                                   │
│   46 │   │   │   env=deploy_script_env,                                                          │
│   47 │   │   │   cwd="grafana-dashboards",                                                       │
│                                                                                                  │
│ /home/erik/miniforge3/lib/python3.10/subprocess.py:369 in check_call                             │
│                                                                                                  │
│    366 │   │   cmd = kwargs.get("args")                                                          │
│    367 │   │   if cmd is None:                                                                   │
│    368 │   │   │   cmd = popenargs[0]                                                            │
│ ❱  369 │   │   raise CalledProcessError(retcode, cmd)                                            │
│    370 │   return 0                                                                              │
│    371                                                                                           │
│    372                                                                                           │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
CalledProcessError: Command '['./deploy.py', 'https://grafana.aws.2i2c.cloud']' returned non-zero exit status 1.