Metrics Server failed for Prometeus Endpoint

[alfkonee]

I have two Knot instances with the same config but on one of the servers when trying to reach the metrics endpoint I get this error

 kresctl metrics
status: 500
body:
500 Internal Server Error

Server got itself in trouble 

with the journal logs giving

Aug 01 22:19:00 HOSTNAME aiohttp.server[495637]: Error handling request
                                                    Traceback (most recent call last):
                                                      File "/usr/lib64/python3.9/site-packages/aiohttp/web_protocol.py", line 452, in _handle_request
                                                        resp = await request_handler(request)
                                                      File "/usr/lib64/python3.9/site-packages/aiohttp/web_app.py", line 543, in _handle
                                                        resp = await handler(request)
                                                      File "/usr/lib64/python3.9/site-packages/aiohttp/web_middlewares.py", line 114, in impl
                                                        return await handler(request)
                                                      File "/usr/lib64/python3.9/site-packages/knot_resolver_manager/server.py", line 62, in error_handler
                                                        return await handler(request)
                                                      File "/usr/lib64/python3.9/site-packages/knot_resolver_manager/server.py", line 243, in _handler_metrics
                                                        body=await statistics.report_stats(),
                                                      File "/usr/lib64/python3.9/site-packages/knot_resolver_manager/statistics.py", line 338, in report_stats
                                                        await _resolver_collector.collect_kresd_stats()
                                                      File "/usr/lib64/python3.9/site-packages/knot_resolver_manager/statistics.py", line 96, in collect_kresd_stats
                                                        stats_raw = await command_registered_workers(cmd)
                                                      File "/usr/lib64/python3.9/site-packages/knot_resolver_manager/kresd_controller/registered_workers.py", line 34, in command_registered_workers
                                                        pairs = await asyncio.gather(*(single_pair(inst) for inst in _REGISTERED_WORKERS.values()))
                                                      File "/usr/lib64/python3.9/site-packages/knot_resolver_manager/kresd_controller/registered_workers.py", line 32, in single_pair
                                                        return sub.id, await sub.command(cmd)
                                                      File "/usr/lib64/python3.9/site-packages/knot_resolver_manager/kresd_controller/interface.py", line 197, in command
                                                        await writer.wait_closed()  # type: ignore
                                                      File "/usr/lib64/python3.9/asyncio/streams.py", line 359, in wait_closed
                                                        await self._protocol._get_close_waiter(self)
                                                      File "/usr/lib64/python3.9/site-packages/knot_resolver_manager/kresd_controller/interface.py", line 176, in command
                                                        _ = await reader.read(2)
                                                      File "/usr/lib64/python3.9/asyncio/streams.py", line 684, in read
                                                        await self._wait_for_data('read')
                                                      File "/usr/lib64/python3.9/asyncio/streams.py", line 517, in _wait_for_data
                                                        await self._waiter
                                                      File "/usr/lib64/python3.9/asyncio/selector_events.py", line 854, in _read_ready__data_received
                                                        data = self._sock.recv(self.max_size)
                                                    OSError: [Errno 22] Invalid argument
Aug 01 22:19:00 Hostname aiohttp.access[495637]: xxx.xxx.xxx.xxx [01/Aug/2024:22:19:00 +0000] "GET /metrics HTTP/1.1" 500 244 "-" "Prometheus/2.53.1"

Here is the config

rundir: /run/knot-resolver
workers: 16
cache:
  storage: /var/cache/knot-resolver
  size-max: 20G
  prefill:
    - origin: "."
      url: https://www.internic.net/domain/root.zone
      refresh-interval: 6h

logging:
  level: info
  groups:
    - system
    - net
    - dnssec
    - resolv
    - select
    - exterr
    - policy

network:
  listen:
    - interface: lo@53
      kind: dns
    - interface: ens192@53
      kind: dns
    - interface: ens192
      kind: dot

management:
  interface: 10.157.2.180@5000
  # unix-socket: /run/knot-resolver/manager.sock

[vcunat]

This error seems weird to me. If I got this, I'd probably first try to update the python3 package (as the asyncio package is always built in, I think).

Maybe @alesmrazek will have an idea.