listener pod stops recording the new metrics after sometime.

[RajatPorwal5]

Checks

• I've already read https://docs.github.com/en/actions/hosting-your-own-runners/managing-self-hosted-runners-with-actions-runner-controller/troubleshooting-actions-runner-controller-errors and I'm sure my issue is not covered in the troubleshooting guide.
• I am using charts that are officially provided

Controller Version

0.9.3

Deployment Method

ArgoCD

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions).
• I've read the Changelog before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes

To Reproduce

1. Deploy the GHA runner controller.
2. Deploy the GH RunnerSet and RunnerSet is running 1000s of jobs per hour.

Describe the bug

I have observed that if runnerset is running thousands of job per hour, listener stop recording the metrics after sometime. Listener continue to work as expected and I can see logs are getting updated as well. It just I can't see any metrics coming out of that listener pod.

There is another Runnerset which is running like ~20-30 jobs per hour, for that listener metrics are getting recorded properly.

Describe the expected behavior

Listener pod should continue to update the metrics.

Additional Context

NA

Controller Logs

NA

Runner Pod Logs

NA

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[nikola-jokic]

Hey @RajatPorwal5,

Can you please submit the listener log?