Enforce timeout and notify when job is waiting for a runner to pick up the job

[radhikari-arch]

Select Topic Area

Question

Body

We have a situations for various reasons the self hosted runners run into issues and jobs are waiting for several hours to pick up the runners with message "Waiting for a runner to pick up this job...". Is there something we can enforce timeout for this? The timeout in job works only when jobs actually started to run. But I am looking for a solution where several jobs if waiting for the runners for sometime would like to timeout and get notified.
And also is there a way we can monitor the queue that are waiting for the runners?

[enviGit]

Hi Rajendra!

You can enforce a timeout for jobs that are waiting for a runner to pick up the job by setting the idle_timeout parameter in your self-hosted runner's configuration file.

To set the idle_timeout for the runner, you will need to modify the runner's configuration file. The configuration file is located in the RUNNER_HOME directory on the machine where the runner is installed. You can set the idle_timeout parameter to the number of seconds that you want to allow the runner to be idle before it is automatically removed from the queue.

For example, to set the idle_timeout to 60 minutes (3600 seconds), you can add the following line to your configuration file:

idle_timeout = 3600

When a runner has been idle for the specified amount of time, it will be automatically removed from the queue and the job will be marked as failed.

As for monitoring the queue of jobs waiting for runners, you can use the GitHub API to retrieve a list of pending jobs. The API endpoint for this is:

GET /repos/{owner}/{repo}/actions/runs/{run_id}/jobs

This will return a list of jobs for the specified run, including their current status (e.g. waiting, in progress, completed). You can use this information to monitor the queue of jobs waiting for runners and take appropriate action if necessary.

[enviGit]

Thank you for providing more context. In that case, you can add a timeout to your job using the timeout-minutes parameter in your workflow YAML file. This parameter specifies the number of minutes a job is allowed to run before it is automatically cancelled. For example, to set the timeout to 30 minutes, you can add the following line to your YAML file:

jobs:
  your_job_name:
    runs-on: self-hosted
    timeout-minutes: 30
    steps:
      # your job steps here

[radhikari-arch]

timeout-minutes activates only when job finds a runner and starts running. In my case, job is just waiting for the runner. Example, this job has been waiting for last 6 hours and shows "Waiting for a runner to pick up this job..."

name: Testing timeout

on:
  workflow_dispatch:

jobs:
    job1:
      name: Job Title
      runs-on: [self-hosted, testing]
      timeout-minutes: 5

      steps:
        - name: Echo out the input variables
          run: |
            echo "place holder"

Looking for timing out and not wait several hours before we know this job did not find its runner.

[enviGit]

You can try to use a combination of the if and needs syntax to implement a timeout for jobs waiting for a runner.

Here is an example:

name: Testing timeout

on:
  workflow_dispatch:

jobs:
  job1:
    name: Job Title
    runs-on: [self-hosted, testing]
    timeout-minutes: 5
    steps:
      - name: Echo out the input variables
        run: |
          echo "place holder"

  job2:
    name: Check for runner
    runs-on: [self-hosted, testing]
    needs: job1
    if: ${{ github.job_status == 'Waiting' }}
    timeout-minutes: 30
    steps:
      - name: Check if runner is available
        run: |
          # Check if runner is available and exit with code 0 if it is
          # Otherwise, exit with a non-zero code to indicate failure
    if: ${{ job2.outcome == 'success' }}

  job3:
    name: Run job after runner is available
    runs-on: [self-hosted, testing]
    needs: job2
    if: ${{ job2.outcome == 'success' }}
    steps:
      - name: Run your job
        run: |
          # Run your job here
    if: ${{ job3.outcome != 'failure' }}

  job4:
    name: Notify if runner is still unavailable
    runs-on: [self-hosted, testing]
    needs: job2
    if: ${{ job2.outcome == 'failure' }}
    steps:
      - name: Send notification
        run: |
          # Send a notification that the runner is still unavailable
          # For example, you could use a Slack or email integration
    if: ${{ job4.outcome != 'failure' }}

In this example, job2 checks for the runner and sets its outcome to success if it's available within the timeout period. If the runner is still unavailable, job2 times out and its outcome is set to failure.

job3 runs your job only if the runner is available (i.e., job2.outcome == success), and job4 sends a notification if the runner is still unavailable (i.e., job2.outcome == failure).

I hope this helps! Let me know if you have any further questions.

[radhikari-arch]

job2 does not even execute until job1 is complete. And github context does not even have the github.job_status. And context of job seems to be limited within the current running jobs. Jobs can't even look for another job context and won't able to know the status.

[JulianJvn]

@radhikari-arch Pretty sure you're talking to an AI chatbot there.

[develop-at-github]

Hi @radhikari-arch,

I have been facing similar problem and created a workaround which is working fine for me so far. It might not be an optimal way, but this is what I did:

Create another workflow, say timeout.yml with following contents

name: Cancel deploy on timeout

on:
    push:
        branches: [main]
    workflow_dispatch:

jobs:
    timeout:
        timeout-minutes: 2
        runs-on: ubuntu-latest
        permissions:
          actions: write
        steps:
            - name: Monitor deployment for timeout and cancel if crossed the threshold
              env:
                GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
              run: |
                sleep 1m
                gh -R {owner}/{repo} run list -w deploy.yml -s queued --json databaseId -q .[].databaseId | xargs gh -R {owner}/{repo} run cancel

How it works?

1. This new workflow runs along with other workflows, depending on the triggers defined.
2. It then waits/sleeps for 1 minute (can be changed as per the use case)
3. Upon waking up, it cancels the deploy.yml workflow if it is still queued.

You can probably use this simple workflow as base and modify as per your needs by applying checks on different statuses and varying timeouts per status.

Hope it helps!

[colemickens]

Another missing feature that makes me wonder how Actions shipped, let alone has stayed in such a similar state for years.

It's like observability of Actions is actually an anti-feature in Microsoft's eyes. Just like the ridiculous way failed job notifications are sent and are rotely unconfigurable, it's outlandish that there's no way to timeout STALLED JOBS that can't run.

[ceejatec]

FYI I adapted @develop-at-github 's suggestion for a scheduled cleanup job that runs hourly. The workflow is in cleanup.yml, and the top of the workflow looks like this:

name: cleanup_self_hosted_agents
on:
  schedule:
    # Run at 10 minutes past every hour
    - cron: '10 * * * *'

jobs:
  # This job cancels any already-queued copies of this workflow. This
  # prevents runaway queues and also cancels any jobs that failed to
  # start due to the corresponding worker being offline, etc.
  prune:
    runs-on: ubuntu-latest
    env:
      GH_TOKEN: ${{ github.token }}
    steps:
      - run: |-
          gh -R couchbasecloud/github-actions-playground run list -w cleanup.yml -s queued --json databaseId -q '.[].databaseId' | sort -nr | tail -n +2 | xargs -r -n 1 gh -R couchbasecloud/github-actions-playground run cancel

That kills any outstanding runs of the workflow, not including (via the tail -n +2) the most recent run which is the run that this job is in. That way the first thing the workflow does is kill any earlier runs that are still queued.

I do run this job on ubuntu-latest so it never gets hung up itself, even if all my self-hosted runners are busy. This eats a few GHA minutes a month, but worth it.

I do agree with @colemickens that this is functionality that should be built in to GHA.

[github-actions[bot]]

🕒 Discussion Activity Reminder 🕒

This Discussion has been labeled as dormant by an automated system for having no activity in the last 60 days. Please consider one the following actions:

1️⃣ Close as Out of Date: If the topic is no longer relevant, close the Discussion as out of date at the bottom of the page.

2️⃣ Provide More Information: Share additional details or context — or let the community know if you've found a solution on your own.

3️⃣ Mark a Reply as Answer: If your question has been answered by a reply, mark the most helpful reply as the solution.

Note: This dormant notification will only apply to Discussions with the Question label. To learn more, see our recent announcement.

Thank you for helping bring this Discussion to a resolution! 💬