Workflow Runs API pagination broken with created and status filters

[abhijit-hota]

Select Topic Area

Bug

Body

I'm trying to fetch all workflow runs for a repository with created and status filters. The corresponding URLs looks like this:

https://api.github.com/repos/ORG/REPO/actions/runs?created=<created>&page=<n>&per_page=100&status=completed

The created filter looks like >=2023-07-13T15:04:05Z07:00. For page 1 to 10, I receive hundred workflow runs per page. But for page > 10, I don't receive any workflow runs.

> go run .

[Last Page: 19]	[Total runs: 1872]

[Page: 1/19]	[100 runs]
[Page: 2/19]	[100 runs]
[Page: 3/19]	[100 runs]
[Page: 4/19]	[100 runs]
[Page: 5/19]	[100 runs]
[Page: 6/19]	[100 runs]
[Page: 7/19]	[100 runs]
[Page: 8/19]	[100 runs]
[Page: 9/19]	[100 runs]
[Page: 10/19]	[100 runs]
[Page: 11/0]	[0 runs]
[Page: 12/0]	[0 runs]
[Page: 13/0]	[0 runs]
[Page: 14/0]	[0 runs]
[Page: 15/0]	[0 runs]
[Page: 16/0]	[0 runs]
[Page: 17/0]	[0 runs]
[Page: 18/0]	[0 runs]
[Page: 19/0]	[0 runs]

Golang code I used for above output

This uses go-github

func fetchWorkflowRuns(ctx context.Context) {

	client := github.NewClient(nil) // With appropriate auth
	
	results := make([]string, 0)

	sixMonthsAgo := time.Now().AddDate(0, -6, 0)
	createFilter := fmt.Sprint(">=", sixMonthsAgo.Format(time.RFC3339))

	lastPage := 1
	for page := 1; page <= lastPage; page++ {
		page := page

		wruns, resp, err := client.Actions.ListRepositoryWorkflowRuns(
			ctx, ORG, REPO,
			&github.ListWorkflowRunsOptions{
				Created:             createFilter,
				Status:              "completed",
				ListOptions: github.ListOptions{
					Page:    page,
					PerPage: 100,
				},
			},
		)
		if err != nil {
			panic(err)
		}

		if page == 1 {
			lastPage = resp.LastPage
			fmt.Printf("[Last Page: %d]\t[Total runs: %d]\n", lastPage, wruns.GetTotalCount())
		}

		fmt.Printf("[Page: %d/%d]\t[%d runs]\n", page, resp.LastPage, len(wruns.WorkflowRuns))

		for _, wrun := range wruns.WorkflowRuns {
			results = append(results, wrun.GetID())
		}
	}

	writeJsonFile(fmt.Sprintf("%s.json", REPO), results)
}

As you can see, even if the first query returns a total of 19 pages and 1872 workflow runs, pages after 10 give no result. Interestingly, if I remove the two filters, the pagination works as intended.

Appreciate any kind of help and/or fix.

[kmadaminjanova0508]

Good time of your day @abhijit-hota
I think you are having an issue with the pagination of the GitHub API when using the created and status filters in combination. The behavior you are observing, where pages after 10 return no results, could be related to how the GitHub API handles these filters.I think one potential explanation for this behavior is that the combination of the created and status filters might be affecting the pagination results. It's possible that the API is not able to accurately paginate the results when both filters are applied, leading to the unexpected behavior you are seeing.To troubleshoot this issue, I recommend trying to Review GitHub API documentation.I mean check the official GitHub API documentation to ensure that you are using the created and status filters correctly. It's possible that there are specific limitations or considerations when using these filters together.or you can also try to test individual filters by making separate API requests with only the created filter and then with only the status filter to see if the pagination works as expected in each case. This will help determine if the issue is related to the combination of filters.Furthmore if you find that the combination of filters is causing issues with pagination, consider alternative approaches such as fetching all runs without filters and then filtering the results locally in your code :)

[davidmreed]

I observe the same behavior, albeit using only a single filter (status="completed"), and from an entirely different stack (Python + github3.py).

[dstopka]

I run into the same issue when listing workflow runs both with curl and go.

[noe-charmet]

Here to report that I am encountering the same issue.

I have been reproducing the issue on the Apache Airflow repository.

From my understanding things break when two conditions are met

• Either the created or the status filter are applied
• We reach the 1000th element with pagination.

For example here when looking for the repositories workflows with a completed status, 100 element per page.

• Page 10 returns 100 results with a total count of 62681
• Page 11 and onwards return 0 results and 0 total count.

The link header is present on page 10 but not on page 11 and onwards. The links it provides look to be the expected ones.

<https://api.github.com/repositories/33884891/actions/runs?per_page=100&page=9&status=completed>; rel="prev", <https://api.github.com/repositories/33884891/actions/runs?per_page=100&page=11&status=completed>; rel="next", <https://api.github.com/repositories/33884891/actions/runs?per_page=100&page=627&status=completed>; rel="last", <https://api.github.com/repositories/33884891/actions/runs?per_page=100&page=1&status=completed>; rel="first"

I have also tested with varying values for the per_page parameter (10, 30, 50, 100). In all cases the results break once we reach ~1000 results.

Removing all filters allow pagination on all the range.

[noe-charmet]

This can also be reproduced using the Github UI.

Once again on the Apache Airflow repository.

If you go the page 40 of workflow runs it works fine.

Once you reach page 41, you get this error screen inviting you to do additional filtering.