GitHub GraphQL changes impacting gh pr merge use

[rob-johansen]

Describe the bug

We have a GitHub Action that is suddenly failing to merge PRs. Here's the relevant configuration from the yaml file:

...
    env:
      PR_URL: ${{github.event.pull_request.html_url}}
      GITHUB_TOKEN: ${{secrets.<redacted>}}
    steps:
      - name: merge
        run: gh pr merge --admin --squash --delete-branch "$PR_URL"

Steps to reproduce the behavior

1. Type the above command.

Expected vs actual behavior

The PR is merged and the branch is deleted.

Logs

Run gh pr merge --admin --squash --delete-branch "$PR_URL"
  gh pr merge --admin --squash --delete-branch "$PR_URL"
  shell: /usr/bin/bash -e {0}
  env:
    PR_URL: <redacted>
    GITHUB_TOKEN: ***
GraphQL: Field 'isMergeQueueEnabled' doesn't exist on type 'PullRequest' (query PullRequestByNumber.repository.pullRequest.isMergeQueueEnabled)
Error: Process completed with exit code 1.

[williammartin]

Thanks for creating this, it is being discussed over from #8352 (comment) down but I'm actually going to redirect that conversation to this specific PR since the original issue was for a slightly different reason.

[marcelodeaguiar]

It is most probably related to the Token,
We simulated login in with my user instead of the token on our automation and the commands run normally.

[williammartin]

I believe we've identified the platform change that has caused this and are working on a remediation. Updates will be provided in this issue moving forwards.

[scotthaleen]

Several others (myself included) started experiencing this issue today as well. Comments are on #5778

[andyfeller]

Note

Monolith changes to restore this capability are enqueued with thanks to cross-team efforts; will watch and report back 🙇

[andyfeller]

@mrsiesta, @marisbest2, @epalacio90, @rwong2888, @kalawoodson, @bmarick, @langchain-infra, @skalpin, @gpamit, @rob-johansen, @scotthaleen : the fix is in, I've done some testing on my own repos locally and met with success. I'd be grateful if you could confirm 🙇

[mrsiesta]

@andyfeller confirmed resolved, thank you for your prompt work to resolve this issue.

[andyfeller]

All the credit goes to @williammartin and @mrgilman to getting down to the problem and fixing it! 🎉

[p-linnane]

Confirming everything is back in working order for Homebrew. Thank you all!

[bmarick]

@mrsiesta, @marisbest2, @epalacio90, @rwong2888, @kalawoodson, @bmarick, @langchain-infra, @skalpin, @gpamit, @rob-johansen, @scotthaleen : the fix is in, I've done some testing on my own repos locally and met with success. I'd be grateful if you could confirm 🙇

Confirmed working in my environment

[andyfeller]

Confirming everything is back in working order for Homebrew. Thank you all!

feels good to take care of @p-linnane instead of always being taken care of ❤️

[langchain-infra]

works for me thank you!

[skalpin]

It is working for us as well. Thank you again for your prompt response!

[scotthaleen]

Fixed for me as well. Thank you.

[gpamit]

Worked for me thank you getting this fixed in such a short time!

[rob-johansen]

Fixed for me too. Thank you!

[mcdurdin]

Yes, fixed for us too e.g. keymanapp/help.keyman.com#962. Thanks for the quick turnaround!

[rwong2888]

Works for us as well!

[twfahey1]

Thanks for quick work on this one team!!

[williammartin]

Firstly I'd like to apologise for the inconvenience this cause you all and thank you for your patience while we investigated and fixed it. I know that having broken core functionality is extremely frustrating.

Now that this fire is out, we'll take some time to understand how we can avoid similar failures in the future.

Thanks everyone for your input here. It's immensely valuable that we have a direct line with the community in moments like this. Cheers.

[dwise1056]

This issue has been affecting our enterprise cloud EMU instance. The following commands have no issue

gh pr create --title "${WIP_ID} ${pr_added_title}" --body-file "${WORKSPACE}/${WIP_ID}_pr_body" --base ${target_branch}
gh pr edit ${pr_number} --add-reviewer ${REVIEWER_ID}
gh pr review --approve ${pr_number} --body "$WIP_ID ${pr_noun} by $REQUESTOR
Executed by ${JOB_AUTOMATION}"

The command experiencing an issue is
gh pr merge --auto --squash --delete-branch ${pr_url}

It returns
GraphQL: Field 'isMergeQueueEnabled' doesn't exist on type 'PullRequest' (query PullRequestByNumber.repository.pullRequest.isMergeQueueEnabled)

The merge occasionally works for some Org repositories. The delete-branch never works for any repositories.

After investigation it appears the issue is related to recent GitHub changes regarding fined-grained tokens and Org level rulesets. Our issue coincided with updating tokens used by an automated process. The new tokens were created as fine-grained tokens, and assigned more permissions than what should be needed to perform the merge and delete.

Below I shared differences between our least affected repository and a repository affected 100% of the time

Before I get to those differences, I did find a work around that resolved the issue for all our repositories. I created a legacy token (classic), and that token has no issue executing the merge and delete-branch. It does exactly what is expected.

As for the differences when using a fined-grained token there was a lot of inconsistency. It happens to some repositories more than others. Repositories with advanced security turned on seem to encounter this less. These same repositories do not seem to inherit rulesets created at the Org level.

Here are diffs between a repository intermittently affected vs the one affected 100% of the time (when using fined-grained tokens).

Repository intermittently affected
from ${GITHUB_API_BASE_URL}/orgs/${ORG_OWNER}/repos
"security_and_analysis": {
"advanced_security": {
"status": "enabled"
},
"secret_scanning": {
"status": "enabled"
},

from ${GITHUB_API_BASE_URL}/repos/${ORG_OWNER}/${REPO_NAME}/branches/${BASE_BRANCH}/protection"
no differences between the 2 repositories. the API confirms they are exactly the same

${GITHUB_API_BASE_URL}/repos/${GITHUB_ORG_OWNER}/${REPO_NAME}/rulesets/rule-suites
Returns nothing. There are no rulesets created at the repo level. We apply org level rulessets. Those rulesets do not appear in the returned API for advanced security enabled repositories.

Repository always affected
from ${GITHUB_API_BASE_URL}/orgs/${ORG_OWNER}/repos
"security_and_analysis": {
"advanced_security": {
"status": "disabled"
},
"secret_scanning": {
"status": "disabled"
},

from ${GITHUB_API_BASE_URL}/repos/${ORG_OWNER}/${REPO_NAME}/branches/${BASE_BRANCH}/protection"
The below ruleset shows up in this API return. The ruleset is created at the org level for all repositories.
"ref": "refs/heads/${BASE_BRANCH}",
"result": "pass",
"evaluation_result": "fail"

${GITHUB_API_BASE_URL}/repos/${GITHUB_ORG_OWNER}/${REPO_NAME}/rulesets/rule-suites/${rule_suite_id}
"ref": "refs/heads/${BASE_BRANCH}",
"result": "pass",
"evaluation_result": "fail"
"rule_evaluations": [
{
"rule_source": {
"type": "ruleset",
},
"enforcement": "evaluate",
"result": "fail",
"rule_type": "pull_request",
"details": "Changes must be made through a pull request."
},
{
"rule_source": {
"type": "ruleset",
},
"enforcement": "evaluate",
"result": "pass",
"rule_type": "non_fast_forward"
},
{
"rule_source": {
"type": "ruleset",
},
"enforcement": "evaluate",
"result": "pass",
"rule_type": "deletion"
}

[williammartin]

@dwise1056 can you create a new issue please, this one is for a platform change that went out yesterday that broke any gh pr merge command that was not using an oauth token or a PAT owned by someone that had never been added to the original MQ feature flag.

[dwise1056]

@williammartin sure, will do.

[marisbest2]

I must say that it was pretty impressive and inspiring that this issue was first reported at 424 EST and fully resolved at 723 EST from a company that doesn't really have a direct support line except through Issues and I'm sure is pretty complex to ship a fix. Great work @williammartin and @andyfeller and @mrgilman

[williammartin]

Just a note here that if you happened to see this in the past short while or do see it in the next short while it's likely a result of a new deployment going out in stages with a request hitting a canary (that says MQ is available) and then a follow up request hitting production (that doesn't support MQ). In theory the deployment should be complete and this should no longer happen so if you do see it again from this comment timestamp onwards, please let me know.