Plan fails if previous commit failed policy checks

[nightmarlin-wise]

Community Note

• Please vote on this issue by adding a 👍 reaction to the original issue to help the community and maintainers prioritize this request. Searching for pre-existing feature requests helps us consolidate datapoints for identical requirements into a single place, thank you!
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.
• If you are interested in working on this issue or have submitted a pull request, please leave a comment.

---

Overview of the Issue

If the previous commit failed policy checks, auto-plan fails with the message

**Ran Plan for dir**: aws/playground/policy-brick workspace: default

**Plan Failed**: All policies must pass for project before running plan.

Reproduction Steps

• Set up an atlantis instance with at least one element of repos.yaml#/policies/policy_sets defined, and has auto-plan enabled
• Open a PR that fails the policy check
• Wait for the policy checks to fail, then push a new commit
• Observe that atlantis produces the above error

I suspect this will also fail if auto-plan is disabled and a manual atlantis plan is run - will see if I can verify this.
A subsequent atlantis plan or push that triggers auto-plan is successfully planned

Logs

Logs

// policy check error on first commit
{"level":"info","caller":"events/events_controller.go:559","msg":"Handling GitHub Pull Request 'opened' event","json":{"gh-request-id":"X-Github-Delivery=REDACTED","repo":"transferwise/repo-name","pull":"216"}}
["omitted... plan & init runs as normal"]
{"level":"error","caller":"events/instrumented_project_command_runner.go:84","msg":"Failure running policy_check operation: Some policy sets did not pass.","json":{"repo":"transferwise/repo-name","pull":"216"},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:84\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).PolicyCheck\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:42\ngithub.com/runatlantis/atlantis/server/events.runProjectCmdsParallel.func1\n\tgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:29"}

// plan error
{"level":"info","caller":"events/events_controller.go:559","msg":"Handling GitHub Pull Request 'updated' event","json":{"gh-request-id":"X-Github-Delivery=REDACTED","repo":"transferwise/repo-name","pull":"216"}}
["omitted... atlantis pulls latest version, discovers updated file & sets up commands - but no policy checks are actually run"]
{"level":"debug","caller":"events/plan_command_runner.go:129","msg":"deleting previous plans and locks","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"debug","caller":"events/project_command_context_builder.go:200","msg":"Building project command context for policy_check","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"debug","caller":"events/project_command_context_builder.go:98","msg":"Building project command context for plan","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"info","caller":"vcs/github_client.go:940","msg":"Updating GitHub Check status for 'atlantis/plan: aws/playground/policy-brick/default' to 'pending'","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"info","caller":"events/plan_command_runner.go:139","msg":"Running plans in parallel","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"debug","caller":"vcs/github_client.go:950","msg":"POST /repos/transferwise/repo-name/statuses/REF returned: 201","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"debug","caller":"events/working_dir.go:109","msg":"clone directory '/home/atlantis/.data/repos/transferwise/repo-name/216/default' already exists, checking if it's at the right commit","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"debug","caller":"events/project_command_runner.go:576","msg":"acquired lock for project","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"info","caller":"events/project_locker.go:86","msg":"Acquired lock with id 'transferwise/repo-name/aws/playground/policy-brick/default'","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"info","caller":"events/working_dir.go:117","msg":"repo is at correct commit \"REF\" so will not re-clone","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"debug","caller":"events/working_dir.go:299","msg":"Comparing PR ref \"REF\" to local ref \"REF\"","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"info","caller":"vcs/github_client.go:940","msg":"Updating GitHub Check status for 'atlantis/plan: aws/playground/policy-brick/default' to 'failure'","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"info","caller":"events/plan_command_runner.go:146","msg":"deleting plans because there were errors and automerge requires all plans succeed","json":{"repo":"transferwise/repo-name","pull":"216"}}
{"level":"error","caller":"events/instrumented_project_command_runner.go:84","msg":"Failure running plan operation: All policies must pass for project before running plan.","json":{"repo":"transferwise/repo-name","pull":"216"},"stacktrace":"github.com/runatlantis/atlantis/server/events.RunAndEmitStats\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:84\ngithub.com/runatlantis/atlantis/server/events.(*InstrumentedProjectCommandRunner).Plan\n\tgithub.com/runatlantis/atlantis/server/events/instrumented_project_command_runner.go:38\ngithub.com/runatlantis/atlantis/server/events.runProjectCmdsParallel.func1\n\tgithub.com/runatlantis/atlantis/server/events/project_command_pool_executor.go:29"}

Happy to provide additional logs upon request.

Environment details

If not already included, please provide the following:

• Atlantis version: v0.37.1
• Deployment method: ecs

Atlantis server-side config file:

repos:
  - id: /.*/
    branch: /^(main|master)$/
    apply_requirements: [approved, mergeable]
    workflow: default

policies:
  owners:
    teams:
      - my-team
  policy_sets:
    - name: aws
      path: policy # local path, ignored when `--update` is used
      source: local

workflows:
  default:
    plan:
      steps:
        - init
        - plan
        - show

    apply:
      steps:
        - apply

    policy_check:
      steps:
        - show
        - policy_check:
            extra_args:
              - "--update"
              - "${opa_policy_url}"
              - "-d"
              - "./policy/data.json"
              - "--namespace"
              - "${namespace}"

metrics:
  prometheus:
    endpoint: "/metrics"

Additional features:

• We have enabled parallel plan & apply
• We have enabled auto-discovery & autoplan-modules
• We have disabled the Terraform plugin cache
• We have allowed atlantis to ignore failed atlantis/apply checks when checking if a PR is mergeable

Additional Context

I believe this was introduced by #5851 - which changed the behaviour to validate that policy checks are passing before running the plan command.

Fixes here could be

• remove the valid.PoliciesPassedCommandReq if present in ctx.PlanRequirements when passed to DefaultCommandRequirementHandler.ValidatePlanProject
  • this has the smallest scope, but if it's possible to run a policy_check before a plan, it may cause a regression for that use case (i can't tell if that is the case)
• not inject it at the top-level for plans (in valid.NewGlobalCfgFromArgs`) when policy checks are enabled
  • this injection has been around for a while though, so maybe best not to change it
• clear policy check statuses from previous runs when using autoplan
  • this would ensure that if a policy_check can be run before a plan, then only policy failures from that run would block the plan

[nightmarlin-wise]

accidentally created slightly early - tidying... tidied!

[nightmarlin-wise]

I tried explicitly setting repos[*].plan_requirements: [] in the server-side repos.yaml, but this still got overwritten somewhere because policy checks are enabled. not entirely sure where this happened

repos:
  - id: /.*/
    branch: /^(main|master)$/
    plan_requirements: []
    apply_requirements: [approved, mergeable]
    workflow: default

[dosubot]

This happens because when policy checks are enabled, Atlantis always injects the policies_passed requirement into the plan requirements at the global config level, regardless of what you set in repos.yaml. Even if you set plan_requirements: [] for a repo, the merging logic in the config parser appends any global requirements (including policies_passed) unless you explicitly disable policy checks for that repo with PolicyCheck: false. So, the only way to prevent this enforcement is to set PolicyCheck: false for the repo in question. This behavior is by design in the current codebase and not something that can be overridden just by setting plan_requirements to an empty list.
Relevant code reference

To reply, just mention @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}

   

[nightmarlin-wise]

the AI mostly just restates my findings. thanks AI.
but given this behaviour was only added recently I would consider it not to be "by design"

i've implemented the smallest possible fix in #5995 , but recognise it may not be correct for all cases and would be happy to discuss other approaches