Jobs running with "labeled" trigger get triggered multiple times

[mkolesnik]

Turns out all our "conditional on label" jobs [1] are getting triggered each time a label is added, so when doing a release the automation actually adds 2 labels ready-to-test and e2e-all-k8s.

[1] (e.g. https://github.com/submariner-io/submariner-operator/blob/devel/.github/workflows/e2e-full.yml#L4-L11)

This action from the CLI perspective is just one action - create the PR with 2 labels, but behind the scenes it translates to 3 GHA trigger events:

1. Open the PR
2. Label PR with ready-to-test
3. Label PR with e2e-all-k8s

In this case for the E2E Full job:

1. Workflow is skipped (condition is false)
2. Workflow runs (condition is true)
3. Workflow runs again (condition is still true)

Only the last workflow's results are displayed on the "Conversation" page in the PR, but if you go to the "Checks" page you can see all workflows.
This means that ~10 jobs are being run extraneously.On operator repository it's even worse since the "All K8s" workflow is being run twice and that has 25 jobs (maybe we should trim that a bit?).

The reason it runs twice is probably due to a race between the labeling itself and generation of the event for GHA, so by the time the event is generated for labeling with ready-to-test the PR already has e2e-all-k8s present, and that gets sent in the payload as well (that's my hypothesis).
The situation is even worse, though, since for any PR that already has one of the labels present, any additional labeling by users will run the workflow again, even when the PR is closed.

So, what can we do?

• We could live with it, wasting time and resources needlessly (not ideal).
• We could stop using label conditionals and just run all the jobs, but the rationale for doing this was we don't want to run the whole CI each time, only when the PR is ready to be merged.
• We could have more sophisticated rules to trigger the jobs only when the PR gets labeled, e.g.:

  if: (github.event.action == 'labeled' && github.event.label.name == 'ready-to-test') || (github.event.action != 'labeled' && contains(github.event.pull_request.labels.*.name, 'ready-to-test'))
  

  But this approach will make "Conversation" view hide any previous run, e.g. when adding 2 labels, if the first one triggered the workflow and the second skipped if, only the skipped one is shown.
  This is also a major problem in case any of the jobs is under marked required under branch protection rule(s), since the last run takes precedence and will cause PR to be un-mergable.
• We could remove labeled as a possible trigger, but then any new PR will necessarily have no labels (as they're not part of the create API), and thus adding labels wont trigger the job.
• We could work around this by using another trigger such as ready_for_review which would mean that to trigger the jobs we need to convert to draft and then back.

[skitt]

We should be able to configure concurrency to cancel previous jobs on the same hash in the same group; see https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#example-using-concurrency-to-cancel-any-in-progress-job-or-run. This isn’t perfect since we’d still start too many jobs, but for double-labeling we would quickly cancel one of each pair.

[mkolesnik]

We should try that and see if it'll help alleviate the problem, though it wont help if an unrelated label is added (jobs will still trigger)

[stale]

This issue has been automatically marked as stale because it has not had activity for 60 days. It will be closed if no further activity occurs. Please make a comment if this issue/pr is still valid. Thank you for your contributions.