Pipeline jobs running before material check when triggered by another pipeline

[asnyder-hv]

Issue Type

• Bug Report

Summary

We've noticed an issue and been able to replicate it in dev, where pipelines can run before a git material check finishes causing them to run with outdated code. This seems to be limited to pipelines that are triggered by other pipelines.

Basic environment details

• Go Version: 22.2.0
• JAVA Version: 17.0.4
• OS: 20.04

Steps to Reproduce

# Note it's possible this may be somewhat intermittent if your material check happens to be very fast. If not reproduced repeat a few times from step 5.

1. Create a pipeline to act as a trigger ("pTrigger" from here out). The job isn't important, in my dev I have it "ls" a server. We use an empty git repo for a material when just running system commands.
2. Create a second pipeline ("pAction" from here out) that has pTrigger as a material as well as a git repo you can test commit to. For testing the job isn't important, in general it would do something with he git code.
3. Set the pipeline scheduling to automatic, then edit the git material and switch the deny list to a whitelist so that commits to the git repo do NOT trigger it.
4. Do an initial run of both pipelines so they are in a clean ready state.
5. Commit and push something to git so that changes should be ingested on the next run of pAction. Nothing should run at this point.
6. Manually run pTrigger, wait for it to complete, automatically triggering pAction
7. Once pAction is running check the changes and see that your code was sadly not pulled in.
8. Optionally re-run, it will work the second time as the first run did trigger a material check, just not in time.

Expected Results

A pipeline triggered by any source, including another pipeline, should wait for all materials to be checked before sending a job to an agent. In the server log that should look similar to this manual run where the MaterialUpdateListener pulls in modifications before the BuildAssignmentService assigns the job.

Actual Results

The pipeline BuildAssignmentService runs before the MaterialUpdateListener pulls in modifications causing this run to be behind.
Two examples, first is dev where the entire (very short) pipeline finished before the material check was complete, the second in prod where just the first stage had started before the material check.

Possible Fix

I've not looked at the source, but if this is reproducible by others, my assumption would be that a piece of code is initiating a material check, but not waiting for it. If thats the case we just need to make that check blocking so that other code does not proceed until the check is complete.

Logs

Let me know if anything outside of images provided would be useful.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days.
If you can still reproduce this error on the master branch using local development environment or on the latest GoCD Release, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.

[stale]

This issue has been automatically marked as stale because it has not had activity in the last 90 days.
If you can still reproduce this error on the master branch using local development environment or on the latest GoCD Release, please reply with all of the information you have about it in order to keep the issue open.
Thank you for all your contributions.