New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Pipeline Activity/Stage Details not working when PluggableSCM material has no modifications for revisions #12718
Comments
The original ticket was too long and couldn't fit the environment section. The thread information was removed because it's too big. Let me know if you need that section. Environment
|
When manually specifying a test from stage details that isn't available in the Pipeline Activity page (something past page 1), this error occurs.
|
Thx for the report. For the pipeline(s) whose pipeline activity page is broken can you check what types of materials they have/had in them? The error is when returning some details related to plugin provided source materials/SCMs, it seems to expect to find at least one modification but there are none (for some reason). I note you have a couple of custom source control plugins - stash.pr and artifactory-scm hence trying to figure out if it relates to a specific plugin. |
Thank you for getting back to me.
There's only a single BitBucket git repo. We're having another issue where BitBucket pull requests aren't building but that's a separate issue.
…On Apr 25, 2024 at 9:30 PM -0400, Chad Wilson ***@***.***>, wrote:
Thx for the report.
For the pipeline(s) whose pipeline activity page is broken can you check what types of materials they have/had in them?
The error is when returning some details related to plugin provided source materials/SCMs, it seems to expect to find at least one modification but there are none (for some reason).
I note you have a couple of custom source control plugins - stash.pr and artifactory-scm hence trying to figure out if it relates to a specific plugin.
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
According to |
What I am trying to figure out where is whether the plugin has a bug; or something that changed along the way that exposed an earlier bug. This type of fundamental invariant "all revisions must have at least one change within them" I don't think has changed or been touched in a very long time, so it's a bit surprising to see this happening, or something happened to your underlying database along the way. (e.g some kind of unsafe maintenance or deletion of stuff from some tables) |
It's using the Stash material type. During the upgrade, we also upgraded to the latest version available of the Stash plugin. If a database migration failed, would I see something in the logs? |
Probably it would log something yes. However there have not been any changes to the DB schema between the two versions you mentioned; nor any data migration scripts run. Are you using the in-memory H2 Database or something external like Postgres? Is there anything special about your upgrade procedure? Did anyone decide to do some magic maintenance on the If you did the upgrade recently and loss of history since then is not a big deal, you might want to consider re-doing it on a backup of your database from that point; or perhaps trying intermediate versions. Or look at the content/size of the I'm afraid it's a bit difficult to help, as I've not seen such a thing happen before as your data seems to be violating a core invariant which has been in GoCD for at least 10 years (as long as it's open source, and probably long). Additionally, that plugin is an only-somewhat-maintained third-party-plugin that I have never used and am not particularly familiar with. Having said that, I can't see what the interaction could be between that plugin and this problem unless something else has got screwed up with your SCMs during the upgrade that has. |
We're using Postgres as the database. For the upgrade, I went through each of these versions first since I didn't know if there were migrations needed.
I downgraded |
That doesn't make a huge amount of sense to me as that.plugin just reports the status of builds back to Bitbucket PRs, not create the pipelines on branches for them to do anything?But maybe I don't understand what you mean. Regardless, not much to be done here without looking at plugin logs and seeing what the problem is/was. |
The other issue we were having was builds not automatically running when changes were made. For some reason now that seems to be working after the plugin downgrade. The go-server logs are filling up so fast they're rotating every minute so I don't have the logs from before the restart. The logs are filling up with this:
The errors in the
I have no idea what's different now than before. I hope the builds continue to work although there are a lot of exceptions. If we had to downgrade back to version |
I don't see why downloading the Builds may start working over time as the database collects modifications for sufficient recent changes in repositories. But if you tried to re-run an old build, or do a comparison over a longer period of time, you are likely to have issues.
If these errors are the reason builds are/were not running I suspect this is all the same root problem. The Your database has become corrupted; or the server is unable to find things that should be there in your database. I suspect there is no going forward from here unless you can figure out what happened to your data in Postgres, determine there is a Postgres-side problem and go back to an earlier snapshot/backup. Or wipe the history for all your pipelines completely in the DB so the DB state is at least consistent. (most people would not want to do this)
This is likely something completely different, or may an issue you had before. This just means the
I think you will likely have ongoing problems given the state.
Should be theoretically possible, yes, but that's going back 3 years. I'd be very surprised if you don't have exactly the same errors/problems running Definitely review the changelogs to see if there is anything that looks like it might not go backwards very cleanly: https://www.gocd.org/releases/ IMHO it would be better to
|
Ok, thank you. I figured it was too good to be true. I'll go back to a backup of the old database and retry the upgrade to see if I can uncover something. I really appreciate your help.
…On Apr 27, 2024 at 1:09 AM -0400, Chad Wilson ***@***.***>, wrote:
> The other issue we were having was builds not automatically running when changes were made. For some reason now that seems to be working after the plugin downgrade.
I don't see why downloading the stash-pr-status plugin would have anything to do with this. Note that this plugin is part of https://github.com/gocd-contrib/gocd-build-status-notifier which is a notification plugin; it doesn't deal with builds directly, just notifies external systems when GoCD events occur. It's different to stash.pr which is part of https://github.com/ashwanthkumar/gocd-build-github-pull-requests.
Builds may start working over time as the database collects modifications for sufficient recent changes in repositories. But if you tried to re-run an old build, or do a comparison over a longer period of time, you are likely to have issues.
> The go-server logs are filling up so fast they're rotating every minute so I don't have the logs from before the restart. The logs are filling up with this:
> 2024-04-26 19:13:12,228 ERROR ***@***.*** for ScheduleCheckListener] BuildCauseProducerService:220 - Error while scheduling pipeline: ApplicationPullRequestFortifyScan
> java.lang.RuntimeException: Could not find modification 280978 in [Material: ***@***.***[additionalData={"BRANCH_TO_REVISION_MAP":"{\"920\":\"b05779b9b5d93fe7c3711ea40b29b6aa4b9c6605\",\"801\":\"e94d382290547ab5b1cfa8f24a45b4c3ef15a286\",\"923\":\"6ea49fbd15aaa1a7c555eb68f096b725e6f97fa4\",\"803\":\"310342e803bc59f208bfbc721cdad4b0e88d6140\",\"926\":\"234efff44d0e4140d3645121cecf11b78dd55b61\",\"936\":\"1c45e0440a65659c6caa4bfd4cdf9d8c8db41832\",\"704\":\"4a7ac324d44c5784bc4551ababc773e9d94bf055\",.......
If these errors are the reason builds are/were not running I suspect this is all the same root problem. The MODIFICATIONS data for a set of revisions is critical to GoCD's evaluation of aspects of the "fan In" algorithm (don't ask me why, it's just what the code does and what I have been told when evaluating trying to drop this data).
Your database has become corrupted; or the server is unable to find things that should be there in your database.
I suspect there is no going forward from here unless you can figure out what happened to your data in Postgres, determine there is a Postgres-side problem and go back to an earlier snapshot/backup. Or wipe the history for all your pipelines completely in the DB so the DB state is at least consistent. (most people would not want to do this)
> The errors in the plugin-stash.pr.log log seem to remain after the downgrade.
> 2024-04-26 18:59:04,062 WARN ***@***.*** for MaterialUpdateListener] GitHubPRBuildPlugin:102 - get latest revision:
> java.lang.RuntimeException: Exception (Process exited with an error: 128 (Exit value: 128)) Occurred: [git, clone, --branch=master, ***@***.***/scm/gtw/application-route.git, /var/lib/go-server/pipelines/flyweight/fb5a90b0-1b00-4c70-950f-e61e08cd2612] - null
> at com.tw.go.plugin.cmd.Console.runOrBomb(Console.java:33)
> at com.tw.go.plugin.git.GitCmdHelper.runAndGetOutput(GitCmdHelper.java:397)
> at com.tw.go.plugin.git.GitCmdHelper.cloneRepository(GitCmdHelper.java:55)
> at com.tw.go.plugin.GitHelper.cloneOrFetch(GitHelper.java:38)
> at in.ashwanthkumar.gocd.github.GitHubPRBuildPlugin.handleGetLatestRevision(GitHubPRBuildPlugin.java:218)
> at in.ashwanthkumar.gocd.github.GitHubPRBuildPlugin.handle(GitHubPRBuildPlugin.java:122)
> at com.thoughtworks.go.plugin.infra.DefaultPluginManager.lambda$submitTo$0(DefaultPluginManager.java:134)
> at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.executeActionOnTheService(FelixGoPluginOSGiFramework.java:205)
> at com.thoughtworks.go.plugin.infra.FelixGoPluginOSGiFramework.doOn(FelixGoPluginOSGiFramework.java:164)
> at com.thoughtworks.go.plugin.infra.DefaultPluginManager.submitTo(DefaultPluginManager.java:131)
> at com.thoughtworks.go.plugin.access.PluginRequestHelper.submitRequest(PluginRequestHelper.java:49)
> at com.thoughtworks.go.plugin.access.scm.SCMExtension.getLatestRevision(SCMExtension.java:108)
> at com.thoughtworks.go.server.service.materials.PluggableSCMMaterialPoller.latestModification(PluggableSCMMaterialPoller.java:59)
> at com.thoughtworks.go.server.service.materials.PluggableSCMMaterialPoller.latestModification(PluggableSCMMaterialPoller.java:44)
> at com.thoughtworks.go.server.service.MaterialService.latestModification(MaterialService.java:127)
> at com.thoughtworks.go.server.materials.LegacyMaterialChecker.findLatestModification(LegacyMaterialChecker.java:50)
> at com.thoughtworks.go.server.materials.ScmMaterialUpdater.insertLatestOrNewModifications(ScmMaterialUpdater.java:55)
> at com.thoughtworks.go.server.materials.ScmMaterialUpdater.addNewMaterialWithModifications(ScmMaterialUpdater.java:70)
> at com.thoughtworks.go.server.materials.PluggableSCMMaterialUpdater.addNewMaterialWithModifications(PluggableSCMMaterialUpdater.java:60)
> at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.addNewMaterialWithModifications(MaterialDatabaseUpdater.java:178)
> at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.initializeMaterialWithLatestRevision(MaterialDatabaseUpdater.java:136)
> at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater$1.doInTransaction(MaterialDatabaseUpdater.java:95)
> at com.thoughtworks.go.server.transaction.TransactionCallback.doWithExceptionHandling(TransactionCallback.java:23)
> at com.thoughtworks.go.server.transaction.TransactionTemplate.lambda$executeWithExceptionHandling$1(TransactionTemplate.java:43)
> at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:133)
> at com.thoughtworks.go.server.transaction.TransactionTemplate.executeWithExceptionHandling(TransactionTemplate.java:40)
> at com.thoughtworks.go.server.materials.MaterialDatabaseUpdater.updateMaterial(MaterialDatabaseUpdater.java:92)
> at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:64)
> at com.thoughtworks.go.server.materials.MaterialUpdateListener.onMessage(MaterialUpdateListener.java:32)
> at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.runImpl(JMSMessageListenerAdapter.java:83)
> at com.thoughtworks.go.server.messaging.activemq.JMSMessageListenerAdapter.run(JMSMessageListenerAdapter.java:63)
> at java.base/java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.commons.exec.ExecuteException: Process exited with an error: 128 (Exit value: 128)
> at org.apache.commons.exec.DefaultExecutor.executeInternal(DefaultExecutor.java:404)
> at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:166)
> at org.apache.commons.exec.DefaultExecutor.execute(DefaultExecutor.java:153)
> at com.tw.go.plugin.cmd.Console.runOrBomb(Console.java:25)
> ... 31 common frames omitted
This is likely something completely different, or may an issue you had before. This just means the git subprocess exited with an error when trying to clone from your bitbucket. The plugin would need to log enough detail for you to debug, but does not seem to do so. Note that this is the third-party stash.pr plugin.
> I have no idea what's different now than before. I hope the builds continue to work although there are a lot of exceptions.
I think you will likely have ongoing problems given the state.
> If we had to downgrade back to version 21.1.0, would that be possible since there are no database migrations?
Should be theoretically possible, yes, but that's going back 3 years. I'd be very surprised if you don't have exactly the same errors/problems running 21.1.0 on the current state of your database though.
Definitely review the changelogs to see if there is anything that looks like it might not go backwards very cleanly: https://www.gocd.org/releases/
IMHO it would be better to
• look at a DB backup/snapshot before you upgraded and look at the amount of data/rows in the tables compared to your current DB. If there are big discrepancies (especially in MODIFICATIONS or PIPELINEMATERIALREVISIONS tables, that might be a clue)
• go back to 21.1.0 with the DB as it was then from a backup. Make sure it is clean/normal. Clean up unrelated errors so you are in a clean state.
• perform upgrade procedure again, but re-check something history-related you know is broken right now at each upgrade increment
—
Reply to this email directly, view it on GitHub, or unsubscribe.
You are receiving this because you authored the thread.Message ID: ***@***.***>
|
Small update. This seems to be happening only to those pipelines that are defined externally from config.xml in pipelines as code. The downgrading of the plugins did not help. It seems that the restart is the thing that makes it temporarily work for a few days before it breaks again. |
If any plugin "downgrade" has a chance of helping, it'd be downgrading Having said this, if the expected data is missing in the database, there's probably no way to recreate it/fix it universally. Perhaps this is something that is reliably happening every now and then due to the pattern of changes on those branches, and how the plugin interacts with them/Bitbucket. If you're able to recreate this on an otherwise "clean" DB (i.e at a starting point all At first glance I don't see how that is possible as there are foreign key constraints on the |
Issue Type
Summary
After upgrade from 21.1.0 to 23.5.0, we're seeing issues browsing Pipeline Activity history. When going past page 1, this error appears at the top of the page. We cannot see or click on
There was an unknown error performing the operation. Possible reason (Server Error)
Basic environment details
23.5.0
17.0.9
RHEL 9.3 Linux 5.14.0-362.18.1.el9_3.x86_64
Arc 1.40.0 (49176) Chromium Engine Version 124.0.6367.79
Additional Environment Details
Steps to Reproduce
Expected Results
Pipeline Activity should work normally.
Actual Results
Page 1 is the only page that is displayed until page 289 where data from 2 years ago is displayed.
Log snippets
Here's the exception from the logs.
The text was updated successfully, but these errors were encountered: