pipeline health POC #59

edmundmiller · 2024-07-20T23:10:12Z

~~Closes~~ A start on #5

I expect it to go through some iteration before actually using it to manage the pipelines, and even then a slow roll-out.

Just looking for anything that anyone absolutely hates, or settings that we can catch that are wrong.

pulumi/github/repos/README.md

pulumi/github/repos/core_repos.yml

ewels · 2024-07-21T05:27:51Z

pulumi/github/repos/pipelines.yml

@@ -0,0 +1,99 @@
+- airrflow


As above - see https://nf-co.re/pipelines.json

Why is this resolved, how come we can't use the pipelines.json file? I don't like the idea of trying to keep these up to date.

By the way we also have https://nf-co.re/pipeline_names.json which is much lighter.

Small iterations 😉

I'll add it to download, but I don't want to start "terraforming" the whole GitHub org because of the scope of this PR.

ewels · 2024-07-21T05:32:14Z

Looking very promising!

The old website PHP code for this is here: https://github.com/nf-core/website/blob/old-site/public_html/pipeline_health.php

That contains all of the logic for stuff like the repo website, the set of minimum keywords and the types of things to do (eg. stripping old bad GHA job names). See the functions called fix_* (mostly).

ewels · 2024-07-21T05:35:12Z

Also this is the new pipeline health page in Astro, if it helps: https://github.com/nf-core/website/blob/main/sites/pipelines/src/pages/pipeline_health.astro

That reports on a lot of this stuff but doesn't set anything.

edmundmiller · 2024-07-21T16:28:16Z

Also this is the new pipeline health page in Astro, if it helps: https://github.com/nf-core/website/blob/main/sites/pipelines/src/pages/pipeline_health.astro

That reports on a lot of this stuff but doesn't set anything.

Ah, sweet!

edmundmiller · 2024-07-22T03:54:39Z

@ewels Ignore the mypy errors will fix that in post. I think this is ready to review and 🚀

pulumi/github/repos/pipelines/testpipeline.py

mashehu

Some thoughts

.github/workflows/repos.yml

docs/1password.md

mashehu · 2024-07-22T07:53:52Z

pulumi/github/repos/Pulumi.dev.yaml

@@ -0,0 +1,5 @@
+config:
+  github:owner: nf-core-tf
+  # https://start.1password.com/open/i?a=O5GICFDKPNABLLVGMKBL5JWDWA&v=rdfcz6oy6qxxrc4clu467a7dmm&i=4ajrv44kc5lcbboa37fr5oydla&h=nf-core.1password.eu


this throws an error for me. is that maybe for your personal account?

Updated the link, you also we're in the Dev vault. Had to make a vault that was specifically accessible to the service accounts, and I didn't want to give them access to everything.

pulumi/github/repos/Pulumi.dev.yaml

pulumi/github/repos/Pulumi.yaml

pulumi/github/repos/README.md

ewels · 2025-02-03T13:47:01Z

Need to check that we don't clobber certain fields, such as repository descriptions if already written.

Similar for repo keywords: should have a minimal set but not clobber custom ones.

This will vary across fields.

mashehu

needs some cleanup

.github/workflows/repos.yml

mashehu · 2025-02-03T14:52:32Z

pulumi/github/repos/core/modules.py

I don't think we need to handle core repos with pulumi

I think not with the "loop" of all of the pipelines. Just as their own individual settings.

I think it can't hurt to import the settings.

pulumi/github/repos/loop_example.py

pulumi/github/repos/pipelines/testpipeline.py

mashehu · 2025-02-03T16:30:32Z

pulumi/github/repos/pipelines/testpipeline.py

+        ),
+    ),
+    visibility="public",
+    topics=TOPICS,  # 'repo_keywords' => 'Minimum keywords set',


Suggested change

topics=TOPICS, # 'repo_keywords' => 'Minimum keywords set',

topics= repo.topics ?? TOPICS, # 'repo_keywords' => 'Minimum keywords set',

could we add missing topics without deleting the existing ones?

yep, that's probably the better approach. the problem is that repo.topics is not just an array, but a pulumi Output object...

pulumi/github/repos/requirements.txt

mirpedrol · 2025-02-04T09:35:14Z

.github/workflows/repos.yml

+  push:
+    branches:
+      - main
+    paths:
+      - "pulumi/github/repos/**/*"
+  pull_request:
+    branches:
+      - main
+    paths:
+      - "pulumi/github/repos/**/*"


Just to understand. This will run on PRs to ops, right? Is this action updating the pipeline health? And are these PRs on ops related to pipeline health?

pulumi/github/repos/pipelines/testpipeline.py

mirpedrol · 2025-02-04T09:44:51Z

pulumi/github/repos/pipelines/testpipeline.py

+        ),
+    ),
+    visibility="public",
+    topics=TOPICS,  # 'repo_keywords' => 'Minimum keywords set',


could we add missing topics without deleting the existing ones?

pulumi/github/repos/pipelines/testpipeline.py

…dules

1. Importing them all by hand, some code duplication and effort, but probably the least likely to blow up 2. Looping through them all We can also start with 1, and then move to 2 once everything is captured in the Pulumi state with 1(which seems like the sane option)

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_default_testpipeline testpipeline:1220601

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_dev_testpipeline testpipeline:1220600

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_TEMPLATE_testpipeline testpipeline:1220597

Co-authored-by: Matthias Hörtenhuber <[email protected]>

Co-authored-by: mashehu <[email protected]>

Co-authored-by: mashehu <[email protected]> Co-authored-by: mirpedrol <[email protected]>

- Added pulumi-onepassword dependency to manage GitHub tokens securely. - Updated Pulumi configuration to include 1Password account details. - Modified testpipeline.py to fetch GitHub token from 1Password and configure GitHub provider accordingly.

mashehu · 2025-06-04T10:31:10Z

@nf-core-bot fix linting

edmundmiller self-assigned this Jul 20, 2024

edmundmiller force-pushed the pipeline_health branch from 28320ba to bd199e9 Compare July 21, 2024 00:26

ewels reviewed Jul 21, 2024

View reviewed changes

pulumi/github/repos/README.md Outdated Show resolved Hide resolved

ewels reviewed Jul 21, 2024

View reviewed changes

pulumi/github/repos/core_repos.yml Show resolved Hide resolved

ewels reviewed Jul 21, 2024

View reviewed changes

edmundmiller changed the title ~~pipeline health~~ pipeline health POC Jul 22, 2024

edmundmiller marked this pull request as ready for review July 22, 2024 03:16

edmundmiller requested a review from a team as a code owner July 22, 2024 03:16

ewels reviewed Jul 22, 2024

View reviewed changes

pulumi/github/repos/pipelines/testpipeline.py Outdated Show resolved Hide resolved

mashehu reviewed Jul 22, 2024

View reviewed changes

edmundmiller mentioned this pull request Jul 22, 2024

Pulumi ESC or GitHub Secrets #60

Open

maxulysse approved these changes Jan 31, 2025

View reviewed changes

mashehu reviewed Feb 3, 2025

View reviewed changes

mirpedrol reviewed Feb 4, 2025

View reviewed changes

edmundmiller added 11 commits June 3, 2025 17:32

docs: Get the project started

d10fd30

chore: Copy over code from teams

d4dc7d4

chore: re-encrypt GitHub token

2efa2be

chore: pulumi import github:index/repository:Repository nf-core-tf mo…

2c2deb6

…dules

docs: Write up some plans

66c4ef4

Add missing repos

7d7df5d

chore: Sort lines

4734c3f

chore: Add pipelines

9654778

Add demo and testpipeline

b07ebfb

Add Old Pipeline Health PHP code

e3f8b97

edmundmiller and others added 18 commits June 3, 2025 17:33

chore: Import Master Branch protection

e96a9ce

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_default_testpipeline testpipeline:1220601

chore: Import dev branch ruleset

df10916

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_dev_testpipeline testpipeline:1220600

chore: Import template branch ruleset for testpipeline

442d925

pulumi import github:index/repositoryRuleset:RepositoryRuleset ruleset_branch_TEMPLATE_testpipeline testpipeline:1220597

chore: Remove duplicate TODO

5bf2407

chore: Match up specs with code

68eedf4

feat: Update dev branch requirements

6104a30

refactor: Move required CI Checks out

5e15568

feat: Add contributors and core permissions

0adf214

refactor: Use NAME variable where ever possible

0a5c862

Remove protect on repo

62c608d

ci: Add template workflow for repos

3df19aa

ci: Remove python-version

4067d72

style: Run pre-commit

46b5d68

Apply suggestions from code review

ec7a222

Co-authored-by: Matthias Hörtenhuber <[email protected]>

chore: Add 1password link

c6df079

docs: Clean up README moving planning to GH issue

7dc11d7

Co-authored-by: mashehu <[email protected]>

refactor: Use Pulumi ESC for dev

1096f8c

docs: Add some nf-core Pulumi starter info

be3d2f2

edmundmiller force-pushed the pipeline_health branch 2 times, most recently from 929d181 to cc5fa7c Compare June 3, 2025 22:54

edmundmiller and others added 4 commits June 3, 2025 17:57

build: Use uv

1db1ddb

chore: Update pulumi-github dependency to version 6.7.2 and remove ruff

ea0511d

fix: Address PR Review comments

df4352c

Co-authored-by: mashehu <[email protected]> Co-authored-by: mirpedrol <[email protected]>

edmundmiller requested review from a team, mashehu and mirpedrol June 3, 2025 23:24

style: Run pre-commit

8ec8496

edmundmiller force-pushed the pipeline_health branch from cc5fa7c to 8ec8496 Compare June 4, 2025 13:22

	topics=TOPICS, # 'repo_keywords' => 'Minimum keywords set',
	topics= repo.topics ?? TOPICS, # 'repo_keywords' => 'Minimum keywords set',

pipeline health POC #59

Are you sure you want to change the base?

pipeline health POC #59

Uh oh!

Conversation

edmundmiller commented Jul 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ewels commented Jul 21, 2024

Uh oh!

ewels commented Jul 21, 2024

Uh oh!

edmundmiller commented Jul 21, 2024

Uh oh!

edmundmiller commented Jul 22, 2024

Uh oh!

Uh oh!

mashehu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ewels commented Feb 3, 2025

Uh oh!

mashehu left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

mashehu commented Jun 4, 2025

Uh oh!

Uh oh!

edmundmiller commented Jul 20, 2024 •

edited

Loading