Feature request: allow workflow configuration in sub-folders

[solarmosaic-kflorence]

Copying this feature request over from https://github.community/t/feature-request-allow-workflow-configuration-in-sub-folders/16776

We use a monorepo setup and currently everything is organized into sub-folders, so for example all services are located in the “services” folder. Currently workflow configuration is the only code which relates to a service that is not in the applicable “services” folder (e.g. “services/foo-service”). In order to maintain clean code separation and ownership, it would be nice if we could place our workflow configuration for a specific service in it’s appropriate folder. The most straightforward solution would seem to be matching on any .github folder in the repo, not just the root one. This would allow us to place workflows in the “services/foo-service/.github/workflows” folder, for example.

[MunchyYDL]

I was searching for a way to organise many workflows in the UI, like by category or by team for a mono-repo. And I was thinking about just being able to create subfolders in the ./github/workflow/ folder, and that each folder would appear as a collapsable section in the UI.

But this proposal would be even better, at least when it comes to separation of files, but I'm not sure how it would be presented in the UI?

[SiavoshZarrasvand]

@MunchyYDL I actually prefer your suggestion.

[amitfmg]

I like the suggestion about collapsible folders on the UI.

[chaddyc]

@MunchyYDL, keen to see if Github will implement/develop this feature.

[garrmark]

I'd like to add my support for the folders under .githuib/workflow that would be displayed as collapsable folders in the UI

[eliezercazares]

This is enough for my needs. I have setup Automated Tests in the Workflows, using the Runners. And it's so many suites and tests that being able to organize my test suites would be cool. Because I also have workflows that do Slack notifications and are reused. So a common/ folder would be good for those.

[zachsa]

I was trying to do:

.github/
  - workflows/
    - dir_1
      - a.yml
      - b.yml
      - etc
    - dir_2
      - a.yml
      - b.yml
      - etc

With the hope that I would see a list of actions as

dir_1_a
dir_1_b
dir_1_c
dir_2_a
dir_2_b
dir_2_c

or something similar. Since the action names are configured in the yml files it would make sense to namespace actions according to the directory they are in. For my use case, aside from namespacing actions via a directory name, nothing would change (though since I use only a small subset of actions functionality I'm not sure if there are other considerations).

It would be nice if it was possible to put .github folders anywhere.

In any case, thank you for the actions tooling! It's pretty amazing.

[krtschmr]

seems quite tricky to scan all folders for a .github folder. subfolders in the .github would be sufficient i believe

[san7890]

It would also be rather nice if there could be a way to run a workflow nested in a submodule, and then leverage that same submoduled workflow to work on the parent repository. This is definitely a much bigger ask than what the original discussion was most likely tailored for, but it would certainly come in use for a repository I currently maintain.

[alfreddagenais]

This will help a lot my team for a mono-repo! I hope to saw this feature soon

[jsoref]

You can do a lot of this by using a services/foo-service/action.yml and then put uses: ./services/foo-service/ into your .github/workflows/foo-service.yml.

It isn't perfect, but, you can do it.

Or, you could just use CODEOWNERS and have that map ownership of .github/workflows/foo-service-* to @foo-service.

Forcing GitHub to recurse through the entire tree, or even a large slice of a tree whenever it receives any event is almost certainly a non-starter. I appreciate the problem, but don't think that asking them to do this is likely to go anywhere.

What you can do is have your workflow load its configuration from a file at a path in your chosen directory. Many public workflows do this, although typically that directory is .github/actions/{action-short-name}/

[jsoref]

https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#example-using-an-action-in-the-same-repository-as-the-workflow

[tn3rb]

Forcing GitHub to recurse through the entire tree, or even a large slice of a tree whenever it receives any event is almost certainly a non-starter.

what do you think is going to happen if you have a bunch of files like

.github/workflows/foo-service.yml
.github/workflows/bar-service.yml
.github/workflows/baz-service.yml

that all implement use to point to additional actions in subfolders like:

# in .github/workflows/foo-service.yml
uses: ./services/foo-service/ 

# in .github/workflows/bar-service.yml
uses: ./services/bar-service/ 

# in .github/workflows/baz-service.yml
uses: ./services/baz-service/ 

GitHub is going to load all of those actions in the root of .github/workflows and THEN have to ALSO go into those subfolders anyways.

Sound like even more work than just recursing through a folder structure like:

.github
    - workflows
        - foo
            - service.yml
        - bar
            - service.yml
        - baz
            - service.yml

which is not an unreasonable request in this day and age

[jsoref]

That isn't how the GitHub Action Runtime works.

When it gets an event, it only needs to enumerate objects in a single folder (on at most two branches -- default and the current one).

For each object, it then decides if the workflow is enabled, and then if the workflow applies.

If it's enabled and it applies, then it dispatches it to a mechanism that will send things to runners which will then digest the uses lines. It's done much later in the pipeline, and it isn't done by enumerating all objects in the repository, it looks for very specific paths ./services/baz-service/ {action.yaml | action.yml }

[jeffrey-aguilera]

find .github/ -type file -name *.yml is incredibly fast. There is really no excuse for preventing users from organizing their workflows.

[sparr]

@jeffrey-aguilera Not only do some repos contain yaml files that aren't github workflows, some repos contain valid github workflows as the content of the repo, not as workflows to be run on that repo.

[sam0x17]

A good workaround is symbolic links in your root directory .github/workflows folder

[krtschmr]

How can i achieve this?

ln -s ?

[raihle]

I'm not sure that works at all, but I know it doesn't work when adding, removing, or renaming workflow files

[socketbox]

This does not work

[sam0x17]

yes it does not work unfortunately. Wish it did

[danielfigueiredo]

I've tried using symbolic links and GHA will raise errors when trying to parse workflow files

[zomgbre]

This feature would be awesome to have for when using both re-usable workflows and the "regular" workflow. Right now, I'm naming things with emojis in front of them to do this. 😂 It gets a little ⚡ if it's a re-usable.

[zomgbre]

All I did was paste in the emoji in vscode in front of the name :P it was a little silly:

name: '⚡ ci'
on:
  workflow_call:
    inputs:
      node-version:
        description: Node version
        type: string
        required: true
      working-directory:
        description: Working directory to use
        type: string
        default: './'

jobs:
  test:
    name: Test
    runs-on: ubuntu-latest
    defaults:
      run:
        working-directory: ${{ inputs.working-directory }}
    steps:
      - name: Checkout repository
        uses: actions/checkout@v3

      - name: Use nodejs ${{ inputs.node-version }}
        uses: actions/setup-node@v3
        with:
          node-version: ${{ inputs.node-version }}
          registry-url: 'https://npm.pkg.github.com'

      - name: Install with package-lock
        run: npm ci
        env:
          NODE_AUTH_TOKEN: ${{ secrets.NODE_AUTH_TOKEN }}

      - name: Compile typescript
        run: npm run build
      - name: Tests
        run: npm run test

I have since abstracted these workflows into a separate repository and now referencing them like so across several similar projects:

name: ci
on:
  pull_request:
    branches:
      - main
      - release/**

permissions:
  contents: read

jobs:
  ci:
    uses: orgname/repo-name/.github/workflows/ci.yml@main
    secrets: inherit
    with:
      node-version: 18

[granthorner]

thanks @zomgbre I guessing there's no way to actually display the emoji directly within the github actions list in a way to help group them?

[jsoref]

You can certainly use emoji in the name: field w/in workflows...
https://github.com/jsoref/sample-nested-workflows/actions

[zomgbre]

You can copy the emojis from here: ❤️
https://gist.github.com/rxaviers/7360908

It's gotta be the actual emoji and not the markdown :heart: Hehe, it seems counter-intuitive a bit because you'd think it would render with the colons.

[granthorner]

Ah that's amazing, that's where I was going wrong....much appreciated for your help @zomgbre

[sam0x17]

Apparently it doesn’t work

…

On Tue, Aug 2, 2022 at 6:32 AM Tim Kretschmer ***@***.***> wrote: How can i achieve this? ln -s ? — Reply to this email directly, view it on GitHub <#18055 (reply in thread)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAOE4LLF6VBUR4F74GVLKSLVXD2KHANCNFSM5YELZWBA> . You are receiving this because you commented.Message ID: ***@***.***>

[ErikBjare]

You mean the symlink suggestion?

[rodrigo-georgian]

Instead of recursing through the whole repo, why not provide a keyword in the root .github/workflows that can be used to point to the "sub-root" of the directory you want to treat as it's own repo (for actions)?

Something like this:

In /.github/workflows/package_1.yaml:

redirects: './package_1'

In /package_1/.github/workflows/package_1.yaml:

# regular github actions config, as if this were the root package

Benefits:

• Almost complete code separability. (Only needs to add that one line / file to the root)
• No need to recurse anything.

[rawnly]

Couldn't this can be archived via the uses keyword as mentioned before?

[Updatephoneme]

Copying this feature request over from https://github.community/t/feature-request-allow-workflow-configuration-in-sub-folders/16776

We use a monorepo setup and currently everything is organized into sub-folders, so for example all services are located in the “services” folder. Currently workflow configuration is the only code which relates to a service that is not in the applicable “services” folder (e.g. “services/foo-service”). In order to maintain clean code separation and ownership, it would be nice if we could place our workflow configuration for a specific service in it’s appropriate folder. The most straightforward solution would seem to be matching on any .github folder in the repo, not just the root one. This would allow us to place workflows in the “services/foo-service/.github/workflows” folder, for example.

[kr99]

This would be a nice feature for our template projects, to make certain workflows optional and easy to tear out (with related scripts). eg. "Not using the event queue system? tear out the workflow and related integration testing scripts by pulling ONE folder..."

Suggested workaround: strictly enforced naming conventions on the workflow files. e.g. "event_queue.integration_test.stage.yml" mirroring a folder src/event_queue or something (I don't like my example a ton, but it illustrates the point)

[tn3rb]

also running a monorepo and having workflow subfolders would be great as we have a lot of granular separation going on.

this would be ideal for us:

.github
    - workflows
        - foo
            - deploy.yml
            - e2e-test.yml
            - unit-test.yml
        - bar
            - deploy.yml
            - e2e-test.yml
            - unit-test.yml
        - baz
            - deploy.yml
            - e2e-test.yml
            - unit-test.yml

[rawnly]

I've been following this thread for a while, if you're interest I made a simple cli in rust that can help with workflows context separation. Is still a WIP, nothing fancy. Hope this helps someone ✌️

https://github.com/rawnly/hawk

[rodrigo-georgian]

Pragmatic solution, thanks @rawnly

[rawnly]

If anyone willing to use it, I just updated and released hawk, now is "generally available" ✌️

[dsdavis4]

Has there been any progress made on this? Being able to define a .github directory within subfolders would be ideal, however what @tn3rb suggested would also be great.

[adikari]

We are using monorepo setup and our .github/worflows is getting out of control. Before Actions, we were using Buildkite and it allowed to add workflows in specific folders for services as suggested by the OP. The root workflow is getting too big and unmanageable and we are considering moving back to Buildkite just for this reason :(

[bite-dayle]

Yeah would love to see this feature added

[joshua-temple]

PLEASE.. this feels like such an obvious miss or the GHA team was strongly under the assumption any given repo would have <5 workflows.

~20 some odd services in one repo that comes to mind, with four environments across five teams.. surprisingly is very fluid, except for the GHA organization.

[MuresanVladMihail]

Still not working.

[jhelmink]

Adjacent topic, but folders for Secrets would also be appreciated...

[apota]

create generic workflows that are heavily parameterized so as to compensate for having multiple workflows for each variation. This will reduce the clutter.

[garrmark]

are you referring to starter workflows or something else?

https://docs.github.com/en/actions/using-workflows/creating-starter-workflows-for-your-organization

[jsoref]

My guess is they are not referring to starter workflows.

It could be a model with fancy matrix definitions to start lots of jobs. (There are other ways too.)

The model would probably be based on unifying workflows that have the same event triggers.

[abhilashindulkar]

Need this feature. It would help us to organize or distinguish the workflows & templates better. Hope to get the resolution for the error soon - invalid value workflow reference: references to workflows must be rooted in .github/workflows

[JohnMops]

Aboslutely needed , migrating from GitLab to GitHub is leaving our mono repos with reusable code without any folder structure and a complete mess

[RevoluPowered]

Working on converting to a mono-repository for a large game and indeed, without this it is really disadvantageous to not be able to scan subfolders for a .github/ yaml file.

Why is this important for a single repo:

• you have BIG_PROJECT_A/ (engine)
• you have the GAME_PROJECT_B/ (game)
• you have SUPPORT_SERVER (backend)

You want to run a build but only if a specific folder is changed.

Why we use a mono repo:

• You can now git bisect your entire project without writing absurd logic.
• One version of the truth, not 10 projects with different combinations unlocking new potential bugs.

When I modify BIG_PROJECT_A/ I would like to run CICD scripts contextually for that specific folder, like version bump, a web app is an example of where this is important when it must speak to other systems.

Currently what I will have to hack together:

• a trigger that listens for a folder modification
• a bunch of logic which is unrelated to each other in the same folder you're talking 20 cicd scripts or more in the single .github folder, which now all need modified to use the paths for subfolders.
• A non elegant non optimal solution.

What would be ideal:

• BIG_PROJECT_A contains a yml file, which contextually runs relative to that folder path as it's working directory.
• YML file requires ZERO changes to work in a mono repository. copy paste and done.
• When executed on the github runner it will check the working directory where the .github/ directory is located, this will then inherit the parent directory's path for the cicd scripts

[jsoref]

You can do most of what you want with a bunch of foo/action.yml files and have most of your logic in the action files instead of in workflows.

... Yes, you still need to define a bunch of jobs in your workflows, and yes, you'll need an action that decides which jobs in which workflows to run based on paths, but, no it isn't the end of the world.

For the latter, you can use: https://github.com/marketplace/actions/verify-changed-files#usage (no, this isn't an endorsement, and yes, you should review third party actions very carefully) or similar.

.github/workflows/build-world.yml:

on:
  push:

jobs:
  big-project-a:
    name: Build big project A
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Verify Changed files
        uses: tj-actions/verify-changed-files@v19
        id: verify-changed-files
        with:
          files: |
            big-project-a/**

      - name: Build big project A
        if: steps.verify-changed-files.outputs.files_changed == 'true'
        uses: ./big-project-a

  game-project-b:
    name: Build game project B
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Verify Changed files
        uses: tj-actions/verify-changed-files@v19
        id: verify-changed-files
        with:
          files: |
            game-project-b/**

      - name: Build game project B
        if: steps.verify-changed-files.outputs.files_changed == 'true'
        uses: ./game-project-b

  support-server:
    name: Build support server
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Verify Changed files
        uses: tj-actions/verify-changed-files@v19
        id: verify-changed-files
        with:
          files: |
            support server/**

      - name: Build support server
        if: steps.verify-changed-files.outputs.files_changed == 'true'
        uses: ./support server

If you're feeling really adventurous, it might be possible to convert workflow files to action files -- alternatively, it's definitely possible to use act to ask it to run things not quite the way github wants.

[AnnaHoege]

@jsoref why not use a path filter instead? If you're watching for file changes in an entire directory anyway, specifying the path could be enough. Then each project gets its own workflow and commonly used workflows can be written generically. Also you'd be getting rid of a third-party-action.

[jsoref]

GitHub doesn't promise that path filters work if too many files change in a single commit/PR. As the described mono-repositories are assumed to have lots of files, they're more likely to trip on this condition.

[AnnaHoege]

Really? That's quite a downside for using path filters. Thanks for the explanation!

[jsoref]

Yep. https://docs.github.com/en/actions/using-workflows/workflow-syntax-for-github-actions#git-diff-comparisons

[alvarotroya]

+1, in large monorepos a better way of organizing workflows is really necessary

[ospikovets]

+1 for this feature! It would be very helpful for better organizing monorepo workflows.

[sourceful-karlson]

+1 for this feature! It would be very helpful for better organizing monorepo workflows x2

[PreK]

+1 Keep things organized!

[triley13]

+1 For organization

[gouravkrosx]

+1 For organization

[aaa-khoa]

+1 for sub folders, we also use a monorepo so we have 20+ workflows