[Meta] Monorepo and package-based publishing

[thompsongl]

EUI has had the long-term goal to increase modularity, and recent efforts related to styling (#3912; moving away from Sass) have opened the door to beginning the process in earnest.

A rough outline of phasing:

Validation

• Research monorepo infrastructure (e.g., yarn workspaces, lerna, turborepo)
• Establish future build system

Utilities

• Establish utilities and services that can exist in a standalone package
• Theming needs to exist in a standalone fashion

Core & Supplements

• Establish which components compose the EUI core package
• Split non-core components into relevant packages
• @elastic/eui should still exist as an all-inclusive package

[chandlerprall]

Three thoughts to help find potential boundaries for packages:

1. look at usage in downstream projects, are there common groupings?
2. do our EUI PRs create logical groups?
3. how about EUI imports of other components?

[thompsongl]

I did an exploratory spike to expand my understanding of monorepos generally and where the pain points will be for EUI specifically. We will pick up the work again soon, but the gist of the outcome is as follows.

Tooling

Dependency management: yarn workspaces
Build system: TBD; turborepo was positive
Release help: TBD; changesets potentially

Phasing

It is unfeasible to begin splitting up @elastic/eui at this moment. We will need to take a phased approach where in certain cases the lines between packages are blurred and/or crossed. For instance, the docs site (src-docs/) presents a unique challenge in that it relies on files generated specifically for its consumption as part of build scrips in @elastic/eui, as well as importing files not part of @elastic/eui's top-level API.

Proposed tentative migration

Phase 1 - Transition

• Keep src-docs/
• Move src/ to packages/eui/
• Split packages only where necessary
  • proptypes-from-ts-props
  • Considerations for packages related to shared test and build utils (tsconfig, eslintrc)
• Be ok with blurred line between packages/eui/ and src-docs/
  • Jest
  • Cypress
  • Sass compilation

Phase 2 - Illustration

• @elastic/eui-illustration
• apps/docs-eui-illustration
• apps/figma-eui-illustration

Phase 3 - Non-component packages

• eui-theme
• eui-utils
• eui-services
• Considerations for similar groups

Phase 4 - Component packages

• Scope TBD
• Progressively reduce packages/eui until it simply imports and re-exports other EUI packages

Other recommendations

• Move all EUI build output into dist/
  • dist/lib, dist/es, etc.
• Find a scripting method to git mv files to packages to ensure git history is maintained

Outstanding questions & further exploration

Testing

• Jest configuration: per package or run from root?
• How does this affect the eventual move to react-testing-library

Dependencies

• Version resolution between packages: manual or hoist to root?
  • babel, @types/*, typescript, moment presented version conflict problems
• Investigate nohoist for packages

Publishing releases

• Investigate changesets
• Further investigate turborepo and alternatives
• CI
• Build and release scripts
  • Does moving to changesets unblock the Jenkins release job??

[yakhinvadim]

Just wanted to mention that we at Search UI are facing the same questions as in "Publishing releases" section.

We also found https://github.com/elastic/apm-agent-rum-js that seemingly had the same challenges and added some scripts for publishing: https://github.com/elastic/apm-agent-rum-js/tree/main/scripts. We plan to chat with them soon, let me know if you'd like to participate.

[thompsongl]

@yakhinvadim Yes, I'd like be involved in that chat!
As we progress with this work I'd also like to talk to the Search UI team about what (if any) EUI packages would be of use. Any input on drawing lines in Phase 3/4 would be helpful.

[yakhinvadim]

@thompsongl Awesome, we'll invite you when we set it up (likely not in the next 2 weeks).

As for which EUI packages to release first: our priorities have shifted, and we paused the project that needed to use EUI. We plan to return to it in the future, but I don't think it'll happen in the next 6 months, maybe a year.

But if you have no other inputs, we were planning to build something like this (early design):

[thompsongl]

Sounds good, @yakhinvadim! Thank you