Review the release process

[kuzdogan]

Currently the release process and the script (scripts/release/main.sh) is cumbersome with many steps and really prone to errors. If anything goes wrong, the script breaks and it's really difficult to start over.

Following our call today, I'll be having a look at the release process:

• Fork the repo and try using lerna for everything: versioning, changelogs, pushing tags, doing the release
  • To do the changelogs with lerna we need to start using the conventional changelogs format
  • Pushing tags to remote via lerna might cause the problem again we had, that sometimes the CI does not catch the new tag when everything's pushed all at once
  • Doing the release probably will only for the whole repo and lerna won't be able to replicate our way of releases, a new release of each sub-module. We're fine with this and we can continue from now on with a single Sourcify repo release, as long as we maintain the package versions. Ideally this release should contain all submodule changelogs
• Branch out from staging to a release branch, do everything there, and merge from there to staging and master
  • In this case the tag will be on this branch's commit but the commit hash should remain consistent when merged to master and staging. Double check that.
• Figure out if we can enforce to use merge commits for PRs to master.
• If we still have a (much smaller) release script after this issue, we should add a check in it for changes in the database migrations. It should check between the last release and the current if there were changes in the migrations, and remind us to run the updates on the database.

[manuelwedler]

As part of this, it would also be nice if we could check if it is possible to enforce a certain commit type to a certain branch via GitHub. More specifically, I would like to see that PRs can only be merged to master via a merge commit.

[manuelwedler]

Another addition: If we still have a (much smaller) release script after this issue, we should add a check in it for changes in the database migrations. It should check between the last release and the current if there were changes in the migrations, and remind us to run the updates on the database.

[kuzdogan]

Looking into lerna docs, doing some thought experiments and some practical experiments at github.com/kuzdogan/sourcify here's what I propose below.

While thinking about the release process it's useful to think about two aspects of the release:

1. Deployment to production: This is handled by just merging the staging to master. We'll build the container based on master and deploy on prod.
2. Publishing packages and containers:
   • Packages: Handled by assigning new package.json versions and npm publishing. Either done by lerna or done at CI by looking at published repo tags.
   • Containers: Triggered by publishing tags for services

Prior to both proposals we do one final release with the current script setup and then move forward with one of them:

Automated proposal

1. Create PR from staging to master
2. Run lerna version --conventional-commits --create-release github

• We can run this on a separate branch to be merged into staging, or directly on staging by disabling branch protections
• This does the following:
  • Lerna chooses the versions automatically by looking at the conventional-commits. It shows the expected versions before for user confirmation.
  • Generates changelogs
  • Creates tags
  • Pushes the tags and commits them all at once
  • Creates the GH releases for each package on given tags

3. Merge staging to master

See https://github.com/kuzdogan/sourcify/releases for examples

This proposal relies on using conventional commits.

• The changelog will be auto-generated looking at the commits.
• commits with feat: will be a feature and do a MINOR bump on the packages.
• commits with fix: will be a bug fix and cause a PATCH
• commits with !feat: will be a breaking change and cause a MAJOR bump
• All of the above will be listed under "Breaking Changes", "Features", "Bug fixes" respectively in the changelog.
• commits without these prefixes will cause a PATCH bump but won't be listed in the changelog.

One concern with this setup is it's fully automatic in versioning and at times it might not result in our intended versions. For example if in a commit or a PR named feat:.... that is intended as a feature for the server, if we touch files in bytecode-utils this will bump MINOR both on server and bytecode-utils.

The other problem is lerna batch pushes the git tags, this will resurrect the CircleCI problem of sometimes not detecting the new git tags. This wouldn't be an issue for packages if we publish after versioning locally, but it's a problem for versioned containers built on the CI. Here we can move container builds to Github Jobs to work around this.

Semi-automated alternative

The other alternative is we choose package versions ourselves via lerna (above it was derived from conventional-commits) giving us more control. Here the changelogs will also be more manual.

1. Create a PR from staging to master
2. Run lerna changed --all --long --parseable to see which packages need version bump
3. Write changelogs for each package with a script, only the content is manual
4. Commit changelogs
5. Run lerna version --no-push
   • Just choose versions for each package
   • Lerna creates tags but doesn't push
6. Push the commits
7. Push tags one by one
8. Create github releases by looking at changelogs and tags
   • This can be error prone

Separate question

One question independent from the proposal choice is if we want to maintain sourcify-monorepo as a package to be released or not. Currently it's handled as a package by lerna. I lean towards getting rid of it as it has no practical use.

[marcocastignoli]

I'm in favor of trying the "Automated proposal" for some time. Is it going to be very difficult to rollback to our current release process if we are unsatisfied?

[kuzdogan]

No we can always go back to any way to release, even manual if needed. We are essentially doing the same things

[manuelwedler]

One concern with this setup is it's fully automatic in versioning and at times it might not result in our intended versions. For example if in a commit or a PR named feat:.... that is intended as a feature for the server, if we touch files in bytecode-utils this will bump MINOR both on server and bytecode-utils.

Some questions about this to make sure I understood everything correctly. So, in general, lerna does detect which packages a commit changes, correct? If we only modify files inside server directory, only server receives a version bump?

On the other side, if a commit modifies two packages, would we have the same changelog message for two packages?

I also saw there is a scope feature in conventional commits. Would it help using this for working around this problem?

The other problem is lerna batch pushes the git tags, this will resurrect the CircleCI problem of sometimes not detecting the new git tags. This wouldn't be an issue for packages if we publish after versioning locally, but it's a problem for versioned containers built on the CI. Here we can move container builds to Github Jobs to work around this.

How sure are you about this? When was the last time that this was a problem? If it's long ago I think it's worth to try again if CircleCI still has a problem with it.

---

I generally like the idea of using conventional commits. I only think we then need to be stricter about processes.

Squashed merge commits might be a problem for conventional commits. What if one PR wants to introduce a feature in two packages? Or introduce two changelog entries for the same package? I think we should just move to usual merge commits.

Also, I think we should enforce the merging strategy via GitHub settings. I see some PRs merged with a different strategy than squashing recently.

This is also an opportunity to start having a cleaner commit history inside PRs. We can have a CI check that makes sure our commits follow the conventional commit format. Enforcing to squash any fixup commits inside PRs could also make sense before merging.

[marcocastignoli]

Squashed merge commits might be a problem for conventional commits. What if one PR wants to introduce a feature in two packages? Or introduce two changelog entries for the same package? I think we should just move to usual merge commits.

I think in general, before merging a PR, we can rebase e reorganize the history as we like.

[kuzdogan]

Some questions about this to make sure I understood everything correctly. So, in general, lerna does detect which packages a commit changes, correct? If we only modify files inside server directory, only server receives a version bump?

Yes

On the other side, if a commit modifies two packages, would we have the same changelog message for two packages?

Also yes see https://github.com/kuzdogan/sourcify/releases

I also saw there is a scope feature in conventional commits. Would it help using this for working around this problem?

I don't think scopes are === npm packages. Seems this is not possible
conventional-changelog/conventional-changelog#948 but we might be able to write a custom config for this.

It might be worth digging a little more as this would resolve the main issue with this approach.

How sure are you about this? When was the last time that this was a problem? If it's long ago I think it's worth to try again if CircleCI still has a problem with it.

I think last time was over a year ago when we last wrote the release script. It might be and hopefully be resolved. Otherwise we can move that part to Github actions easily as I said.

...I only think we then need to be stricter about processes.

AFAIK there are also other tooling that prevents pushing non-conforming commit messages. I can have a look. But I don't know if we want ALL commits to conform

---

I think in general we need to be mindful when we are merging something and then it should be alright?

[kuzdogan]

But also, if conventional commits will bring more overhead to organizing commits etc. it's not favorable. By doing the tagging and publishing on staging we get rid of some parts of the script that is prone to errors. Then we can semi-automate to write changelogs manually and release.

My feeling now more is that conventional commits are designed for non-monorepos and it get's quite tricky with a monorepo with different packages

[manuelwedler]

But also, if conventional commits will bring more overhead to organizing commits etc. it's not favorable. By doing the tagging and publishing on staging we get rid of some parts of the script that is prone to errors. Then we can semi-automate to write changelogs manually and release.

What we could also do is writing changelog entries inside the PR of the change. You would gather these at the top of the changelog as unreleased until they are released finally. I worked like that in past. I think our current approach of writing the entries during release is error prone and it is easy to forget something. So, we wouldn't need conventional commits to solve that.