Converging CI

[tmdzk]

Summary:

The current status of CI is very complex due to all the tool that we use. It is very confusing for everyone to understand what run where and which tests are mandatory or not.

What do we currently have:

• Github Action: that run simple jobs like DCO-checks or Netlify jobs (magma website)
• Jenkins https://jenkins.magmacore.org/ : Runs heavy task like integration test on magma core, this jenkins has a couple of packet https://metal.equinix.com/ (ex Packet), also build all the Vagrant boxes (dev) environment automatically
• CircleCi https://app.circleci.com/pipelines/github/magma/magma: that runs every single jobs: dockerized/on baremetal/in virtualized environment. We hacked around to plug our own environments using a custom VPN server and a scheduling api that allow to run remote jobs on Baremetal

Goal of that proposal:

Converge to a simple and unique solution for CI under the same tool.

#5064

[mattymo]

@tmdzk Should bring up the topic of the current onboarding of other 3rd party code analysis tools in this discussion?

[tmdzk]

We can bring it but the main focus is align all entities around magma on a converging solution.

If you want to understand what @Scott8440 is experimenting just search for "fuzz" or "code coverage" in Github issue or Zenhub

[tmdzk]

https://app.zenhub.com/workspaces/magma--devops-5f98596bdbbaee000ef2c1f6/issues/magma/magma/5076

I created an epic for the Benchmark of Github Action

[hcgatewood]

I'm fine with the coexistence of GitHub Actions and CircleCI, where CircleCI is the principal tool and GitHub Actions is for smaller stuff like DCO check, PR linting, etc

I think the real win here will be transitioning as much of Jenkins etc to centralized CircleCI. Having a single entry-point + global view of our testing functionality will be a huge win

[tmdzk]

The problem is that CircleCI Scale plan (the one that allow custom runners) is way too expensive and we can't use it..

[hcgatewood]

I see, sad. Does GitHub Actions resolve this issue for us or ?

Also, I've otherwise very content with CircleCI. Let's make sure any migration to GitHub Actions follows a measured rollout, cause non-integ jobs are pretty great on CircleCI right now