Switch coverage providers?

[adamjstewart]

Summary

We should consider switching coverage providers.

Rationale

We've been proudly using codecov since the beginning of the project, but codecov has become unusable over the last year. The vast majority of tests result in 404 errors during coverage upload: codecov/codecov-action#903. The "solution" is to use our codecov secret during upload: codecov/feedback#126. However, secrets are not shared with forks, where 99% of our PRs originate from. The codecov team has not provided any recommendations for how to use codecov for open source projects, and coverage uploads now have a nearly 100% failure rate: codecov/feedback#301.

Implementation

My first thought was to replace codecov in a single PR, but I think it's actually much easier and safer to simply add additional coverage providers and test them jointly for a few weeks before making a final decision. This will prevent service disruption and allow us to explore features before making a decision.

Alternatives

There are many alternatives to codecov:

• coveralls
• codacy

Spack used to use coveralls, but we switched to codecov because it offered GitHub integration (you could see the lines of coverage in your PR). However, codecov abandoned GitHub integration, meaning they no longer have any significant advantage over coveralls or codacy other than support for more languages (we only care about Python anyway).

Additional information

It would be interesting to get feedback from all of these providers to see what features they offer and whether or not they suffer from the same GitHub API rate limit that codecov does:

• codecov: @thomasrockhu-codecov @rohan-at-sentry
• coveralls: https://calendly.com/coveralls-support/demo
• codacy: https://www.codacy.com/bookdemo?hsLang=en

[adamjstewart]

I've removed codecov as a required status check since it almost never passes. For now, we'll have to manually verify whether or not coverage really decreases.

There is some movement upstream: codecov/feedback#301 (comment)

I think it's still worth adding coveralls and codacy to CI just to see how they compare and see if they can get us reliable coverage reports while we wait for codecov to come back online.