Feature request — Invert output to find flaky tests and help determine test failures

[seanabraham]

Thanks for the great tool! I've played with it and looked through the documentation and was looking for an option to invert the output. That is, I'd hope for an option where the tool can output which lines in the git diff are covered as opposed to not covered.

My use case is that I see a broken test that seems unrelated to my changes. I'd like to run just that test, collect coverage data and then intersect that with the git diff to see which (if any) of my changes may have been responsible for breaking the test.

This functionality may be broadly useful since if none of the lines in the git diff are covered when executing a given failing test it's a good sign that the test may be flaky. That's actually the main technique proposed in paper on identifying flaky tests: https://www.cs.cornell.edu/~legunsen/pubs/BellETAL18DeFlaker.pdf

If I understand correctly, this is not currently possible but I could be missing something. Thanks in advance!

[Bachmann1234]

You might wanna check out the json report. @ZSmallX just added a bit that shows the lines that are covered.