Create a failure test rule

[FaBrand]

When testing for analysis-phase-failures in rules this rule provides otherwise necessary boilerplate that simply allows to test for correct Error messages

I discovered this reoccuring rule while testing some of my rules and thougt it might be useful to others as well.

[FaBrand]

@brandjon @tetromino Do you have any feedback on this?

[FaBrand]

Bump :)

[tetromino]

Thank you for the PR. I agree with the idea and the code; I've searched and found a few independent reimplementations of the exact same logic, so I think the new rule would be useful.

My comments are not about the code, but about all the boring parts (wording, documentation, and tests):

• this is not a general failure test rule - it's rule for testing analysis-time failure. Therefore, it should be named analysis_failure_test (and the file named accordingly)
  • File docstring should be more clear about this - e.g. something like "A test verifying that another target cannot be analyzed"
• You need a docstring for the rule function and all attributes; you can use analysis_test.bzl as an example. The docstring must include a usage example. Otherwise, how could a user trying out this rule know that, for instance, they need to set the target_under_test attribute?
• You need a test for the new test. You'd want to verify that for a target which fails analysis, analysis_failure_test succeeds. You'd want to verify that for a target which passes analysis, analysis_failure_test fails at bazel test time (not at analysis time). This means you need shell tests which launch bazel (a bit like tests/analysis_test_test.sh does).

Nits:

• Copyright year is 2021 :)

[FaBrand]

@tetromino Thanks for the detailed feedback :)
I updated the change with your findings and also added the stardoc target for the new rule and generated the docs.
However i'm not sure if i have much leverage (or maybe missed something) on generating good documentation there.
e.g. the target_under_test or the rule docs attribute can't be set through the analysistest.make and stardoc generates empty defaults. The module docstring does not seem to be included as well.

[tetromino]

Sorry for very delayed review; I'm on leave. But as for the generated docs problem, I realized that we were using a very old Stardoc release via the (deprecated) Federation repo. We'll fix that by #327

[FaBrand]

@tetromino Sry for the even later reaction :/

I fixed the docs and squashed to commits for a cleaner history

[FaBrand]

Kind of forgot this PR 🙈 I updated everything to the latest version. Could you please update your review @tetromino ?
Or is ownership handled differently now @brandjon @comius @hvadehra @c-mita ?

[FaBrand]

ping @brandjon @comius @c-mita

[FaBrand]

Bit later, ping @tetromino 😅