Test::Smoke is missing some output from the test summary

[thibaultduponchelle]

Observed when investigating https://perl5.test-smoke.org/report/5022575
The logfile linked with that report contains:

# Test process timed out - terminating
make: *** [makefile:856: test_harness] Error 1
error while running harness target 'test_harness': 2 at /home/perl/smoke/ts/Test-Smoke-1.72-custom/bin/../lib/Test/Smoke.pm line 195.
[2022-09-07 04:04:48-0400]
[2022-09-07 04:04:48-0400] ../t/re/pat_advanced.t......................................FAILED
[2022-09-07 04:04:48-0400]     No plan found in TAP output

I was able to reproduce this failure by slightly altering the perl5.git code using:

diff --git a/t/re/pat_advanced.t b/t/re/pat_advanced.t
index 2844861..2981612 100644
--- a/t/re/pat_advanced.t
+++ b/t/re/pat_advanced.t
@@ -2383,6 +2383,7 @@ EOF
         todo_skip('Triggers thread clone SEGV. See #86550')
          if $::running_as_thread && $::running_as_thread;
         watchdog(10);
+        sleep 11;
         like("\x{00DF}", qr/[\x{1E9E}_]*/i, "\"\\x{00DF}\" =~ /[\\x{1E9E}_]*/i was looping");
     }

which will ensure the test times out.

When running the smoke I then get the same output in the report and in the log.
When running make test_harness manually:

Test Summary Report
-------------------
re/pat_advanced.t                                                  (Wstat: 9 (Signal: KILL) Tests: 1635 Failed: 0)
  Non-zero wait status: 9
  Parse errors: No plan found in TAP output

In mktest.rpt (and in smokecurrent.log) two things are missing:

• the 'Non-zero wait status', this is probably easy enough to fix1
• the (Wstat: 9 (Signal: KILL) Tests: 1635 Failed: 0) information. This contains the most useful info since it shows how many tests were already run, how many failures and what happened with the test (i.e. it got killed)

IMO this info should be included in the report (and in the log).
And/or include all of the make test_harness output in the log, a WIP for that is PR abeltje/Test-Smoke#68 but it has several open questions

(Ideally the report would also contain # Test process timed out - terminating but it doesn't look so easy to link that with a particular test file; doing that may need changes in perl5.git)

---

Footnotes

There are several places that match on Non-zero exit, changing these to: Non-zero (?:exit|wait) is probably enough ↩