Account for time spent in garbage collection

[jobh]

Avoid flaky DeadlineExceeded due to garbage collection.

See #3975 (comment).

[Zac-HD]

OK, I really like this approach! You've already handled most of the tricky double-counting concerns too, so I have fewer nitpicks than I feared 🙂

[jobh]

All tests passed, finally. There were more complications than I expected, the two major ones are commented above.

The scrutineer part is probably just due to more variation in traces; which can be seen as a good thing, as the variation covers more of real-world usage (although some globbing would be nice to have in that list).

The recursion depth thing may or may not be a problem in practice, I just don't know. I set out to kill one source of flakiness, only to create another. Not happy with that one is ok actually.

Anyway: Ready for re-review :-)

[Zac-HD]

(jumping on a flight to PyCon now, but the Scrutineer thing is not a problem, and I'll aim to review the unraisable problem later)

[jobh]

(jumping on a flight to PyCon now, but the Scrutineer thing is not a problem, and I'll aim to review the unraisable problem later)

Thanks @Zac-HD! Have fun at PyCon!

[Zac-HD]

(just want to note that I haven't forgotten about this; it's just tricky enough that I need a dedicated chunk of time to think about it and make sure I've tested all the edge cases. thanks for your patience 🙏)

[Zac-HD]

Thanks @jobh, I've added a last test based on your repro and I think we're ready to merge!

[jobh]

@Zac-HD I think it's ready now.

Review guidance:

• There are no changes in the users of gc_cumulative_time(), i.e. the actual accounting.
• The "freeze time in conftest.py" patching is reimplemented as separate gc callbacks, to flatten the stack directly and also by avoiding callback indirection.
• The new recursion test is completely rewritten.

Additionally,

• I wrapped the actual test execution (inside execute_once) with ensure_free_stackframes, for maximal consistency. Now, it turned out this wasn't enough to remove it from conjecture/engine.py, since the one in engine.py also protects data generation. Rather than trying to narrow down exactly where the stack is needed, I just left it in place.
• Would you support globbing the scrutineer ignores, something like jobh@5ddfd2e (perhaps additionally simplified by not supporting "**")?
• Globs in scrutineer ignores
• New env variable (for internal use) HYPOTHESIS_NO_TRACEBACK_TRIM. Running in verbose mode is not a realistic option for deeply recursive strategies, but I still sometimes want to see the full stack trace. (I think I just patched it out when working on recursion earlier)

[tybug]

New env variable (for internal use) HYPOTHESIS_NO_TRACEBACK_TRIM

Yes please! I've hacked this locally many times while tracking down internal errors.

test_mostly_does_not_duplicate_blocks_even_when_failing

undoubtedly from #3962. We can just add {1, 4} as a possibility there. I'll do so in #4007 if you don't here

[jobh]

undoubtedly from #3962. We can just add {1, 4} as a possibility there. I'll do so in #4007 if you don't here

Thanks @tybug ! I think #4007 is the better home for it.

[Zac-HD]

Github apparently didn't notify me for a few days (???), but this looks great! Thanks for persisting through @jobh, it's the accumulation of this kind of careful handling that makes our user experience so good 🤩