Create generator and such-that max retries

[jobtravaini]

Context

A common exception when using a Schema to generate inputs with schema-generators is the Couldn't satisfy such-that predicate after 10 tries..

Uncaught exception, not in assertion.
expected: nil
  actual: clojure.lang.ExceptionInfo: Couldn't satisfy such-that predicate after 10 tries.
{:pred #object[schema_generators.generators$eval23572$fn__23573$fn__23574 0x4b466c6b "schema_generators.generators$eval23572$fn__23573$fn__23574@4b466c6b"], :gen #clojure.test.check.generators.Generator{:gen #object[clojure.test.check.generators$gen_bind$fn__7933 0x19220a00 "clojure.test.check.generators$gen_bind$fn__7933@19220a00"]}, :max-tries 10}
 at clojure.test.check.generators$fn__8055.invokeStatic (generators.cljc:435)
    clojure.test.check.generators/fn (generators.cljc:434)

This is more common for complex schemas with several conditions, and even with targeted generators sometimes a "not so good" generator can cause flakiness to tests when you are using schema to generate inputs.

Issues

It is understandable the suggested approach to create a specific generator for each use case as suggested here #5. There is a implicit downside to this approach which is maintenance of the generator and bug smashing during flakiness of tests.

A developed target generator for a complex defined Schema usually is not a great property tester because developers have tendencies to test the happy path and known paths, this behavior is translated into generators which create only small subsets of the property testing domain.

There is a trade-off between number of retries and domain specific generators for a Schema, such as processing power, test speed, domain property coverage, flakiness.

These trade-offs (and probably others not described here) should be weighted by the development team when deciding which approach to follow to create a test suite for their domain.

Suggestion

Allow targeted generators (as the best option suggested by #5) but also allow to configure such-that max-retries through a Dynamic Var, a function overload on generator, or another option we set fit to allow such feature to be available to development teams.

[w01fe]

Hi, thanks for the detailed report! Definitely open to making this configurable, do you happen to know which such-that is causing the problem? (The such-that that we expose should be able to take additional parameters, but it looks like there are a few hardcoded ones that could be problematic as well)

[jobtravaini]

Hello! Thanks for opening the possibility.

In my use case, the such-that which is causing flakiness is the extend-protocol implementation for the CompositeGenerator protocol here.

If you are comfortable with adding me as a contributor, I can open a PR with a suggestion of change and we can advance to a final solution from there. What do you think?

[w01fe]

A PR would be appreciated, thanks!

for implementation, I like the idea of the fourth argument overload to generator, which could be a function from Schema to max-retries. Open to other options too.

[jobtravaini]

Hello!

Opened #29