[spec] Improve generation failure exception
Description
It's pretty easy to write a spec whose generator fails like this:
This is of course expected in many ways, but it's a very unhelpful error. Some things that could make this better include:
Including the spec that failed in the exception. I only see one invocation of gen/such-that in spec.clj, and it appears to have the spec's form at hand. gen/such-that takes an exception constructor where this could be used.
Allow max-tries to be changed from the hardcoded value of 100. When dealing with an intermittent failure, it can be useful to crank down max-tries to a very small number, making the failure easier to reproduce.
Environment
Activity
These are reasonable suggestions and this area is likely to evolve in tandem with test.check to provide better info.
This simple change to clojure.spec.alpha/gensub did the trick for me (note: only path and form have meaningful content. Likely I do not need form when I have path but I know too little to be sure)
(gen/such-that #(valid? spec %) g 100)
(gen/such-that #(valid? spec %) g {:max-tries 100, :ex-fn (fn [{:keys [gen pred max-tries]}] (ex-info (str "Couldn't satisfy such-that predicate after " max-tries " tries.") {:path path, :form form}))}) ; Or perhaps (abbrev form)instead of form as used elsewhere?
This results in the following, useful error:
ExceptionInfo: Couldn't satisfy such-that predicate after 100 tries. {:path [:cbm/invoiceGroup], :form :cbm/invoiceGroup}
While before the error was:
ExceptionInfo: Couldn't satisfy such-that predicate after 100 tries. {:pred #object[clojure.spec.alpha$gensub$fn__1876 0x3a0dfc51 "clojure.spec.alpha$gensub$fn__1876@3a0dfc51"], :gen #clojure.test.check.generators.Generator{:gen #object[clojure.test.check.generators$gen_fmap$fn__14242 0x2ef1f55e "clojure.test.check.generators$gen_fmap$fn__14242@2ef1f55e"]}, :max-tries 100}
Here, gen and pred come from test.check. Spec’s spec doesn’t seem useful either:
:spec #object[clojure.spec.alpha$every_impl$reify__2254 0x15a33c73 "clojure.spec.alpha$every_impl$reify__2254@15a33c73"]}
The full modified code of gensub is here, in a gist: https://gist.github.com/holyjak/8cadc0d939c8e637ef6bf75b070d28b4
NOTE: It would be even better to be able to include an example of failed value and/or explain-data of it but we will likely need support from test.check for that.
Added clj-2097-1.patch Oct 2nd 2019.
Added clj-2097-2.patch 2019-10-03 this time with the pom change to update test.check
When is available, we can use failed-value in the :ex-fn to add something like this to the ex-info map:
:explain
(->> example
(explain-data spec)
:clojure.spec.alpha/problems)
which will make it much easier to understand why the generator failed to generate something that did not match the spec.
