[spec] Improve generation failure exception

Description

It's pretty easy to write a spec whose generator fails like this:

This is of course expected in many ways, but it's a very unhelpful error. Some things that could make this better include:

  • Including the spec that failed in the exception. I only see one invocation of gen/such-that in spec.clj, and it appears to have the spec's form at hand. gen/such-that takes an exception constructor where this could be used.

  • Allow max-tries to be changed from the hardcoded value of 100. When dealing with an intermittent failure, it can be useful to crank down max-tries to a very small number, making the failure easier to reproduce.

Environment

None

Activity

Show:
Jakub Holý
October 3, 2019, 8:40 AM
Edited

When is available, we can use failed-value in the :ex-fn to add something like this to the ex-info map:

:explain
(->> example
(explain-data spec)
:clojure.spec.alpha/problems)

which will make it much easier to understand why the generator failed to generate something that did not match the spec.

Jakub Holý
October 3, 2019, 8:37 AM

Added clj-2097-2.patch 2019-10-03 this time with the pom change to update test.check

Jakub Holý
October 2, 2019, 3:50 PM

Added clj-2097-1.patch Oct 2nd 2019.

Jakub Holý
October 2, 2019, 11:38 AM
Edited

This simple change to clojure.spec.alpha/gensub did the trick for me (note: only path and form have meaningful content. Likely I do not need form when I have path but I know too little to be sure)

(gen/such-that #(valid? spec %) g 100)

(gen/such-that #(valid? spec %) g {:max-tries 100, :ex-fn (fn [{:keys [gen pred max-tries]}] (ex-info (str "Couldn't satisfy such-that predicate after " max-tries " tries.") {:path path, :form form}))}) ; Or perhaps (abbrev form)instead of form as used elsewhere?

This results in the following, useful error:

ExceptionInfo: Couldn't satisfy such-that predicate after 100 tries. {:path [:cbm/invoiceGroup], :form :cbm/invoiceGroup}

While before the error was:

ExceptionInfo: Couldn't satisfy such-that predicate after 100 tries. {:pred #object[clojure.spec.alpha$gensub$fn__1876 0x3a0dfc51 "clojure.spec.alpha$gensub$fn__1876@3a0dfc51"], :gen #clojure.test.check.generators.Generator{:gen #object[clojure.test.check.generators$gen_fmap$fn__14242 0x2ef1f55e "clojure.test.check.generators$gen_fmap$fn__14242@2ef1f55e"]}, :max-tries 100}

Here, gen and pred come from test.check. Spec’s spec doesn’t seem useful either:

:spec #object[clojure.spec.alpha$every_impl$reify__2254 0x15a33c73 "clojure.spec.alpha$every_impl$reify__2254@15a33c73"]}

The full modified code of gensub is here, in a gist: https://gist.github.com/holyjak/8cadc0d939c8e637ef6bf75b070d28b4

NOTE: It would be even better to be able to include an example of failed value and/or explain-data of it but we will likely need support from test.check for that.

Alex Miller
January 11, 2017, 2:41 PM

These are reasonable suggestions and this area is likely to evolve in tandem with test.check to provide better info.

Assignee

Unassigned

Reporter

import

Labels

Approval

None

Patch

Code

Priority

Major