Provide sample failed value to such-that's ex-fn for better error messages

Description

The `:ex-fn` that we can pass to `such-that` gets the argument `{`:gen` .., `red` .., `:max-tries` ..}`. It would be very helpful to include also a sample value that failed the predicate.

This is especially useful for Clojure Spec that uses

(such-that #(valid? spec %) g 100)

for custom generators defined for a particular spec. Currently we can learn that the spec for `:erson` failed to match any of the generated values - but if we had also a sample failed value available, we could run `(spec/explain-data spec <the value>)` on it and learn e.g. that the problem was that erson :address :zip had too few numbers.

So provide an additional key, something like `failed-value` or `sample-failed-value` or `first-failed-value`.

Environment

None

Activity

Show:
Jakub Holý
October 3, 2019, 9:23 AM
Edited

Tested with the REPL:

 

Jakub Holý
January 2, 2020, 4:18 PM

Now trully added the missing tcheck-156-2.patch including the renaming and gfredrick’s feedback

gfredericks
January 2, 2020, 4:20 PM

Most of the feedback surrounded these first impressions (added here for posterity – they’ve largely been addressed in the latest patch):

  1. maybe there should be a check at the beginning that max-tries is positive

  2. instead of using or, you should anticipate that falsey values might be legitimate

  3. is it obvious that the first generated value is the best example?

 

gfredericks
January 2, 2020, 4:27 PM

tcheck-156-2.patch looks good to me

Jakub Holý
January 6, 2020, 9:32 AM

As Gary mentioned, 1 and 2 have been adressed. Regarding 3, I agree we don’t (cannot?) know which example is a good/best one. My thinking is that the last example is likely huge and overwhelming. The first example is minimal, which is good. The bad thing is that it can be so minimal as to be useless (nil, empty (), …) - but the user can tune that by chain multiple such-that to exclude these trivial cases. In pseudocode:

;; instead of (such-that pred gen):
(->> gen
(such-that #(not (empty? %)))
(such-that pred)))

Therefore returning the first one is better than the last or a random one, I think.

 

PS: This is necessary for

 

 

Assignee

gfredericks

Reporter

Jakub Holý

Labels

None

Approval

None

Patch

Code

Priority

Major
Configure