It can be really useful for optimization of generative testing(with specs) if values outputted by generators included attempts and time as metadata.
Example:
(-> (s/exercise string? 1)
first
meta)
;; => {:time-ms 2, :attemts 1}
It can help in finding slowest generators which can benefit the most from customization.
Comment made by: jare
but I don't think it's obvious how this approach applies to some of the combinators, like gen/fmap
The metadata shouldn't be available from withing the function that gen/fmap applies. Instead the time spent inside the function and other performance stats should be assoc-ed with the gen/fmap inputs stats and then returned attached to the outputs. So the data will be carried by the context between nested gen/ calls.
The metadata shouldn't be available from withing the function that gen/fmap calls.
Alternatively you can allow to peak into the inputs stats but not alter it. Not sure about that. It may be too confusing.
Comment made by: jare
Actually if you allow peeking it might be used not only for debugging but also to alter the generation for performance.
For amortisation of the cumulative generator complexity, to meet some deadline by switching routes.
- - - -
@gfredericks For some reason I can't add a new comments to any issue so I update the old one.
For:
complexity function should output
here :time-ms 5 is time spent inside f function.
For example, let's say I have:
When I generate a value from g2, what kind of information will I get? Will it be about g1, or about f, or about both somehow?
I believe your comment permissions are fixed.
Okay, I can see that it might be a well-defined concept. It would be a non-trivial amount of work, though, and I'd want to be sure that there wouldn't be a major performance difference. If there is, we'd probably want a way to have it off by default, and make sure that there's not a performance hit even when off.
Comment made by: jare
Yeah works now. Thx.