Metadata for tuning on generated values

Description

It can be really useful for optimization of generative testing(with specs) if values outputted by generators included attempts and time as metadata.
Example:

(-> (s/exercise string? 1)
first
meta)
;; => {:time-ms 2, :attemts 1}

It can help in finding slowest generators which can benefit the most from customization.

Environment

None

Activity

Show:
import
February 25, 2019, 3:13 AM

Comment made by: jare

but I don't think it's obvious how this approach applies to some of the combinators, like gen/fmap

The metadata shouldn't be available from withing the function that gen/fmap applies. Instead the time spent inside the function and other performance stats should be assoc-ed with the gen/fmap inputs stats and then returned attached to the outputs. So the data will be carried by the context between nested gen/ calls.

The metadata shouldn't be available from withing the function that gen/fmap calls.

Alternatively you can allow to peak into the inputs stats but not alter it. Not sure about that. It may be too confusing.

import
February 25, 2019, 3:20 AM

Comment made by: jare

Actually if you allow peeking it might be used not only for debugging but also to alter the generation for performance.
For amortisation of the cumulative generator complexity, to meet some deadline by switching routes.

  • - - - -

@gfredericks For some reason I can't add a new comments to any issue so I update the old one.

For:

complexity function should output

here :time-ms 5 is time spent inside f function.

gfredericks
March 3, 2019, 3:08 AM

For example, let's say I have:

When I generate a value from g2, what kind of information will I get? Will it be about g1, or about f, or about both somehow?

gfredericks
March 9, 2019, 11:40 PM

I believe your comment permissions are fixed.

Okay, I can see that it might be a well-defined concept. It would be a non-trivial amount of work, though, and I'd want to be sure that there wouldn't be a major performance difference. If there is, we'd probably want a way to have it off by default, and make sure that there's not a performance hit even when off.

import
March 10, 2019, 12:19 AM

Comment made by: jare

Yeah works now. Thx.

Assignee

Unassigned

Reporter

import

Approval

None

Patch

None

Priority

Minor