Skip to:

  1. Skip to Jira Navigation
  2. Skip to Side Navigation
  3. Skip to Main Content

New stats feature: Adds ability to assign labels to test cases to report the test case distribution when running a test

Description

Inspired in the analog feature that exists in Haskell's QuickCheck. Adds a classify fn that is intended to be used as a wrapper of prop/for-all, returning a property (a generator) appropriate for test.check/quick-check that will augment the result-map returned by the underlying property, adding the collected labels under the :labels key. Also triggers a new event :stats in the default-reporter-fn whose default implementation calls test.check.stats/print, printing the classification of trials with the following format:

12.7% :lt-30 14.5% :gte-30 29.1% :lt-30, :lt-20 43.6% :lt-30, :lt-10, :lt-20

(note that multiple labels might be assigned to some test cases)

I think it answers the question "How could we collect stats about the sorts of things generated?" from the test.check design page

Environment

None

Attachments

1
  • 23 Nov 2016, 11:53 PM

Activity

Show:

Nicolás Berger November 23, 2016 at 11:53 PM

Added new patch TCHECK87-add-stats-feature-3.patch rebased on current master with some squashed commits. Also fixes the print-stats test in ClojureScript.

Nicolás Berger November 17, 2016 at 1:27 PM

Added a new patch TCHECK87-add-stats-feature-2.patch that is rebased on current master. I don't think it's the final version (I'd like to find a way to not pollute the main quickcheck loop with the labels stuff for example) but it's hopefully getting closer.

Nicolás Berger April 19, 2016 at 3:19 PM

Replaced with new patch that adds both stats/classify & stats/collect as discussed. I think this new patch implements what was discussed, covering cases 2 & 5 with function names analog to the haskell impl, leaving out any special treatment for nil (it is a valid label) and not adding :stats/ignore as I suggested in a previous comment.

Nicolás Berger April 11, 2016 at 4:49 AM

We talked about maybe treating nil as a flag for not labelling in collect, but I don't particularly like that idea I don't think.

Perhaps we can use a namespaced keyword to signal that a label should be ignored? Something like clojure.test.check.stats/ignore. This way we can easily implement classify in terms of collect by creating a function that returns stats/ignore when pred doesn't match:

(defn collect [prop label-fn] (gen/fmap (fn [{:keys [args] :as result-map}] (let [label (apply label-fn args)] (if (= ::ignore label) result-map (update result-map :labels conj label)))) prop)) (defn classify [prop pred label] (collect prop (fn [& args] (if (apply pred args) label ::ignore))))

Another option could be to add an extra arity in collect, to receive a flag about whether nil should be treated as a label or if it should be ignored. I like :stats/ignore more

gfredericks April 6, 2016 at 4:09 AM

I should also not forget that this might overlap with changes to address the "Test Failure Feedback" issue on the confluence page: http://dev.clojure.org/display/design/test.check

Details

Assignee

Reporter

Patch

Code and Test

Priority

Created December 3, 2015 at 11:28 PM
Updated November 23, 2016 at 11:53 PM

Flag notifications