LazyTransformer does a lot of work to be a seq. Instead, switch to creating a transforming iterator.
Change sequence to wrap iterator-seq around the transforming iterator.
Change the iterator-seq implementation to be chunked. IteratorSeq will no longer be used but is left in case of regressions for now.
Change Eduction to provide iteration directly via the transforming iterator.
Extend eduction to support multiple xforms.
alpha5 == before any change
alpha6 == after clj-1669-6.patch was applied
beta3 == latest, includes range enhancement, expanding mapcat enhancement, etc
;; using java 1.8
(def s (range 1000))
(def v (vec s))
(def s50 (range 50))
(def v50 (vec s50))
(into  (->> s (interpose 5) (partition-all 2)))
(into  (->> s (eduction (interpose 5) (partition-all 2)))) *
1 chunking transform
(into  (map inc s))
(into  (map inc) s)
(into  (sequence (map inc) s))
(into  (eduction (map inc) s))
(doall (map inc (eduction (map inc) s)))
2 chunking transforms
(into  (map inc (map inc s)))
(into  (comp (map inc) (map inc)) s)
(into  (sequence (comp (map inc) (map inc)) s))
(into  (eduction (map inc) (map inc) s)) *
(doall (map inc (eduction (map inc) (map inc) s))) *
(into  (mapcat range (map inc s50)))
(into  (sequence (comp (map inc) (mapcat range)) s50))
(into  (eduction (map inc) (mapcat range) s50)) *
(sort (eduction (map inc) s))
(->> s (filter odd?) (map str) (sort-by last))
(->> s (eduction (filter odd?) (map str)) (sort-by last))
used comp to combine xforms as eduction only took one in the before case
Look at perf for:
->> eduction transformation
transformation comparison that doesn't support chunking
more into vector iteration case
The -5 patch is same -3 except all uses of IteratorSeq have been replaced with a ChunkedCons that is effectively a chunked version of the old IteratorSeq. While no one calls it, I left IteratorSeq in the code base in case of regression.
Generally, the chunked iterator seq reduces the cost in a number of the worst cases and also is a clear benefit in making seqs over a result of eduction or sequence faster to traverse (as they are now chunked).
I think the one potential issue is that seqs over iterators are now chunked when they were not before which could change programs that expect their stateful iterator to be traversed one at a time. This change could be isolated to just to sequence and seq-iterator and mitigated by not changing RT.seqFrom() and seq-iterator to use the new chunking behavior only in sequence and/or with a new chunked-iterator-seq to make it more explicit. The sequence over xf is new so no possible regression there, everything else would just be opt-in.
push as is but leave unresolved, for perf tweaks
clj-1669-6 is identical to clj-1669-5 but removes two commented out debugging lines that were inadvertently included.
updated for beta3 numbers