Reduce time spent creating persistent data structures

Description

Problem:

Tests (see 0001-Playing-with-mutable-collections.patch, where normal, mutable java ArrayLists and HashMaps are used) indicate that a significant amount of time is spent creating the collections needed to hold the parsed json.

Benchmarks:

Current master (d744fb19ff5528320b51316b8d5364049d34797d)

10b

100b

1k

10k

100k

507.105455 ns

2.105140 µs

10.761450 µs

127.119782 µs

1.255270 ms

With 0001-Benching-with-non-pds.patch applied:

10b

100b

1k

10k

100k

444.380352 ns

1.350424 µs

6.255172 µs

75.492237 µs

770.156074 ms

Suggestion:

Explore different ways of efficiently producing persistent data structures when parsing json. One such solution would be to back fly-weight maps and vectors with normal java collections.

Environment

None

Activity

Show:
Luc Préfontaine
April 26, 2021, 8:27 PM
Edited

I don’t disagree with the purpose. I do disagree with the bottom line comparison benchmark however.

You really want to hit the nail in terms of speed ?

Look at this.

Alex Miller
April 16, 2021, 7:04 PM

, the goal here is not to change the behavior to mutable, but to explore the costs. There are a lot of potential options in getting from reading values off a stream to a persistent collection.

Luc Préfontaine
March 30, 2021, 6:40 PM

Well, the goal being to return a persistent structure, that ‘extra’ time is worth it. Now if there’s a speed improvement backing parsing with Java structures and then returning a persistent structure, great.

Transient structures are already speeding things up. But if the benchmark Java structures + conversion to persistent structures is not an improvement, I can’t see how is this useful from Clojure. 😬

Converting from Json to persistent structures and vice-versa to me has more value in the rest of my code. The conversion process itself to/from Json is a very little piece of the whole processing done with the persistent structure.

Erik Assum
March 30, 2021, 6:29 PM

I am aware of the fact that this patch does not return a PDS. The patch is more a “proof” that creating the PDS takes time.

Luc Préfontaine
March 30, 2021, 5:26 PM
Edited

Hi,

Your test doesn’t return a persistent structure if I read carefully ? (Or my eyes are really bad that’s possible ). Or it’s not shown in the patch ?

If not then you should add the conversion step of the result in your benchmark. When I decode JSON in Clojure. I expect persistent structures, not Java ones.

Assignee

Unassigned

Reporter

Erik Assum