Open issues

Memory issue when reading a string


[imported from]


I came across a really bad behavior of the reader inside the browser, and scratched my head for the last 3 days trying to find a solution which I expose below (but you might have a better alternative, so I'd like feedback before starting the full process of submitting a patch through JIRA).

The context: I'm developing a web app that reads an edn (ajax call), and parses the result with read-string. As it is pure edn, I use the version in edn.cljs, but the problem also affects reader.cljs.
The problem I'm facing: I read a 3M edn file (hash-map containing a lot of strings of roughly 1000 characters). The memory usage for the resulting object blew up to around 90M...

I started exploring the heap and found out that the read-string* function resulted in concatenated strings of one character each, so each character in the strings ended up using 32 bytes! The problem seems to be adding one character at a time to the StringBuffer. While I understand the use of the StringBuffer for performance reasons, I found no way to "force" a conversion to a primitive string once reading the string is over.

On the other hand, using plain clojurescript str does the work, and I didn't notice a significant performance overhead. Memory usage went down to 7M. (In fact, since memory usage went down, I assume there is even less pressure on the GC). Here is my new read-string* in edn.cljs:

1 2 3 4 5 6 7 8 9 10 (defn- read-string* [rdr _ opts] (loop [sb "" ch (read-char rdr)] (case ch nil (err/throw-eof-reading rdr :string \" sb) \\ (recur (str sb (escape-char sb rdr)) (read-char rdr)) \" sb (recur (str sb ch) (read-char rdr)))))

For the time being, I ship this patched version with my app, but others might benefit from this, or you might have an idea for a more performant low-level solution. (In fact, I'm not really sure why this works because str also uses a StringBuffer internally).

Another side note: escape-char takes sb as argument, but it is never used in that function...

Thanks in advance for your feedback!







Nicola Mometto