TRDR edn/read throws pushback overflow exception on java.io.PushbackReader for files with symbols beginning with dash

Description

This is pretty specific, and it appears that there is a reasonable workaround, but I haven't dug into the root cause to see whether other cases might trigger this.

With tools.reader version 1.3.2, if you create a reader using (java.io.PushbackReader. (clojure.java.io/reader "some-file.txt")), and it contains a symbol that begins with a dash character, e.g. `->` or `-main`, and use the tools.reader's clojure.tools.reader.edn/read function repeatedly on the file until EOF is reached, it will throw an exception with cause "Pushback overflow exception".

You can take the same file and instead create a reader using (clojure.tools.reader.reader-types/push-back-reader (clojure.java.io/reader "some-file.txt")), and it reads the values correctly, no exception thrown.

I came across this while trying both clojure.edn/read and tools.reader's EDN reader to read about 12,000 different project.clj files from many open source projects on Github.com, and was able to find this common denominator to their contents by trimming the file down smaller and smaller until I got it down to this. Even a file containing the two characters dash followed by a newline causes it.

Example REPL session to reproduce the problem:

$ clojure -Sdeps '{:deps {org.clojure/tools.reader {:mvn/version "1.3.2"}}}'

;; setup steps where return vales aren’t interesting

(import '(java.io PushbackReader))

(require '[clojure.java.io :as io])

(require '[clojure.tools.reader.edn :as tredn])

(require '[clojure.tools.reader.reader-types :as rt])

(spit "test-file.txt" "-\n")

 

;; This works as expected

user=> (tredn/read (rt/push-back-reader (io/reader "test-file.txt")))

-


;; This fails

user=> (tredn/read (PushbackReader. (io/reader "test-file.txt")))

Execution error (IOException) at java.io.PushbackReader/unread (PushbackReader.java:155).

Pushback buffer overflow


;; PushbackReader with non-default pushback buffer size of 2 also works for this and other examples I tried

user=> (tredn/read (PushbackReader. (io/reader "test-file.txt") 2))

-

Environment

None

Activity

Show:
Andy Fingerhut
February 13, 2020, 6:57 AM

I tested the same results with:

Ubuntu 18.04 Linux + AdoptOpenJDK 1.8.0_222-b10 + Clojure 1.10.1 + tools.reader 1.3.2

Ubuntu 18.04 Linux + OpenJDK Runtime Environment Zulu11.35+15-CA (build 11.0.5+10-LTS) + Clojure 1.10.1 + tools.reader 1.3.2

Ubuntu 18.04 Linux + OpenJDK Runtime Environment (build 11.0.6+10-post-Ubuntu-1ubuntu118.04.1) + Clojure 1.10.1 + tools.reader 1.3.2

Andy Fingerhut
February 13, 2020, 5:48 PM

Another variant that avoids the exception is to use the java.io.PushbackReader constructor with.2 args, with the 2nd arg equal to 2, specifying a pushback buffer size of 2 instead of the default of 1. I am only claiming that this technique avoids the exception in the cases I have tested, not that I have any kind of reasoning about the code’s behavior in general that I know that will always avoid the exception.

Assignee

Unassigned

Reporter

Andy Fingerhut

Labels

None

Approval

None

Patch

None

Priority

Minor