With core.async and AOT seeing NoSuchFieldError __thunk__0__
Description
Environment
Activity
Kevin Downey March 1, 2023 at 6:22 PM
yes, KEYWORD_CALLSITES is particularly vulnerable to this because the check it uses is just .isBound, so no amount of pushing different bindings for it will “turn it off”
Alex Miller March 1, 2023 at 5:59 PM
I suspect this is similar to https://clojure.atlassian.net/browse/CLJ-1620 (but different case). In general the compiler maintains state in a bunch of dynamic variables (like KEYWORD_CALLSITES) and there are a few places where a new set of bindings is pushed - eval, load, etc. Those binding sets are inconsistent and it seems like either nested context sometimes modifies the parent context, or doesn’t when it should. It is interesting to diff the generated bytecode depending on whether you compile a or b first in the last example and see the changes in both a$f (which gets stuff it shouldn’t) and b__init (which doesn’t get stuff it should).
Kevin Downey February 24, 2023 at 10:56 PM
here is a repro without any external deps as a transcript of a shell history:
% mkdir x
% cd x
% mkdir src target
% clojure -Sdeps '{:paths ["src" "target"]}'
Clojure 1.11.1
user=> (spit "src/a.clj"
(pr-str
'(do
(ns a)
(defmacro m []
(require 'b)
nil)
(defn f []
(m)))))
nil
user=> (spit "src/b.clj"
(pr-str
'(do
(ns b)
(def x {:x 1})
(def y (:x x)))))
nil
user=> (binding [*compile-path* "target"]
(compile 'a))
a
user=> (load "/b")
Execution error (NoSuchFieldError) at b__init/load (REPL:1).
__thunk__0__
user=>
Kevin Downey February 24, 2023 at 10:33 PMEdited
It is possible to repro this without explcitly aot compiling clojure.tools.analyzer.jvm.utils
at all
Alter the example above to just aot compile
clojure.core.async
Alter the example above to produce an uberjar of the project
Launch a clojure repl from the uberjar and require
clojure.tools.analyzer.jvm.utils
The issue seems to be when the aot compilation of clojure.tools.analyzer.jvm.utils
is triggered transitively from clojure.core.async
clojure.lang.Compiler/KEYWORD_CALLSITES
is bound in the compiler, causing it to generate keyword callsite code for a top level form in clojure.tools.analyzer.jvm.utils
which doesn’t work correctly, resulting in the missing static field for the thunk.
With core.async 1.5.648 when aot compilation of clojure.core.async
triggers transitive aot compilation of clojure.tools.analyzer.jvm.utils
clojure.lang.Compiler/KEYWORD_CALLSITES
is not bound.
I believe this is properly a clojure compiler bug, because it can be triggered by any macro that loads code during macro expansion, and effects potentially all the dynamic var state used in the compiler.
Kevin Downey February 24, 2023 at 9:57 PM
when clojure.tools.analyzer.jvm.utils is aot compiled then clojure.core.async is aot compiled, the javap -v output for clojure.tools.analyzer.jvm.utils__init doesn’t contain __thunk__0__
when clojure.core.async is aot compiled, then clojure.tools.analyzer.jvm.utils is aot compiled, the javap -v output for clojure.tools.analyzer.jvm.utils__init seems to refer to mention __thunk__0__ in the constant pool, generates code to read and write that static field, but doesn’t actually have such a field.
Repro below… (and GitHub - seancorfield/analyzer-async: Repro for AOT compilation problem with Clojure core.async and tools.analyzer )
deps.edn:
{:paths ["src" "resources"] :deps {org.clojure/clojure {:mvn/version "1.11.1"} org.clojure/core.async {:mvn/version "1.6.673"}} :aliases {:build {:deps {io.github.clojure/tools.build {:git/tag "v0.9.2" :git/sha "fe6b140"}} :ns-default build}}}
build.clj:
(ns build (:require [clojure.tools.build.api :as b])) (defn aot [_] (b/delete {:path "target"}) (b/compile-clj {:basis (b/create-basis {}) :class-dir "target/classes" :uber-file "target/example.jar" :ns-compile ['clojure.core.async 'clojure.tools.analyzer.jvm.utils]}))
Building reproduces:
$ clojure -T:build aot Execution error (NoSuchFieldError) at clojure.tools.analyzer.jvm.utils__init/load (REPL:259). __thunk__0__ Full report at: /tmp/clojure-11118341833155594225.edn Execution error (ExceptionInfo) at clojure.tools.build.tasks.compile-clj/compile-clj (compile_clj.clj:114). Clojure compilation failed, working dir preserved: /tmp/compile-clj13767047880507073890 Full report at: /tmp/clojure-3581990850772262725.edn
This is new behavior since https://clojure.atlassian.net/browse/ASYNC-248 which now defers loading of clojure.core.async.impl.ioc-macros until needed.
Also see https://clojure.atlassian.net/browse/CLJ-1886
The lookup thunk in question has got to be https://github.com/clojure/tools.analyzer.jvm/blob/master/src/main/clojure/clojure/tools/analyzer/jvm/utils.clj#L259 which looks like the only thing top-level that would generate one.