With core.async and AOT seeing NoSuchFieldError __thunk__0__

Description

Repro below… (and GitHub - seancorfield/analyzer-async: Repro for AOT compilation problem with Clojure core.async and tools.analyzer )

deps.edn:

{:paths ["src" "resources"] :deps {org.clojure/clojure {:mvn/version "1.11.1"} org.clojure/core.async {:mvn/version "1.6.673"}} :aliases {:build {:deps {io.github.clojure/tools.build {:git/tag "v0.9.2" :git/sha "fe6b140"}} :ns-default build}}}

build.clj:

(ns build (:require [clojure.tools.build.api :as b])) (defn aot [_] (b/delete {:path "target"}) (b/compile-clj {:basis (b/create-basis {}) :class-dir "target/classes" :uber-file "target/example.jar" :ns-compile ['clojure.core.async 'clojure.tools.analyzer.jvm.utils]}))

Building reproduces:

$ clojure -T:build aot Execution error (NoSuchFieldError) at clojure.tools.analyzer.jvm.utils__init/load (REPL:259). __thunk__0__ Full report at: /tmp/clojure-11118341833155594225.edn Execution error (ExceptionInfo) at clojure.tools.build.tasks.compile-clj/compile-clj (compile_clj.clj:114). Clojure compilation failed, working dir preserved: /tmp/compile-clj13767047880507073890 Full report at: /tmp/clojure-3581990850772262725.edn

This is new behavior since https://clojure.atlassian.net/browse/ASYNC-248 which now defers loading of clojure.core.async.impl.ioc-macros until needed.

Also see https://clojure.atlassian.net/browse/CLJ-1886

The lookup thunk in question has got to be https://github.com/clojure/tools.analyzer.jvm/blob/master/src/main/clojure/clojure/tools/analyzer/jvm/utils.clj#L259  which looks like the only thing top-level that would generate one.

Environment

None

Activity

Show:

Kevin Downey March 1, 2023 at 6:22 PM

yes, KEYWORD_CALLSITES is particularly vulnerable to this because the check it uses is just .isBound, so no amount of pushing different bindings for it will “turn it off”

Alex Miller March 1, 2023 at 5:59 PM

I suspect this is similar to https://clojure.atlassian.net/browse/CLJ-1620 (but different case). In general the compiler maintains state in a bunch of dynamic variables (like KEYWORD_CALLSITES) and there are a few places where a new set of bindings is pushed - eval, load, etc. Those binding sets are inconsistent and it seems like either nested context sometimes modifies the parent context, or doesn’t when it should. It is interesting to diff the generated bytecode depending on whether you compile a or b first in the last example and see the changes in both a$f (which gets stuff it shouldn’t) and b__init (which doesn’t get stuff it should).

Kevin Downey February 24, 2023 at 10:56 PM

here is a repro without any external deps as a transcript of a shell history:

% mkdir x % cd x % mkdir src target % clojure -Sdeps '{:paths ["src" "target"]}' Clojure 1.11.1 user=> (spit "src/a.clj" (pr-str '(do (ns a) (defmacro m [] (require 'b) nil) (defn f [] (m))))) nil user=> (spit "src/b.clj" (pr-str '(do (ns b) (def x {:x 1}) (def y (:x x))))) nil user=> (binding [*compile-path* "target"] (compile 'a)) a user=> (load "/b") Execution error (NoSuchFieldError) at b__init/load (REPL:1). __thunk__0__ user=>

Kevin Downey February 24, 2023 at 10:33 PM
Edited

It is possible to repro this without explcitly aot compiling clojure.tools.analyzer.jvm.utils at all

  1. Alter the example above to just aot compile clojure.core.async

  2. Alter the example above to produce an uberjar of the project

  3. Launch a clojure repl from the uberjar and require clojure.tools.analyzer.jvm.utils

The issue seems to be when the aot compilation of clojure.tools.analyzer.jvm.utils is triggered transitively from clojure.core.async clojure.lang.Compiler/KEYWORD_CALLSITES is bound in the compiler, causing it to generate keyword callsite code for a top level form in clojure.tools.analyzer.jvm.utils which doesn’t work correctly, resulting in the missing static field for the thunk.

With core.async 1.5.648 when aot compilation of clojure.core.async triggers transitive aot compilation of clojure.tools.analyzer.jvm.utils clojure.lang.Compiler/KEYWORD_CALLSITES is not bound.

I believe this is properly a clojure compiler bug, because it can be triggered by any macro that loads code during macro expansion, and effects potentially all the dynamic var state used in the compiler.

Kevin Downey February 24, 2023 at 9:57 PM

  1. when clojure.tools.analyzer.jvm.utils is aot compiled then clojure.core.async is aot compiled, the javap -v output for clojure.tools.analyzer.jvm.utils__init doesn’t contain __thunk__0__

  2. when clojure.core.async is aot compiled, then clojure.tools.analyzer.jvm.utils is aot compiled, the javap -v output for clojure.tools.analyzer.jvm.utils__init seems to refer to mention __thunk__0__ in the constant pool, generates code to read and write that static field, but doesn’t actually have such a field.

Details

Assignee

Unassigned

Reporter

Priority

Created February 23, 2023 at 8:55 PM
Updated March 1, 2023 at 6:22 PM