refer underlies require, use, and refer-clojure use cases and is not particularly efficient at its primary job of copying symbol/var mapping from one namespace to another.
Approach: Some improvements that can be made:
Go directly to the namespace mappings and avoid creating filtered intermediate maps (ns-publics)
Use transients to build map of references to refer
Instead of cas'ing each new reference individually, build map of all changes, then cas
For (:require :only ...) case - instead of walking all referred vars and looking for matches, walk only the included vars and look up each one
There are undoubtedly more dramatic changes (like immutable namespaces) in how all this works that could further improve performance but I tried to make the scope small-ish for this change.
While individual refer timings are greatly reduced (~50% reduction for (refer clojure.core), ~90% reduction for :only use), refer is only a small component of broader require load times so the improvements in practice are modest.
expr in a new repl
(in-ns 'foo) (clojure.core/refer 'clojure.core)
(in-ns 'bar) (clojure.core/refer 'clojure.core :only '[inc dec])
(require '[clojure.core.async :refer (>!! <!! chan close!)])
Patch appears correct but we should consider:
non-idiomatic use of if instead of when makes branches hard to read
non-idiomatic indentation of if (both branches on one line) hard to read
I don't think not found should be an IllegalAccessError, but the original code did this already, so ...
the optimistic concurrency loop around the swap will never give up, is this ok?
Patch updated to address first screening comment. I didn't actually find any case of the second one? Give me a pointer.
I suspect that this will not matter much for peak performance. Given the attached picture of clojure.instant loading, this patch may shave a bit off startup time.
(picture made with Bytestacks, profile recorded a JDK11 debug build from today)