Reload on a function-by-function basis for improved performance in large projects

Description

CURRENT SITUATION

Currently tools.namespace reloads portions of a given Clojure project on a file-by-file (really, namespace-by-namespace) basis. While it works great and performs quite well for most projects — ones which feature namespaces which compile quickly, dependency graphs which don't have too many nodes, and/or dependency graphs which aren't very deep — developers experience a (potentially) significant slowdown the less the project conforms to these three features.

POTENTIAL SOLUTION

A potential solution is this:

  • Walk the forms in all the project's namespaces (significant up-front cost but amortized with incremental reloads)

  • Create a dependency graph of (namespaced, non-local) vars, not just namespaces

  • Detect whether a var's compile-time value (e.g. in the case of a `defn`, its metadata, params, and unevaled body) has changed

  • If so:

  • Reload all the vars which depend on it (according to the dependency graph previously created)

  • Compile it

  • Update the dependency graph

  • Detect whether a var has been added/created

  • If so:

  • Compile it

  • Update the dependency graph

As for tests, it would be nice to re-run only the tests affected by a change to a var, not all the tests in that namespace every time the namespace changes at all.

Essentially, this feature would increase reload granularity (and reload speed) in the same way that namespace-reloading increases granularity (and speed) from recompiling the entire project.

Environment

None

Activity

Show:
import
July 24, 2016, 4:27 PM

Comment made by: alexandergunnarson

Sorry, the nested lists got messed up. Should be:

  • Walk the forms in all the project's namespaces (significant up-front cost but amortized with incremental reloads)

  • Create a dependency graph of (namespaced, non-local) vars, not just namespaces

  • Detect whether a var's compile-time value (e.g. in the case of a `defn`, its metadata, params, and unevaled body) has changed

    • If so:

      • Reload all the vars which depend on it (according to the dependency graph previously created)

      • Compile it

      • Update the dependency graph

  • Detect whether a var has been added/created

    • If so:

      • Compile it

      • Update the dependency graph

Stuart Sierra
October 21, 2016, 2:22 PM

This is a fascinating idea, one which I would be happy to see developed in the future. However, it is sufficiently different from tools.namespace that it would best be pursued in a completely separate project with a different name.

import
October 21, 2016, 9:19 PM

Comment made by: alexandergunnarson

That makes sense. Thanks for your input!

Declined

Assignee

Stuart Sierra

Reporter

import

Labels

None

Approval

None

Patch

None

Priority

Major