Calculation of dependencies after compilation could be improved

Description

We have a fairly large CLJS project (some ~50k lines of CLJS in ~700 files, ~15 external dependencies) and the hot reload provided by Figwheel Main has started to be unusable in its current state. The (incremental) compilations may take anything from 20 seconds to over 2 minutes even with some trivial changes to a single file.

We have transitioned to mostly evaluate forms directly to the REPL, but when evaluating forms for the first time in a NS that has not been used in the session yet it takes a substantial amount of time (~10s of seconds) of time before completing the first eval. The evals after the first one are much faster.

This does not seem to be the case if I just type in the CLJS REPL opened by figwheel.main, it just evaluates what’s been given to it, immediately. (EDIT: This seems to be caused by the fact that Fireplace in VIM does a “preload” of an NS that has not yet been loaded, effectively require-reloading it causing compilation)

I did some research on what should be causing this and the first thing that I noticed was that the dependencies are calculated per ns-require and are always submitted as is. For some namespaces, having for instance 6-10 other dependencies causes this to send out the same amount of goog dependency definitions (of ~500kb in size) that consists of mostly duplicate information.

This can be fixed at least so that the submission of dependencies is done only once, reporting the unique depdendency trees. I have included a suggested patch that does the compactification of deps and it seems to already do a multiple second improvement to the situation.

Also I'd like to see if there is a way to skip the recalculation of dependencies if they have not changed, but this needs a cache of somekind to be introduced and I'd first like to discuss this with someone more familiar with this.

Also I try to find time to pinpoint more of the spots where it is slowing down, since almost all of our other projects that are way smaller in size, we have not been experiencing this at all.

This is my first attempt at contributing to this project so any advice is more than welcome.

 

EDIT: Attached is verbose output from CLJS compiler after a file has been saved and Figwheel triggers the recompilation:

 

 

I dug around a bit in figwheel-core -project and it seems to be computing the same dependencies that CLJS compiler is also computing, and it takes most of the time seen above. It has also an internal cache, so if the deps tree has not changed, the sequential updates are much faster, resulting in ~under 20 secs of wall clock time when doing sequential watch-compiles.

So – I think the dependency computation caching could be a possible answer to this, also Figwheel-core could be patched to reuse the same mechanism if this was built in the compiler itself?

 

 

Environment

OS X, Figwheel Main 0.2.3, Latest Clojurescript compiler

Status

Assignee

Unassigned

Reporter

Mikko Harju

Labels

None

Approval

None

Patch

Code

Priority

Minor
Configure