Support monorepo internal relative dependencies

Description

This was hinted at in https://clojure.atlassian.net/browse/TDEPS-123?focusedCommentId=26919&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-26919

There are repos which contain many dependencies, where those dependencies in the repo are inter-dependent on each other. Currently there is no mechanism for consuming a dependency from a repo which is set up like that.

 


 

In concrete terms, I’d like this deps.edn to work with the :dev alias activated.

Instead it fails with

It doesn’t because Edge is a monorepo. As part of that monorepo both juxt.edge/lib.app.dev and juxt.edge/lib.app depend on juxt.edge/edge.system via :local/root. In more abstract terms, Edge currently is used by cloning it and building your own application on top of it that way. This works great with {:local/root}. However, I would also like to support the use via a git dependency (like above) in order to not have users require the full Edge repo as part of their repo. I would like to stress that I want to support both use cases.

Options considered for solving this:

  • Use a hash to determine if the contents of the two directories are equivalent

    • Con: More code than path comparison

    • Con: If a user creates a file in the directory (e.g. an emacs swap file or similar) then the dependencies will no longer compare. (Could mark the libs as RO to avoid this)

    • Pro: Covers cases where dependencies have been inlined into projects, but are comparable (uncommon use case though - weak)

  • Create a new dependency type :repo/root which varies it’s behavior depending on how it is used. If it’s parent is a :local or :git It will use that to set it’s basis, similar to how *the-dir* works for just :local/root

    • Pro: Very specific, narrow use-case.

    • Con: Lots of work to rework tda to make dependencies be aware of the parent

Environment

None

Activity

Show:
Dominic Monroe
January 31, 2020, 10:02 AM

Based on our discussion yesterday, I’ve expanded on the use-case. There’s now a concrete example of exactly what I’d like to achieve. I’ve included the possible solutions I’ve thought of, and tried to put down some pros/cons for them.

Alex Miller
February 12, 2020, 9:05 PM

I think the first option seems fine - you can use File.getCanonicalPath() on each side in the comparison to take care of a lot of "equivalent but equal" kinds of problems.

Dominic Monroe
February 14, 2020, 7:57 AM

Comparing local/root paths as equivalent doesn’t solve my problem. I had a bug in my patch.

This is because the full paths of the git dependencies are slightly different. They are grouped by lib then sha, instead of by URL then sha. This means that the transitive dependencies of lib.app and lib.app.dev are actually in different paths. I’ll remove that as an option.

Dominic Monroe
February 14, 2020, 8:30 AM

I’ve attached a patch which uses approach 2: it computes a checksum for the directories and compares them. The directories have to be exactly equal (no more or less files), and laziness is used, so it should abort as soon as there’s a discrepancy.

 

I’ve also added an additional con to approach 2.

Dominic Monroe
August 13, 2020, 9:41 PM

A while ago we discussed the option of a “nearest git repository” on slack. I wanted to record that. But it’s also lacking a problem statement.

The idea was a new coordinate type which would search for its nearest repo. In the case of working within the repo, it would search upwards and find the .git folder. In the case of a git/url, it would do the same. This would solve the transitive dependency problem.

Assignee

Unassigned

Reporter

Dominic Monroe

Labels

None

Approval

None

Patch

None

Priority

Major
Configure