-
-
Notifications
You must be signed in to change notification settings - Fork 273
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support a [sources]
section in Project.toml for specifying paths and repo locations for dependencies
#3783
Conversation
Personal views on these:
Updating the source seems the most reasonable thing to do here. Mostly because it's the same thing done for Manifest.toml.
Also update the source.
I think |
More questions:
|
I think it should error, unless it is in # With MyFakePackage appearing in [compat]
ERROR: Compat `MyFakePackage` not listed in `deps`, `weakdeps` or `extras` section at ...
I think it should (similar to
I think that the manifest would take precedence but there should be a warning message that the manifest file is out of date and should be updated. i.e., similar behavior to when you change the ┌ Warning: The project dependencies or compat requirements have changed since the manifest was last resolved.
│ It is recommended to `Pkg.resolve()` or consider `Pkg.update()` if necessary.
└ @ Pkg.API ~/.julia/juliaup/julia-1.10.0+0.aarch64.apple.darwin14/share/julia/stdlib/v1.10/Pkg/src/API.jl:1800 |
I underspecified a bit perhaps. What I means was, you have a manifest with some |
Oh, I see. Then yes I would agree with you the |
I haven't tried this PR yet, but a possible use-case I see for this feature is having test-only dependencies which aren't registered in General, sometimes they aren't really worth being registered on their own. This came up for example in JuliaHEP/Geant4.jl#8, where the current solution is to manually check out the repository. |
Will this be useful for
|
This should work now, see added test. |
I am cautiously excited about this! I'm curious about two use cases though, and it's not immediately clear if this helps with these or not:
|
Yes, that should work fine and it will point to it as a "dev-dependency".
At least for now, I didn't plan on making it "recursive". It's possible but it also has many failure modes where you encounter conflicting information and you may also have to keep downloading stuff as you "discover" more and more URL sources. As a first cut I don't think that will be enabled but it can be further discussed later of course. |
Would it by any chance work if I specify the subpackages in the top-level project though, even if they are not direct dependencies of the top-level project themselves? So instead of the example above, you'd have something like this: [deps]
LocalPackage = "5217a498-cd5d-4ec6-b8c2-9b85a09b6e3e"
[source]
LocalPackage = {path = "packages/LocalPackage"}
LocalSubPackage = {path = "packages/LocalSubPackage/"} And [deps]
LocalSubPackage = "e30172f5-a6a5-5a46-863b-614d45cd2de4" Adding some CI checks and simple tooling to keep a global |
In theory, I think it could be made to work if that package is also added to |
cd8a9d9
to
78f4587
Compare
… path and repo locations for dependencies
78f4587
to
431cc2b
Compare
Just tested this out, and the use-case I had in mind seems to work great! I'm very excited, thank you for the awesome work @KristofferC. This, together with the app stuff, and pkgimages are all coalescing into a very cool story for julia code management and sharing. Regarding the questions you posed, I think I agree with @MilesCranmer's answers, they all seem reasonable. Generally, my mental model is that the Project.toml should describe what we desire, and the Manifest.toml should describe what specifically was done. So being able to describe paths, urls, and branches in the Project, and have those take priority over what is written in the Manifest seems correct and ergonomic to me. Regarding API, is there a way we can make |
I think that makes sense to do automatically because a |
The design decision here all look good to me. Thanks for this lovely feature! |
Thanks for picking this up and pushing it forward @KristofferC; sorry I let it drop/droop so long ago. I agree it will be a great feature! |
To echo, thank you for working on this! I'm very interested in whether this might simplify the 'monorepo workflow' discussed here: #3263 (comment) Am I right in thinking that a top-level, single Manifest.toml will be required in this case? IIUC, I think 'yes' because Or is there a mechanism for all of A, B & C etc. to share a single |
Thank you for working on this! I am testing using this in a name = "Foo"
uuid = "7694c9c6-b3cb-4fb3-a3a7-392d5a2739ee"
authors = ["nhz2 <nhz2@cornell.edu>"]
version = "0.1.0" In [deps]
Foo = "7694c9c6-b3cb-4fb3-a3a7-392d5a2739ee"
Test = "8dfed614-e22c-5e08-85e1-65c5234f0b40"
[sources]
Foo = {path = ".."} If I run the tests using Here is the full error.julia> Pkg.activate(;temp=true)
Activating new project at `/tmp/jl_oAEynA`
julia> Pkg.develop(path="/home/nathan/Documents/myenv/Foo")
Resolving package versions...
Updating `/tmp/jl_oAEynA/Project.toml`
[7694c9c6] + Foo v0.1.0 `~/Documents/myenv/Foo`
Updating `/tmp/jl_oAEynA/Manifest.toml`
[7694c9c6] + Foo v0.1.0 `~/Documents/myenv/Foo`
julia> Pkg.test("Foo")
Testing Foo
ERROR: could not find project file for package `Foo [7694c9c6]` at `/tmp/`
Stacktrace:
[1] pkgerror(msg::String)
@ Pkg.Types ~/github/Pkg.jl/src/Types.jl:68
[2] collect_project(pkg::Pkg.Types.PackageSpec, path::String)
@ Pkg.Operations ~/github/Pkg.jl/src/Operations.jl:253
[3] collect_fixed!(env::Pkg.Types.EnvCache, pkgs::Vector{Pkg.Types.PackageSpec}, names::Dict{Base.UUID, String})
@ Pkg.Operations ~/github/Pkg.jl/src/Operations.jl:329
[4] resolve_versions!(env::Pkg.Types.EnvCache, registries::Vector{…}, pkgs::Vector{…}, julia_version::VersionNumber, installed_only::Bool)
@ Pkg.Operations ~/github/Pkg.jl/src/Operations.jl:392
[5] up(ctx::Pkg.Types.Context, pkgs::Vector{…}, level::Pkg.Types.UpgradeLevel; skip_writing_project::Bool, preserve::Nothing)
@ Pkg.Operations ~/github/Pkg.jl/src/Operations.jl:1610
[6] up
@ ~/github/Pkg.jl/src/Operations.jl:1589 [inlined]
[7] up(ctx::Pkg.Types.Context, pkgs::Vector{…}; level::Pkg.Types.UpgradeLevel, mode::Pkg.Types.PackageMode, preserve::Nothing, update_registry::Bool, skip_writing_project::Bool, kwargs::@Kwargs{…})
@ Pkg.API ~/github/Pkg.jl/src/API.jl:381
[8] up
@ ~/github/Pkg.jl/src/API.jl:355 [inlined]
[9] up
@ ~/github/Pkg.jl/src/API.jl:163 [inlined]
[10] #resolve#143
@ ~/github/Pkg.jl/src/API.jl:387 [inlined]
[11] resolve
@ ~/github/Pkg.jl/src/API.jl:386 [inlined]
[12] (::Pkg.Operations.var"#118#123"{String, Bool, Bool, Bool, Pkg.Operations.var"#131#136"{…}, Pkg.Types.PackageSpec})()
@ Pkg.Operations ~/github/Pkg.jl/src/Operations.jl:1885
[13] with_temp_env(fn::Pkg.Operations.var"#118#123"{…}, temp_env::String)
@ Pkg.Operations ~/github/Pkg.jl/src/Operations.jl:1767
[14] (::Pkg.Operations.var"#116#121"{…})(tmp::String)
@ Pkg.Operations ~/github/Pkg.jl/src/Operations.jl:1872
[15] mktempdir(fn::Pkg.Operations.var"#116#121"{…}, parent::String; prefix::String)
@ Base.Filesystem ./file.jl:835
[16] mktempdir(fn::Function, parent::String)
@ Base.Filesystem ./file.jl:831
[17] mktempdir
@ ./file.jl:831 [inlined]
[18] #sandbox#115
@ ~/github/Pkg.jl/src/Operations.jl:1823 [inlined]
[19] test(ctx::Pkg.Types.Context, pkgs::Vector{…}; coverage::Bool, julia_args::Cmd, test_args::Cmd, test_fn::Nothing, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool)
@ Pkg.Operations ~/github/Pkg.jl/src/Operations.jl:2034
[20] test
@ ~/github/Pkg.jl/src/Operations.jl:1978 [inlined]
[21] test(ctx::Pkg.Types.Context, pkgs::Vector{…}; coverage::Bool, test_fn::Nothing, julia_args::Cmd, test_args::Cmd, force_latest_compatible_version::Bool, allow_earlier_backwards_compatible_versions::Bool, allow_reresolve::Bool, kwargs::@Kwargs{…})
@ Pkg.API ~/github/Pkg.jl/src/API.jl:474
[22] test(pkgs::Vector{Pkg.Types.PackageSpec}; io::Pkg.UnstableIO, kwargs::@Kwargs{})
@ Pkg.API ~/github/Pkg.jl/src/API.jl:158
[23] test(pkgs::Vector{Pkg.Types.PackageSpec})
@ Pkg.API ~/github/Pkg.jl/src/API.jl:147
[24] test
@ ~/github/Pkg.jl/src/API.jl:146 [inlined]
[25] test(pkg::String)
@ Pkg.API ~/github/Pkg.jl/src/API.jl:145
[26] top-level scope
@ REPL[7]:1
Some type information was truncated. Use `show(err)` to see complete types. |
But in |
[sources]
section in Project.toml for specifying relative path and repo locations for dependencies[sources]
section in Project.toml for specifying paths and repo locations for dependencies
I will fix this in an upcoming PR. |
Wow, so excited this is merged! Does this automatically show up in the "nightly" Julia release, or do I need to do something to play around with this (since Pkg is developed sort-of-independently)? |
When Pkg is bumped on nightly it will be in the nightly release. Probably easiest to wait for that. |
This is the PR to add thins automatically to |
Current nightly has these changes - this Project.toml acted as expected.
|
…d repo locations for dependencies (#3783) * Support a `[sources]` section in Project.toml for specifying relative path and repo locations for dependencies --------- Co-authored-by: Jacob Quinn <quinn.jacobd@gmail.com>
I have the feeling that the second statement a bit contradicts the first one. Project.toml describes what [in general] we want, and Manifest.toml should describe what [in the specific installation of the package] was done. So Manifest should take precedence. Main package (GenFSM) Project.toml:
But when I try to specify to use a local version of the dependant package, instead of modify Manifest.toml and use that path in my particular installation, it returns an error:
|
What do you want |
Granted, I could be completely wrong, but I don't think that |
So say you add:
and then do a |
I think the difference between |
…d repo locations for dependencies (#3783) * Support a `[sources]` section in Project.toml for specifying relative path and repo locations for dependencies --------- Co-authored-by: Jacob Quinn <quinn.jacobd@gmail.com>
Extracted and tweaked based on #3263.
Currently requires JuliaLang/julia#53233
Some questions:
add
the package with another URL? Error or update the source?path
.free
do when there is a source specified.TODO:
free
(update_package_free
)add
(update_package_add
)rm
resolve
with non source in manifest and source in project.