Dependencies Calculation

[QwertyJack]

I found this tool very nice as the official mirroring tool cas-mirror is not free.

Currently only blacklist and whitelist is considered, not including dependencies. How can mirror the minimal packages with dependencies?

[ericdill]

Conda-mirror doesn't support this currently. You'd need to start from the packages you wanted, build the dependency graph from those packages and then pass all of those packages to the whitelist argument via the yaml config

[ericdill]

@QwertyJack if you are interested in contributing this feature, I'd be happy to help guide you in the right direction and review a PR

[analog-cbarber]

It is not clear exactly what this should mean. One possible interpretation would be to essentially run a solve for a given set of package specs and just include those packages. That would mean that you would need to run the conda solver, either through the python api or by invoking conda in another process on an empty environment and capturing the relevant details (possible using the --json) option.

Another possible use case would be to include all packages matching the specs along with any packages that match any of those packages requirements. In this case, you would have to at least be able to handle conda package specifiers, which unfortunately appears to be non-trivial (see https://github.com/conda/conda/blob/master/conda/models/version.py), so you would want to either copy that or add a dependency on conda itself (and essentially require that conda-mirror only be installed in the base repo).

[analog-cbarber]

I suppose another way to implement the second scenario would be to run conda search --json on each package and then recursively on any dependencies until you have explored the graph for that channel.

Running conda in a separate process also would allow you to use mamba instead (although I don't think that is necessarily a speed win when just doing search).

[analog-cbarber]

The conda search trick would work but would be pretty slow. You need to make sure to use --override-channels and the --use-index-cache flag also helps, but even so it takes about half a second per query on my fast macbook.

Might be ok for an initial implementation but would be much faster to directly pull in the package spec matching code.

[analog-cbarber]

If you include dependencies, then you probably also want to consider dependencies both in a specific platform subdir (e.g. linux-64) and in the noarch subdir.

[analog-cbarber]

One issue with dependencies is that sometimes they do not include any version constraint and will match any version of a target package. You may not want to include every possible match, so you would probably want some way to have constraints on what gets brought in by dependencies.

One way to do this would be to honor any version constraint from the whitelist over one from a dependency list; e.g. if your white list included python >=3.7 then even if there isa dependency like python >=3.5, it would not be used. I don't think we want to attempt to mimic a full solver so we don't want this to be too complicated.

Another possible approach would be to have an option to apply an exclusion list AFTER the inclusion/whitelist and any dependencies are applied.

[analog-cbarber]

I think that for mirroring large channels like conda-forge, you probably need a way to exclude some packages from dependency consideration.

[fhoehle]

IIUC this is resolved by #42