enable query for a package / profile combination, local and/or remote

[a4z]

I would like to be able to query if a package for a given profile (and option set) exists in the local cache, or remote, or if it would be needed to be built.

Currently I can do this for the local cache via some complex workarounds, but it should be simple doable via conan search -pr profilename package/name@ + remote option if remote search is wanted.
optimally, this should return something usable also from helper scriptes

• I've read the CONTRIBUTING guide.

[memsharded]

Hi @a4z

The problem is that there is no 1:1 match between profiles and package-ids. There are many things that affect the computation of a package-id, from the recipe package_id() method (with settings and options erasure, or the compatible_packages mechanism), to options definitions from overrides downstream, from versions of the dependencies (that are not defined in a profile, and can also be overridden from downstream)

What we are doing to alleviate the problem of missing binaries is improving the html output table of the conan search <pkg-ref> --table=file.html, to make it interactive and be it a tool to search, filter, navigate and analyze binaries.

This is already being implemented in #6832

[a4z]

this is not the same
a profile contains build requirements, like cmake, and options.
as it seems, there is no other way to find out which of my packages have already been rebuilt with a new build requirement in a profile, like cmake 3.17.0, and which not
without using a package with a profile and wait for an error message if it does not exist, or something like that.

[memsharded]

Build-requirements do NOT affect at all to the package-id. You can add them, remove them, change the version, and still the final package ID is the same, so it will not tell you if you need to build that package or not.

There are plans to change this, but at the moment, it behaves this way.

[a4z]

ok, then something different

 conan search  zlib/1.2.11@ --json search.json
Existing packages for recipe zlib/1.2.11:

    Package_ID: 3fb49604f9c2f729b85ba3115852006824e72cab
        [options]
            minizip: False
            shared: False
        [settings]
            arch: x86_64
            build_type: Release
            compiler: Visual Studio
            compiler.runtime: MD
            compiler.version: 16
            os: Windows
        Outdated from recipe: False

    Package_ID: d057732059ea44a47760900cb5e4855d2bea8714
        [options]
            minizip: False
            shared: False
        [settings]
            arch: x86_64
            build_type: Debug
            compiler: Visual Studio
            compiler.runtime: MDd
            compiler.version: 16
            os: Windows
        Outdated from recipe: False

but the profile has more, especially the c++ standard

[settings]
os=Windows
arch=x86_64
arch_build=x86_64
compiler=Visual Studio
compiler.version=16
compiler.cppstd=17
build_type=Debug
os_build=Windows

any chance this will be added in the output?
still need to check if this can be queried .... but the query language seems somehow difficult, compared with just passing a profile

[memsharded]

The above packages do not show compiler.cppstd in the search, because they were created without it (or with compiler.cppstd=None).
If you inject the above profile into that search, it will return no binary packages, and this is expected, because there are no zlib packages built with compiler.cppstd=17 (in ConanCenter)

This is a search that you can do today with the --query argument in conan search <pkg-ref> --query="compiler=gcc and compiler.cppstd=17 and ...", but still that will not help much, because it will tell you there are no binaries.

The functionality being implemented will be more intuitive and will allow debugging, because you can start filtering by OS, compiler, etc, and narrow the search, and then realize that the discrepancy of the compiler.cppstd is the one that is causing the "missing binary".

Another separate issue is the current treatment of compiler.cppstd in ConanCenter, this is something we are currently investigating, because the current approach (build binaries only for the default cppstd (compiler.cppstd=None) of the compiler) has some limitations.

But the functionality requested will better be addressed with this html search and filtering. As a perfect match or a very good distance measure can't be easily defined, because of this lack of 1:1 correspondence, it is much better to provide some useful "debugging" functionality instead, and this is our plan.

[a4z]

so the best I can do is take a profile, translate it to a query string , and use this for conan serach.
But I need to know internals of that package, if its C , cppstd doe not matter and will give me no result, like for zlib , while if its a CPP project, I need to pass in this option....

this does not seem to help me for e.g. automation of some things
say I have a list of packages, and I want to rebuild everything, just to be sure they all exist for a given profile on the remote all our users should use, but if a package exists in the local cache with this profle, just skip the build and do not call conan create .
How could I do that without saying:
hey conan, do you have a package foo that was build with profile bar?
if it exists, fine, upload foo, if not , build it and then upload

because this means, if I change the cppstandard from 14 to 17, zlib. openssl, .. would not needed to be rebuilt, but other packages that make use of the property do. and this might also be the case for other settings I do in profiles.
and this would accelerate my builds, but I can not query that
I need to do workarounds from which I do not exactly know how they look...

a related question, is it possible to get a package id of a package with a profile without building it ?
this would possible allow me the lookup I would like to do

some more context, I have due to cross compiling and host + target confige 4 + 4 + 4 + 4 + 2 + 2 + 2 builds for each of my packages
this is a lot of build time
if I do a change a profile, and I do not need to rebuild some of them, or all, it would be good to know
how can I have that if I can not say, hey conan, package foo for profile bar, is it in my local cache, or in my remote?

[Minimonium]

In my opinion, the table solution is a bad UX. It's multistep and pain with cli.
The conan search reference -pr=profile should look up the recipe, perform package_id with a given profile and print all found hashes (including compatible). In case there are no hashes found - the command should return an error, it would make it more useful for automation cases.

[memsharded]

conan search reference -pr=profile should look up the recipe, perform package_id

This is a small UX improvement, and most likely not enough. Finding this is as "easy" as doing the above query, converting the profile to a -query="compiler=gcc and compiler.version=..., etc. As explained above, in many cases it will show no binaries. You start this search because you are missing a binary. If you provide the profile as the input, the logical result is that there are no binaries. Another bad thing is that also the profile doesn't have a 1:1 correspondence with the binary that you are missing. When you are missing some binary of a package in your dependency graph, it might be because of requirements revision (if you are using a different package_revision_mode), or maybe some of the consumers did set an option for this package. It is not even about good or bad UX, doing a conan search reference -pr=profile will actually make impossible to find what is happening, because you are looking in the wrong place to start with.

What makes sense is to take the output of the "package missing" exception error, and start filtering all the binaries for that package based on that values. There is no single universal approach to find what is happening, sometimes it might be an option, sometimes it might be the version of a requirement that affects the package_id. The only way to have something remotely useful is to allow users to filter, order and "debug" what is happening in an interactive and visual way.

[memsharded]

Hi @a4z

this does not seem to help me for e.g. automation of some things
say I have a list of packages, and I want to rebuild everything, just to be sure they all exist for a given profile on the remote all our users should use, but if a package exists in the local cache with this profle, just skip the build and do not call conan create .
How could I do that without saying:
hey conan, do you have a package foo that was build with profile bar?
if it exists, fine, upload foo, if not , build it and then upload

At least from our experience with many other users this might not be the optimal way to think about this problem. When CI kicks in, first you have the cache empty. You cannot use the same cache for different CI jobs, because the Conan cache is not concurrent. So caches are populated from scratch when you start working. It is not that you have a package in the local cache for that profile.

Also, you only build when code changes, and when there are new things to build. Not having binaries is an error in the pipeline, and CI jobs shouldn't be building binaries other than the one that is changing the source code. This is the reason Conan fails by default when binaries are missing, instead of --build=missing them by default. Because that is the path to silently skip errors in the pipeline, and re-build things many more times than necessary without realizing.

The "why this binary is missing" shouldn't be part of the automated CI. This should only happen if something goes wrong, and then you need human intervention to investigate why. The tools we are implementing for this are aimed for this use case, not for automation.

because this means, if I change the cppstandard from 14 to 17, zlib. openssl, .. would not needed to be rebuilt, but other packages that make use of the property do. and this might also be the case for other settings I do in profiles.
and this would accelerate my builds, but I can not query that
I need to do workarounds from which I do not exactly know how they look...

If you are creating new profiles to build your products with, that typically involves launching a "products" build, that specifically takes this new profile and launches a build for packages to create binaries for them.

This is the same logic, lets say, as if you depend on boost, and you plan to use a new boost version. You don't implement the logic of "I check if the new version package is there, and if not, I create it". If someone requires the new version of boost, and that package is not already existing there, it is an error. The package should be created first. Otherwise, it is very easy to have different process to be building in parallel and uploading simultaneously boost (probably causing different package revisions unnecessarily). The same logic should be applied for binaries for different configurations.

[a4z]

you are talking about CI, parallel build and upload, but not about the developer who creates such things and who might have restricted resources.
If I spare 22 builds for each of zlib, openssl and libcurl, and 2 or 3 others, because they do not need to be rebuild when something changes in the profile, than this would help me.

I have a list of 15 packages,
I change something different in a profile
I want to know which packages need to be rebuild
I want to have this as a list before I start any building

then I might rebuild everything in order that needs to be build new with the new profiles
and upload them to artifactory others start then using these packages, I have no idea about their apps

once this works for me on my limited machine, I put it into CI system which has also not endless resources available, so it does not spend useless time in building packages that do not need to be rebuild, and block resources other are waiting for.
(please note the order, I have already users using the packages before I put them into CI)

this is my mental picture at the moment how it should work for me in my situation

and yes, I know about --build=missing, but this will not give me the list that I would like to have.
it will 'just work', and let me with a local cache from which I do not know which package has newly been built, and which has been reused

but what I understand it the following,
you have no idea how to put this functionality into conan, or it might be to expensive to add, so you argue against it.

so I will start writing queries for the list of packages I have, with the internal knowledge I have, producing something that works for me, because this might be faster than discussing usecases here.

[memsharded]

but what I understand it the following, you have no idea how to put this functionality into conan, or it might be to expensive to add, so you argue against it.

Yes, that is partially true. I definitely know how to add the above conan search -pr=profile command, but I am completely sure that this will be just digging deeper and causing technical debt, without really solving the issue. This will not help to solve your problem. When something is not feasible, or we don't have the resources, or it is computationally too expensive, I say so. In this case, it is more that that approach will really not alleviate the real pain.

I have a list of 15 packages,
I change something different in a profile
I want to know which packages need to be rebuild
I want to have this as a list before I start any building

There are mechanisms in Conan to do that probably way better than with searches. Basically:

• conan graph lock . --profile=mynewprofile --build captures the graph
• conan graph build-order . --build=missing will tell you what to build, and specifically in what order, which is something really hard to do via conan search. It will not put in that list packages that you already have in the cache or even remotely, and will take into account the whole graph, the package_id() of the recipes, the default_package_id_mode, and everything that is needed to know accurately what needs to be built and what not.

Have you had a look at those commands? I think they can be useful for your use case. I am not here to argue to avoid doing things, I am really trying to help.

[a4z]

this means, I would need to add a temp/fake consumer project, consuming the packages I build
install them for this project, and use the graph file and work with this?

I can have a look at that, this basically falls under the work around way, but if it is not too complex and allows me to extract the info I need more easy than currently, I can give it a trial
will have to test/check that approach, havent looked too deep into the graph command so far, still somehow new, but this is now a good reason to do so.
Thanks for the hint!

[memsharded]

this means, I would need to add a temp/fake consumer project, consuming the packages I build
install them for this project, and use the graph file and work with this?

The graph lock command can work without actually installing the heavy binaries, only the recipes. Typically, it can be good enough to have a plain conanfile.txt with the bottom consumer packages, and run the conan graph lock over that conanfile.txt, and the dependencies will be automatically managed.

Otherwise you would need to encode somewhere the full dependency graph to be able to iterate the packages in the right order. If the graph grows too large, or it is dynamic, this can be cumbersome and error prone.

We call it the "products" approach. You could have different conanfile.txt for different products. Now you do a change in one profile, and you want to have binaries for that profile for a subset of your packages, but not all. Then you fire the build with that profile with one conanfile.txt and not the other products one.

Let me know how it goes with this approach. Thanks!

[a4z]

finally had time to explore the graph command. This is for sure some help, even if I do not understand why it is so hard to get a package id for a profile + package combination.

so what about adding a command

conan id -pr profilename -n name/1.2.3@
or
conan id -pr profilename -n name/1.2.3@ /path/to/folder
if the recipe is just local in my dev folder

the id is used to identify packages and also for some functionality, like conan upload ... , conan

and with the id I can have my search workaround more easy then now :-)

[a4z]

I am not sure why this is now closed, can I get now the package id, or cache location, via some command ?
this does not feel at all like the feature I requested or would need

given a package name and a profile
I can query if the package exists local (or remote)

and if it exists local

given a package name and a profile
I can enough info to get the path in the file system where the package is

is this now all the sudden possible? through an improved HTML output ?
without workaround by generating a graph .... ?

[memsharded]

the new conan graph explain --requires=pkg/version -pr=myprofile searches for binaries in cache and remotes, looking for the closest match, reporting if it finds an exact match or what are the differences of the existing binaries. Closing this tickets as solved by #14694