[feature] missing prebuilt package should have capability to provide more details

[fulara]

This is somewhat a duplicate of:
#3316 which was closed by #3438
Its better now, but maybe it oculd be slightly better?

Its still guessing game for me:

Back story:

Right now due to a various choices we have made we do not encourage our users to use --build missing whenever they see it. instead we have prebuilt binaries in our artifactory.

We are often met with an error like:
Missing prebuilt library X.

Conans error is very cryptic here, it basically does not provide any diagnostic why the available binaries you have are not compatible with your package, its always guessing game.

It would be nice if there was an option to ask conan for verbose output where it would state:

You have available prebuilt bianry with following configs:
" now prints a matrix of all prebuilt binaries for that particular version"

Whereas you require this package to have a particular set of options:
" now prints a set of option that conan searched for."

Right now what it prints is soemthing like this:

bzip2/1.0.8: WARN: Can't find a 'bzip2/1.0.8' package for the specified settings, options and dependencies:
- Settings: arch=x86_64, build_type=Debug, compiler=gcc, compiler.version=8, os=Linux, os.version=rh6
- Options: build_executable=True, fPIC=True, shared=False
- Dependencies: 
- Package ID: f610127d9b93ffcf379adc4fddaeeacc2308a891

ERROR: Missing prebuilt package for 'bzip2/1.0.8'
Try to build it from sources with "--build bzip2"
Or read "http://docs.conan.io/en/latest/faq/troubleshooting.html#error-missing-prebuilt-package"

Okay it prints the Options conan is looking for - but it does not highlight what are the available packages.
My way of solving this is to go to artifactory, and click through prebuilt binaries and compare each option by hand which particular option has not been found.
Couldnt it just dump that information straight out?
And actually I dont think its possible to list prebuilt binary options via conan CLI?

• [X ] I've read the CONTRIBUTING guide.

[memsharded]

It is possible to do a couple of things:

• You can do a conan search <pkg-ref> --table=file.html, and it will create a table with settings and options for the available binaries.
• You can also do a conan search <pkg-ref> -q "os=Windows AND compiler=gcc", and query/filter the search results.

The first is not added automatically to the output because it is too much to generate an html file and open a browser, will be more confusing to the user.

[fulara]

Thats lovely.
Its probabyl exactly what I was looking for.

Closing.

[fulara]

Hello @memsharded
I am opening this issue, I think conan should definitely have more verbose syntax.
this is the error I get:

Library/1.2@user/channel: WARN: Can't find a 'Library/1.2@library/stable' package for the specified settings, options and dependencies:
- Settings: arch=x86_64, build_type=Release, compiler=gcc, compiler.libcxx=libstdc++11, compiler.version=8, cppstd=17, os=Linux, os.version=rh7
- Options: shared=True
- Dependencies: AnotherLibrary/0.7@user/channel
- Package ID: d435ee8036bb0df9cf015e57c4d79566293f3644

ERROR: Missing prebuilt package for 'Library/1.2@user/channel'

Problem here turned out to be that i changed the default_package_id_mode. and Library was not build with this patch_mode version.

However the issue is much greater than that,
You pointed me towards conan search.
In cases where the problem is with binary compatibility across the tree - conan search advice is not a very good one - as problem is with versioning_schema mismatch rather than some options mismatch.

This is basically what I am doing every other day ( as we publish binaries into artifactory. ) Every other day I see missing prebuilt and conan is not good at all at telling me exactly why packages were not mismatch.

Mind you I am now very very good at guessing the mismatch rootcause - however if anyone else would see these issues it would take them hours of frustration to understand what the hell is going on.

I think conan should try to analyze available prebuild binaries and hint why its not wroking.
In worst case it should hint what commands will guide the user to find the rootcause.
linking to troubleshooting is just not good enough, Especially as it just tells the user to just 'build' missing tree, rather than instructing on how to find it.

Consider below example:

$ cat liba/conanfile:
from conans import ConanFile, tools

class Conan(ConanFile):
    name = "liba"
    pass

$cat libb/conanfile.py 
from conans import ConanFile, tools

class Conan(ConanFile):
    name = "libb"
    requires="liba/1.0"
    pass

$ cat libc/conanfile.py 
from conans import ConanFile, tools

class Conan(ConanFile):
    name="libc"
    requires="libb/1.0", "liba/1.1"

using global_package_id_mode patch_mode will error with:

libb/1.0: WARN: Can't find a 'libb/1.0' package for the specified settings, options and dependencies:
- Settings: 
- Options: 
- Dependencies: liba/1.1
- Package ID: 96f50b175e9b178f3e456b660ca2f11dbea3b6ef

ERROR: Missing prebuilt package for 'libb/1.0'
Try to build it from sources with "--build libb"
Or read "http://docs.conan.io/en/latest/faq/troubleshooting.html#error-missing-prebuilt-package"

Of course this is a trivial example, only dependencies has any entries.
I think it would be possible for conan to iterate over prebuilt packages (even if hidden behind a flag) and print exactly why each prebuilt package is not compatible.

[memsharded]

@uilianries

I would try to add in:

    def _raise_missing(self, missing):
        ...
        dependencies = [str(dep.dst) for dep in node.dependencies]
        raise_package_not_found_error(conanfile, ref, package_id, dependencies,
                                      out=conanfile.output, recorder=self._recorder)

• Do not modify the current information
• Add a new displayed field, that lists the information from the ConanInfo.requires, which is the one used to compute the hash.

I think that might alleviate a bit this pain.

[fulara]

Hm, not sure how exactly would that work but.
We were thinking that if you are worried about introducing a overhead on this, maybe adding a flag --debug-missing or sth and then conan would actually scan the prebuilt packages that exist and compare everything one by one providing some sort of table?

I was thinking of looking into that and contributing but would definitlely need some help.
similarly how conan search works but rather: 1st column would be: what your current setup requires and all other columns would be what you need.

[memsharded]

I was thinking of looking into that and contributing but would definitlely need some help.
similarly how conan search works but rather: 1st column would be: what your current setup requires and all other columns would be what you need.

Well, the problem is that conan search does not (and can not) handle dependencies information, for that you need the full graph evaluation, which is unfeasible. If it is only a settings and options configuration, then yes, conan search (I guess you mean the --table argument) might be improved to show more information. Probably some combination of the --query argument, that could be used to highlight cells in the html table?

What I was thinking above is to display not only the direct dependencies (requires) information, but showing all the dependencies, including transitive, in the form that they define the package ID, i.e, something like (pkgA/1.X.Y, pkgB/1.2.3@user/channel#revision, etc). I think that could help a bit to identify possible missing reasons regarding dependencies and package_id_mode.

[Marcin-Radecki]

Hi @memsharded, @uilianries, I think displaying requires in the form of package id could definitely help, but what it is important here IMHO is that in the missing prebuilt package error message we miss comparison about requires for existing packages and expected package id. By expected I mean the one that was computed at the time of graph was computed during package creation.

On the example below, translating it on language on conan commands, considering one more time error example generated from conan version based on @uilianries PR (I am using general.default_package_id_mode=patch_mode):

libb/1.0: WARN: Can't find a 'libb/1.0' package for the specified settings, options and dependencies:
- Settings:
- Options:
- Dependencies: liba/1.1
- Package ID: 96f50b175e9b178f3e456b660ca2f11dbea3b6ef
- Requirements: liba/1.1.0

ERROR: Missing prebuilt package for 'libb/1.0'
Try to build it from sources with "--build libb"
Or read "http://docs.conan.io/en/latest/faq/troubleshooting.html#error-missing-prebuilt-package"

Added string in the error Requirements: liba/1.1.0 improves things as it indicates that problem might be with package id, but it does not compare requirements with existing packages in a cache on on a remote. I think in high level we can describe this feature as --table, but for requires - I understand --table is useful compare existing packages with regards to options and settings, but it does not compare requires between packages.

Now to find out what happened, I need to compare manually using below commands:

(conan-venv) marol@marol-PC:~/conan-playground$ conan search libb/1.0@ --raw
    Package_ID: bbb6f0a470bcc35d1f65668e7de42fb11ab8c4de
        [requires]
            liba/1.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9
        Outdated from recipe: False

(conan-venv) marol@marol-PC:~/conan-playground$ conan get libb/1.0@:bbb6f0a470bcc35d1f65668e7de42fb11ab8c4de
[settings]


[requires]
    liba/1.0.0

[options]


[full_settings]


[full_requires]
    liba/1.0:5ab84d6acfe1f23c4fae0ab88f26e3a396351ac9

[full_options]


[recipe_hash]
    a4e9337f85079557e16ad3c706928400

[env]

Now I compare manually [requires] for my package in the cache bbb6f0a470bcc35d1f65668e7de42fb11ab8c4de with the one conan wants when bulding libb 96f50b175e9b178f3e456b660ca2f11dbea3b6ef: they are different since requires for the former is 1.0.0, and for the latter is 1.1.0. This comparison, done manually, leads to immediate thought that the problem is with package abi mode; had it been semver_direct_mode, the [requires] would had been 1.X.Y and 1.X.Y respectively, therefore package ids for them would had been equal, therefore there would be no missing prebuilt package mismatch in the first place.

So the question is, is such comparison would be possible to implement in conan?

[memsharded]

but it does not compare requirements with existing packages in a cache on on a remote.

This is part of the UI problem. With packages in the cache or in a remote? And if there are several remotes, in which one of them? Or in all of them? And then, what are the results? Which packages are the "closest" to compare? Because in terms of "distance", a package binary that only changes 1 option, for example shared: True->False would be definitely very close, but probably will annoy you, because that is not what you are looking for. Even if this concept of distance is somehow possible, should it display the distance to all possibilities in cache and each remote? Then distance for versions might work for semver, but OpenSSL doesn't use semver, so results might be quite useless. Same if using recipe_revision_mode, what would be the "closest" revision if that is a hash?

So from different points of views, this seems a UX and functional challenge. I think instead of comparison, some kind of highlighting in the HTML table generated with conan search --table might be the best solution. It might be even interactive. Though it is true that only contains settings/options, but not dependencies information... Any thoughts?

[fulara]

With packages in the cache or in a remote? And if there are several remotes, in which one of them? Or in all of them?

That should operate on the same level as the command itself was invoked, I got kinda lost not sure if you are thinking of conan search or install in here, but if install then follow the behavior of the install. if -r specified search only in that remote otherwise everywhere.

Which packages are the "closest" to compare? Because in terms of "distance"

In terms of distance I was always thinking of 'the number of the options that the build you requested has different from available packages. 1 difference counts as one. and sort on that, but this is an extra stuff really, just highlithing that this particular thing mismatches would be great.

... Then distance for versions ...

Not sure how distance for versions fits in here? Its either a match or it is not a match 0/1.
e.g you want to use openssl 1.0.0 (shared=False) and zlib 1.2.11 - you put them both in your requires, but openssl depends on zlib
In remote you have:
a) openssl 1.0.0 built against zlib 1.2.8 shared=True - versioning schema of zlib in openssl is full_package_mode
b) openssl 1.0.0 built against zlib 1.2.8 shared=True - versioning schema of zlib in openssl is semver-mode
c) openssl 1.0.0 built against zlib 1.2.11 shared=True - versioning schema is full_package_mode
Then the scores here would be:
a) score: -2
it uses zlib 1.2.8 but since its full_package_mode so dependency -> -1
shared=True whereas we want shared=False -> -1
b) score -1:
it uses zlib 1.2.8 but since its semver_mode then its fine -> 0
shared=True whereas we want shared=False -> -1
c) score -1
uses zlib 1.2.11 - matches -> 0
shared=True whereas we want shared=False -> -1

Unfortunatelly the conan search --table without dependency information is rarely useful.
Probably it would be enough to display the requires in the conaninfo.txt form:

    bzip2/1.0.8:76a4a7324a2083cb6964ea8321da67a1ceb31b50
    disruptor/2.Y.Z

Together with options
Now that I look at it we basically want to compare the conaninfo.txt of a build that does not exist vs a conaninfo.txt of the available packages. Thats actually what we are doing manually now - when we get build missing we go to artifactory and open every single conaninfo.txt and compare.
Ordering by the distance would be great, but not necessary so.
However conan search has the disadvantage of not displaying the setup you need. :(

[fulara]

Hello @jgsogo
Unfortuantelly the change that has been made barely makes a dent in the issue.
It does not help at all, the 'requires:' is something thats obvious.
Problem here is why conan in particular did not like any of the prebuilt packages being available.

Do you want me to write:
#6364 (comment)
into a separate issue?

Into a new issue?

[memsharded]

I think whatever is done, cannot be coupled to the current graph resolution algorithm and error message, it will be too complicated.

What I was suggesting is trying to improve over the conan search command. It might still have a few technical challenges, specially performance and a poor definition of distance that can render the feature mostly useless (or worse, an infinite source of new issues, maintenance and feature requests). One possibility was to add some interactive utilities in the HTML output of the conan search --table feature, but at the moment that only represents settings/options. But sounds to me the way to go. If this is something that you think it will have value, please open a new issue for this feature request.

[fulara]

thanks @memsharded
After some more thinking I think the changes from #6700 are good step forward and now finally it prints the full package requirements.
so: settings, options, requires,

Now I will proceed to request a feature to add requires into conan search and then both of these would print out a complete set of data.

Then in order to actually understand whats missing from available build matrix would be:

• conan install -> this would print --build missing, unfortunatelly this is human string, however parsing this wouldnt be that difficult. so parse that and store it in a file.
• conan search $(parsed_data_from_conan_install), that would give us available matrix. then matching the result of both that should be relatively straightforward.

Its too bad that conan wont provide that functionality out of the box, but maybe after this is possible, something more elegant would unveil.