Conan fails to detect the compiler although `CC` and `CXX` are set #14575

kordejong · 2023-08-25T11:07:59Z

Environment details

Operating System+version: ubuntu-22.04
Compiler+version: 13.1.0
Conan version: 2.0.9
Python version: 3.11.4

Steps to reproduce

I am creating a Conda package of my software, as part of a Github C/I workflow. My software depends on other software (e.g.: Boost). For dependencies for which there isn't a Conda package available I use Conan to install them.

Here is the idea, scattered over the various files (.github/workflows/linux-conda.yml, environment/conda/build.sh):

Create a Conda environment
Install the boa Conda package builder and execute: conda mambabuild environment/conda --override-channels --channel conda-forge

Then build.sh is executed which includes:

conan profile detect                                                                                             
conan install . \                                                           
    --profile=default \                                                                                          
    --settings:host compiler.cppstd=17 \                                                                         
    --settings:build compiler.cppstd=17 \                                                                        
    --settings:build build_type=Release \                                                                        
    --build=missing \                                                                                            
    --output-folder=build

This results in an error (see log below). Conan is not able to correctly detect the compiler. The CC and CXX variables are set, so I would expect that those are used to detect the compiler.

I tried passing in the compiler explicitly, when calling conan install, using --setting compiler=$CXX, but that failed with ERROR: 'settings.compiler' value not defined. Maybe because the whole compiler section is missing from the profile?

I think this is a bug, given that the compiler is installed and pointed to by the standard variables. If not, then my question is how to get Conan to detect the compiler correctly in this situation.

I use the same conan commands to install Conan packages outside of the Conda build environment (on Linux, macOS, Windows), and those work fine. It seems that the Conda build environment is confusing Conan. Maybe because of the atypical setting of CC and CXX?

Logs

2023-08-24T16:10:09.7983374Z CC and CXX: /usr/share/miniconda/envs/test/conda-bld/lue_1692893106258/_build_env/bin/x86_64-conda-linux-gnu-cc, /usr/share/miniconda/envs/test/conda-bld/lue_1692893106258/_build_env/bin/x86_64-conda-linux-gnu-c++ 
2023-08-24T16:10:09.7984827Z ERROR: Not able to automatically detect '/usr/share/miniconda/envs/test/conda-bld/lue_1692893106258/_build_env/bin/x86_64-conda-linux-gnu-cc' version
2023-08-24T16:10:09.7985898Z No compiler was detected (one may not be needed)
2023-08-24T16:10:09.7986712Z WARN: This profile is a guess of your environment, please check it.
2023-08-24T16:10:09.7987819Z WARN: The output of this command is not guaranteed to be stable and can change in future Conan versions.
2023-08-24T16:10:09.7988301Z WARN: Use your own profile files for stability.
2023-08-24T16:10:09.7988673Z Saving detected profile to /home/runner/.conan2/profiles/default
2023-08-24T16:10:09.8382941Z + conan install . --profile=default --settings:host compiler.cppstd=17 --settings:build compiler.cppstd=17 --settings:build build_type=Release --build=missing --output-folder=build
2023-08-24T16:10:10.1836158Z ERROR: 'settings.compiler' value not defined

Somewhat later on in the log:

2023-08-24T16:10:10.3898835Z Detected profile:
2023-08-24T16:10:10.3899230Z [settings]
2023-08-24T16:10:10.3899457Z arch=x86_64
2023-08-24T16:10:10.3899668Z build_type=Release
2023-08-24T16:10:10.3900101Z os=Linux

The text was updated successfully, but these errors were encountered:

memsharded · 2023-08-25T11:18:20Z

Hi @kordejong

It seems that the command <compiler> --dumpversion is failing in this setup. Could you please execute the command and paste here the output?

The relevant code is:

ret, out = detect_runner('%s -dumpversion' % compiler_exe)
if ret != 0:
    return None
compiler = "gcc"
installed_version = re.search(r"([0-9]+(\.[0-9])?)", out).group()

In any case, the right approach is not to rely on the auto-detect, not pass it in the settings, but define the input -s compiler.version=7 right version in your command (better put it in a profile file than use the command line), and forget about the conan autodetect profile. As suggested in the docs, the output of this command should not be trusted in production.

kordejong · 2023-08-25T12:40:33Z

Hi @memsharded,

Thanks for the feedback!

It seems that the command <compiler> --dumpversion is failing in this setup. Could you please execute the command and paste here the output?

The output is: 13.1.0

In any case, the right approach is not to rely on the auto-detect, not pass it in the settings, but define the input -s compiler.version=7 right version in your command (better put it in a profile file than use the command line), and forget about the conan autodetect profile. As suggested in the docs, the output of this command should not be trusted in production.

I understand, but I don't control the Conda build environment. The compiler version can change at any time, for example. My current strategy is to call conan detect and then pass knowledge I do know (compiler.cppstd, compiler.build_type) to conan install. I think the auto-detection of the environment is useful in this case.

If I have to I can add logic to my build.sh to write a profile, like you suggest. The Conda build environment has various variables I can use for that. I would be replicating some of the logic from conan profile detect though ;-) .

memsharded · 2023-08-25T14:49:56Z

The output is: 13.1.0

Uhm, I was expecting something different. Is this the whole output? No other characters or text? It is output to stdout or stderr?
It seems the code above works for this return, and the regex is valid, installed_version should be "13.1", not sure why it is failing for your case

Do you know how to run Conan from source in that environment? Adding a few prints() in that code might give some more hints.

kordejong · 2023-08-26T14:38:32Z

I will try to find out what is going on.

GDelevoye · 2023-10-29T19:20:54Z

I have the same problem. The autodetect is broken within a conda environment (Ubuntu LTS 22.04, x86) with the cxx-compiler package (this is a generic way to ensure that the end-user has a modern C++ compiler installed even if they don't have any sudo access on the system)

My settings :

Conan version 2.0.13
Ubuntu LTS 22.04, x86
conda 1.4.2
mamba 23.3.1
gcc (installed via mamba cxx-compiler) : 12.3.0-2

In a previous project , I used conan 1.59 with autodetect and it worked fine within conda/cxx-compiler (still works today), which was really great. But today I tried to modernize my build toolchain and it fails with conan v2.0

The workaround that I found was to check my gcc versionmanually and edit the ~/.conan/profiles/XXXX manually to specify the right parameters by hand like :

[settings]
arch=x86_64
build_type=Release
compiler=gcc # This was missing
compiler.cppstd=gnu20 # This was missing
compiler.libcxx=libstdc++11 # This was missing
compiler.version=12 # This was missing
os=Linux # This was there

Then it worked.

I know and understand that the autodetect is not necessarily 100% reliable on all systems, and I truly understand that but I still really upvote this issue. If it can work on Ubuntu, Windows and MacOS with conda, that would be really great for the scientific community. The end-users must often install the software on clusters, remote servers, or professional laptops where they don't have any sudo access; nor necessarly a modern compiler installed, and many of them are not programmers. Developpers, on the other hand, are often single +/- isolated programmers who want to broadcast their software quick&dirty with conda, AND use conan to spare themselves the nightmare of the dependencies. Using conda+conan allows to bypass all these issues very easily for everyone

memsharded · 2023-10-29T22:46:44Z

Hi @GDelevoye

Thanks for the feedback.
We are definitely wanting to try to fix it and make it work. It would be great to have some help and understand why it would be failing in those environment. The current code to detect compiler is:

def _gcc_compiler(compiler_exe="gcc"):
        if platform.system() == "Darwin":
            # In Mac OS X check if gcc is a fronted using apple-clang
            _, out = detect_runner("%s --version" % compiler_exe)
            out = out.lower()
            if "clang" in out:
                return None, None

        ret, out = detect_runner('%s -dumpversion' % compiler_exe)
        if ret != 0:
            return None, None
        compiler = "gcc"
        installed_version = re.search(r"([0-9]+(\.[0-9])?)", out).group()
        # Since GCC 7.1, -dumpversion return the major version number
        # only ("7"). We must use -dumpfullversion to get the full version
        # number ("7.1.1").
        if installed_version:
            ConanOutput(scope="detect_api").info("Found %s %s" % (compiler, installed_version))
            return compiler, Version(installed_version)

What is the output of running gcc -dumpversion in your setup?

Never heard back from @kordejong, did you manage to check what was happening in your setup?

kordejong · 2023-10-30T08:29:37Z

Never heard back from @kordejong, did you manage to check what was happening in your setup?

No, still on my todo list to look into this, but flooded with other stuff right now. I wrote a script which writes a profile, replicating some of Conan's autodectect logic.

GDelevoye · 2023-11-03T07:01:28Z

@memsharded Here is the dumpversion on my system :

(base) delevoye@Dell-G15-5530:~$ which gcc
/home/delevoye/miniforge3/bin/gcc
(base) delevoye@Dell-G15-5530:~$ gcc -dumpversion
12.3.0
(base) delevoye@Dell-G15-5530:~$ conda deactivate
delevoye@Dell-G15-5530:~$ which gcc
/usr/bin/gcc
delevoye@Dell-G15-5530:~$ gcc -dumpversion
11
delevoye@Dell-G15-5530:~$ gcc -dumpfullversion
11.4.0
delevoye@Dell-G15-5530:~$

Regarding CC and CXX

(base) delevoye@Dell-G15-5530:~$ echo $CC$
/home/delevoye/miniforge3/bin/x86_64-conda-linux-gnu-cc$
(base) delevoye@Dell-G15-5530:~$ echo $CXX$
/home/delevoye/miniforge3/bin/x86_64-conda-linux-gnu-c++$
(base) delevoye@Dell-G15-5530:~$ conda deactivate
delevoye@Dell-G15-5530:~$ echo $CC$
$
delevoye@Dell-G15-5530:~$ echo $CXX$
$

Whereis/Which

(base) delevoye@Dell-G15-5530:~$ whereis gcc
gcc: /usr/bin/gcc /usr/lib/gcc /usr/share/gcc /home/delevoye/miniforge3/bin/gcc /usr/share/man/man1/gcc.1.gz
(base) delevoye@Dell-G15-5530:~$ echo $CX$
$
(base) delevoye@Dell-G15-5530:~$ echo $CXX$
/home/delevoye/miniforge3/bin/x86_64-conda-linux-gnu-c++$
(base) delevoye@Dell-G15-5530:~$ conda deactivate
delevoye@Dell-G15-5530:~$ whereis gcc
gcc: /usr/bin/gcc /usr/lib/gcc /usr/share/gcc /usr/share/man/man1/gcc.1.gz

GDelevoye · 2023-11-03T07:04:32Z

Maybe what I can suggest here is to always use fulldumpversion, and do something without a regular expression like :

stdout_gcc = my_magic_function_dumpfullversion()
major_version = stdout_gcc.split('.')[0] # Should work with both long and and short versions
if major_version:
    try:
        minor_version = ".".join(stdout_gcc.split('.')[1:])
    except IndexError:
        minor_version = 0

GDelevoye · 2023-11-03T07:22:32Z

Note also that the conda environment uses an abstraction layer with symlinks as was visible in OP

2023-08-24T16:10:09.7984827Z ERROR: Not able to automatically detect '/usr/share/miniconda/envs/test/conda-bld/lue_1692893106258/_build_env/bin/x86_64-conda-linux-gnu-cc' version

I have no idea how much this symlink system varies between windows, Ubuntu and MacOS for conda, and I have no idea of how much it impacts the way you autodetect

memsharded · 2023-11-03T10:22:19Z

Maybe what I can suggest here is to always use fulldumpversion,

There are flavors of gcc that don't support -dumpfullversion, we found that -dumpversion was the most common one.

memsharded · 2023-11-03T10:26:32Z

Regarding the CC, CXX variables the code is:

 cc = os.environ.get("CC", "")
    cxx = os.environ.get("CXX", "")
    if cc or cxx:  # Env defined, use them
        output.info("CC and CXX: %s, %s " % (cc or "None", cxx or "None"))
        command = cc or cxx

So it seems you are checking the wrong env-var CX is incorrect, need to check CC.
Then it would be good to know what would be the output of:

/usr/share/miniconda/envs/test/conda-bld/lue_1692893106258/_build_env/bin/x86_64-conda-linux-gnu-cc -dumpversion

I think that is the one that the detect is failing to parse. So far it seems that the detection process is correctly using that compiler from the environment, calling it, and just failing to understand its output.

GDelevoye · 2023-11-03T10:35:58Z

There are flavors of gcc that don't support -dumpfullversion, we found that -dumpversion was the most common one.

this SO post suggests an easy way around it that works for older versions too

gcc -dumpfullversion -dumpversion

So it seems you are checking the wrong env-var CX is incorrect, need to check CC.

I changed CX by CC in my previous post and I replaced the output with the correct result

Then it would be good to know what would be the output of:
   /usr/share/miniconda/envs/test/conda-bld/lue_1692893106258/_build_env/bin/x86_64-conda-linux-gnu-cc --dumpversion

Maybe you meant a single dash instead of double dash ?

If I try both :

(base) delevoye@Dell-G15-5530:~$ whereis x86_64-conda-linux-gnu-cc
x86_64-conda-linux-gnu-cc: /home/delevoye/miniforge3/bin/x86_64-conda-linux-gnu-cc
(base) delevoye@Dell-G15-5530:~$ x86_64-conda-linux-gnu-cc -dumpversion
12.3.0
(base) delevoye@Dell-G15-5530:~$ x86_64-conda-linux-gnu-cc --dumpversion
x86_64-conda-linux-gnu-cc: error: unrecognized command-line option '--dumpversion'; did you mean '-dumpversion'?
x86_64-conda-linux-gnu-cc: fatal error: no input files
compilation terminated.
(base) delevoye@Dell-G15-5530:~$ x86_64-conda-linux-gnu-cc -dumpfullversion
12.3.0
(base) delevoye@Dell-G15-5530:~$ x86_64-conda-linux-gnu-cc --dumpfullversion
x86_64-conda-linux-gnu-cc: error: unrecognized command-line option '--dumpfullversion'; did you mean '-dumpfullversion'?
x86_64-conda-linux-gnu-cc: fatal error: no input files
compilation terminated.

memsharded · 2023-11-03T10:57:09Z

Maybe you meant a single dash instead of double dash ?

Yes, definitely, my mistake, fixed.

memsharded · 2023-11-03T11:04:04Z

I cannot see why 12.3.0 would fail to be parsed. I am trying to understand the possible source. I assume that it is printed to stdout and not stderr, is this correct?

A this point, the best option could be to add some prints() to the detect function:

def _gcc_compiler(compiler_exe="gcc"):
    try:
        if platform.system() == "Darwin":
            # In Mac OS X check if gcc is a fronted using apple-clang
            _, out = detect_runner("%s --version" % compiler_exe)
            out = out.lower()
            if "clang" in out:
                return None, None

        ret, out = detect_runner('%s -dumpversion' % compiler_exe)
        if ret != 0:
            return None, None
        compiler = "gcc"
        installed_version = re.search(r"([0-9]+(\.[0-9])?)", out).group()
        # Since GCC 7.1, -dumpversion return the major version number
        # only ("7"). We must use -dumpfullversion to get the full version
        # number ("7.1.1").
        if installed_version:
            ConanOutput(scope="detect_api").info("Found %s %s" % (compiler, installed_version))
            return compiler, Version(installed_version)
    except (Exception,):  # to disable broad-except
        return None, None

And debug what is happening. If you need guidance or help, I might be able to connect like right now or on monday. I am also available in the CppLang slack, so we could discuss and debug it live, that might be the fastest way to sort this out.

memsharded · 2024-01-10T10:59:14Z

Hi @GDelevoye

Did you manage to add those prints and check what is happening? That would help to move and fix this.

hansenms · 2024-02-13T20:57:35Z

@memsharded I think the problem is this logic here:

command = cc or cxx
# ...
if "gcc" in command or "g++" in command or "c++" in command:
  gcc, gcc_version, compiler_exe = _gcc_compiler(command)
  if platform.system() == "Darwin" and gcc is None:
      output.error("%s detected as a frontend using apple-clang. "
                   "Compiler not supported" % command)
  return gcc, gcc_version, compiler_exe

In a conda environment, the CC and CXX variables point to /opt/conda/envs/<env name>/bin/x86_64-conda-linux-gnu-cc and /opt/conda/envs/<env name>/bin/x86_64-conda-linux-gnu-c++ and command ends up being /opt/conda/envs/<env name>/bin/x86_64-conda-linux-gnu-cc and thus _gcc_compiler(command) is never called.

A possible fix is:

if "cc" in command or "g++" in command or "c++" in command:
    # ...

I put up a PR as an example.

memsharded self-assigned this Aug 25, 2023

memsharded added the stage: triaging label Aug 25, 2023

This was referenced Feb 13, 2024

Switched to cc pattern in compiler detection #15664

Merged

Conan Build ismrmrd/ismrmrd#247

Draft

memsharded closed this as completed in #15664 Feb 15, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Conan fails to detect the compiler although `CC` and `CXX` are set #14575

Conan fails to detect the compiler although `CC` and `CXX` are set #14575

kordejong commented Aug 25, 2023 •

edited

Loading

memsharded commented Aug 25, 2023

kordejong commented Aug 25, 2023 •

edited

Loading

memsharded commented Aug 25, 2023 •

edited

Loading

kordejong commented Aug 26, 2023

GDelevoye commented Oct 29, 2023 •

edited

Loading

memsharded commented Oct 29, 2023

kordejong commented Oct 30, 2023

GDelevoye commented Nov 3, 2023 •

edited

Loading

GDelevoye commented Nov 3, 2023 •

edited

Loading

GDelevoye commented Nov 3, 2023

memsharded commented Nov 3, 2023 •

edited

Loading

memsharded commented Nov 3, 2023 •

edited

Loading

GDelevoye commented Nov 3, 2023 •

edited

Loading

memsharded commented Nov 3, 2023

memsharded commented Nov 3, 2023

memsharded commented Jan 10, 2024

hansenms commented Feb 13, 2024 •

edited

Loading

Conan fails to detect the compiler although CC and CXX are set #14575

Conan fails to detect the compiler although CC and CXX are set #14575

Comments

kordejong commented Aug 25, 2023 • edited Loading

Environment details

Steps to reproduce

Logs

memsharded commented Aug 25, 2023

kordejong commented Aug 25, 2023 • edited Loading

memsharded commented Aug 25, 2023 • edited Loading

kordejong commented Aug 26, 2023

GDelevoye commented Oct 29, 2023 • edited Loading

memsharded commented Oct 29, 2023

kordejong commented Oct 30, 2023

GDelevoye commented Nov 3, 2023 • edited Loading

GDelevoye commented Nov 3, 2023 • edited Loading

GDelevoye commented Nov 3, 2023

memsharded commented Nov 3, 2023 • edited Loading

memsharded commented Nov 3, 2023 • edited Loading

GDelevoye commented Nov 3, 2023 • edited Loading

memsharded commented Nov 3, 2023

memsharded commented Nov 3, 2023

memsharded commented Jan 10, 2024

hansenms commented Feb 13, 2024 • edited Loading

Conan fails to detect the compiler although `CC` and `CXX` are set #14575

Conan fails to detect the compiler although `CC` and `CXX` are set #14575

kordejong commented Aug 25, 2023 •

edited

Loading

kordejong commented Aug 25, 2023 •

edited

Loading

memsharded commented Aug 25, 2023 •

edited

Loading

GDelevoye commented Oct 29, 2023 •

edited

Loading

GDelevoye commented Nov 3, 2023 •

edited

Loading

GDelevoye commented Nov 3, 2023 •

edited

Loading

memsharded commented Nov 3, 2023 •

edited

Loading

memsharded commented Nov 3, 2023 •

edited

Loading

GDelevoye commented Nov 3, 2023 •

edited

Loading

hansenms commented Feb 13, 2024 •

edited

Loading