Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[poc] Add lapack/3.10.1 #15556

Closed
wants to merge 3 commits into from
Closed

Conversation

samuel-emrys
Copy link
Contributor

@samuel-emrys samuel-emrys commented Jan 30, 2023

Specify library name and version: lapack/3.10.1

Initial proof of concept lapack recipe implementation.

This is a draft to illustrate the requirement to model internal gcc libraries, as presented in conan-io/conan#15128. This is a blocker for this recipe.

An additional blocker for this recipe is that it will fail to build when the user profile defines CC, CXX or tools.build:compiler_executables. This results in the following error:

lapack/3.10.1@user/testing: CMake command: cmake -G "Unix Makefiles" -DCMAKE_TOOLCHAIN_FILE="/home/user/.conan/data/lapack/3.10.1/user/testing/build/fd2230a7a63c79d2e75d3f38ce3ae3cd07a0705c/build/Release/generators/conan_toolchain.cmake" -DCMAKE_INSTALL_PRE[5762/46341]
owlings/.conan/data/lapack/3.10.1/user/testing/package/fd2230a7a63c79d2e75d3f38ce3ae3cd07a0705c" -DCMAKE_GNUtoMS="OFF" -DBUILD_TESTING="OFF" -DBUILD_SHARED_LIBS="ON" -DLAPACKE="ON" -DCBLAS="ON" -DCMAKE_POLICY_DEFAULT_CMP0091="NEW" -DCMAKE_BUILD_TYPE="Release" "/home/user/.
conan/data/lapack/3.10.1/user/testing/build/fd2230a7a63c79d2e75d3f38ce3ae3cd07a0705c/."                                                                                                                                                                                              
-- Using Conan toolchain: /home/user/.conan/data/lapack/3.10.1/user/testing/build/fd2230a7a63c79d2e75d3f38ce3ae3cd07a0705c/build/Release/generators/conan_toolchain.cmake                                                                                                        
-- Conan toolchain: Setting BUILD_SHARED_LIBS = ON                                                                                                                                                                                                                                       
-- The Fortran compiler identification is GNU 12.2.0                                                                                                                                                                                                                                     
-- The C compiler identification is Clang 15.0.7                                                                                                                                                                                                                                         
-- Detecting Fortran compiler ABI info                                                                                                                                                                                                                                                   
-- Detecting Fortran compiler ABI info - done                                                                                                                                                                                                                                            
-- Check for working Fortran compiler: /home/user/.conan/data/gcc/12.2.0/user/testing/package/e22f47f196f50cff15626bfbdc57fadd84b7f986/bin/gfortran-12.2.0 - skipped                                                                                                             
-- Detecting C compiler ABI info                                                                                                                                                                                                                                                         
-- Detecting C compiler ABI info - done                                                                                                                                                                                                                                                  
-- Check for working C compiler: /usr/bin/clang - skipped                                                                                                                                                                                                                                
-- Detecting C compile features                                                                                                                                                                                                                                                          
-- Detecting C compile features - done                                                                                                                                                                                                                                                   
-- Checking if build type is 'Coverage'                                                                                                                                                                                                                                                  
-- Checking if build type is 'Coverage': 0                                                                                                                                                                                                                                               
-- Performing Test _frecursiveFlag                                                                                                                                                                                                                                                       
-- Performing Test _frecursiveFlag - Success                                                                                                                                                                                                                                             
-- Build tests: OFF                                                                                                                                                                                                                                                                      
-- Reducing RELEASE optimization level to O2                                                                                                                                                                                                                                             
-- Looking for Fortran NONE - found                                                                                                                                                                                                                                                      
-- Looking for Fortran INT_CPU_TIME - found                                                                                                                                                                                                                                              
-- Looking for Fortran EXT_ETIME - not found                                                                                                                                                                                                                                             
-- Looking for Fortran EXT_ETIME_ - not found                                                                                                                                                                                                                                            
-- Looking for Fortran INT_ETIME - found                                                                                                                                                                                                                                                 
-- --> Will use second_INT_ETIME.f and dsecnd_INT_ETIME.f as timing function.                                                                                                                                                                                                            
-- Build deprecated routines: OFF                                                                                                                                                                                                                                                        
-- Build single precision real: ON                                                                                                                                                                                                                                                       
-- Build double precision real: ON                                                                                                                                                                                                                                                       
-- Build single precision complex: ON                                                                                                       
-- Build double precision complex: ON                                                                                                       
-- Using supplied NETLIB BLAS implementation                                                                                                
-- CBLAS enable                                                                                                                                                                                                                                                                          
-- Detecting Fortran/C Interface                                                                                                                                                                                                                                                         
-- Detecting Fortran/C Interface - Found GLOBAL and MODULE mangling                                                                                                                                                                                                                      
-- Verifying Fortran/C Compiler Compatibility                                                                                                                                                                                                                                            
-- Verifying Fortran/C Compiler Compatibility - Failed                                                                                                                                                                                                                                   
CMake Error at /usr/share/cmake/Modules/FortranCInterface.cmake:400 (message):                                                                                                                                                                                                           
  The Fortran compiler:                                                                                                                                                                                                                                                                  
                                                                                                                                                                                                                                                                                         
    /home/user/.conan/data/gcc/12.2.0/user/testing/package/e22f47f196f50cff15626bfbdc57fadd84b7f986/bin/gfortran-12.2.0                                                                                                                                                          
                                                                                                                                                                                                                                                                                         
  and the C compiler:                                                                                                                                                                                                                                                                    
                                                                                                                                                                                                                                                                                         
    /usr/bin/clang                                                                                                                                                                                                                                                                       
                                                                                                                                                                                                                                                                                         
  failed to compile a simple test project using both languages.  The output                                                                                                                                                                                                              
  was:                                                     

This is because the gcc recipe fails to override what's been defined in the user profile. This can be rectified by adding the following to the gcc recipe when tools.build:compiler_executables hasn't been defined in the user profile, but CC and CXX have been.

def package_info(self):
    self.conf_info.define("tools.build:compiler_executables", {"c": cc, "cpp": cxx, "fortran": fc})

I think this is intended behaviour, but this also illustrates a need to be able to override/dictate a profile for a recipe that's been built by a compiler pulled in via tool_requires independently of what's been specified by the user. This profile shouldn't be propagated to any other recipes except those that directly specify the compiler package via tool_requires.

Relates to conan-io/conan#4509


* Add recipe
* Add test files
* Uses gcc recipe to get gfortran rather than the gfortran
  recipe as it's broken. This won't work for windows or macOS
* Add separate components for blas, cblas, lapack and lapacke
  to correctly model the relationship between these dependencies
  and allow the fortran libraries to be linked independently of
  the C wrapper libraries.
* Add conan v2 test package
* Remove references to conan gcc package as it is not yet mature
  enough to support this recipe. Specifically, it does not model
  the libraries it provides, so a runtime linkage to gfortran
  can't be appropriately modelled.
@conan-center-bot
Copy link
Collaborator

Conan v1 pipeline ❌

Failure in build 1 (0aa646a7982676ac580bf903d7dad48aa98577d6):

  • lapack/3.10.1@:
    Error running command conan info lapack/3.10.1@#f091ad35540050583e26795bcf7b35f7 --json {jsonName} --dry-build -pr {profileName}:
    [settings]
    arch=x86_64
    build_type=Release
    compiler=gcc
    compiler.libcxx=libstdc++11
    compiler.version=5
    os=Linux
    [options]
    lapack:shared=False
    
    ...
    WARN: gcc/12.2.0: requirement zlib/1.2.13 overridden by lapack/3.10.1 to zlib/1.2.12 
    ERROR: Conflict in binutils/2.38:
        'binutils/2.38' requires 'zlib/1.2.13' while 'gcc/12.2.0' requires 'zlib/1.2.12'.
        To fix this conflict you need to override the package 'zlib' in your root package.
    

Note: To save resources, CI tries to finish as soon as an error is found. For this reason you might find that not all the references have been launched or not all the configurations for a given reference. Also, take into account that we cannot guarantee the order of execution as it depends on CI workload and workers availability.


Conan v2 pipeline (informative, not required for merge) ❌

Note: Conan v2 builds are informative and they are not required for the PR to be merged.

The v2 pipeline failed. Please, review the errors and note this will be required for pull requests to be merged in the near future.

See details:

Failure in build 1 (0aa646a7982676ac580bf903d7dad48aa98577d6):

  • lapack/3.10.1@:
    Error running command conan graph info --requires lapack/3.10.1@#f091ad35540050583e26795bcf7b35f7 -f json -pr:h /home/conan/w/prod-v2_cci_PR-15556/1/c394dd6d-d47a-4737-bec7-e0dcf8d6d918/profile_linux_11_libstdcpp11_gcc_release_64.-shared-False.txt -pr:b /home/conan/w/prod-v2_cci_PR-15556/1/c394dd6d-d47a-4737-bec7-e0dcf8d6d918/profile_linux_11_libstdcpp11_gcc_release_64.-shared-False.txt:
    -------- Input profiles --------
    Profile host:
    [settings]
    arch=x86_64
    build_type=Release
    compiler=gcc
    compiler.libcxx=libstdc++11
    compiler.version=11
    os=Linux
    [options]
    */*:shared=False
    
    Profile build:
    [settings]
    arch=x86_64
    build_type=Release
    compiler=gcc
    compiler.libcxx=libstdc++11
    compiler.version=11
    os=Linux
    [options]
    */*:shared=False
    
    
    -------- Computing dependency graph --------
    zlib/1.2.12: Not found in local cache, looking in remotes...
    zlib/1.2.12: Checking remote: conan-center-v2
    zlib/1.2.12: Trying with 'conan-center-v2'...
    Downloading conanfile.py
    Downloading conan_export.tgz
    Downloading conanmanifest.txt
    Decompressing conan_export.tgz
    zlib/1.2.12: Downloaded recipe revision b1fd071d8a2234a488b3ff74a3526f81
    gcc/12.2.0: Not found in local cache, looking in remotes...
    gcc/12.2.0: Checking remote: conan-center-v2
    gcc/12.2.0: Trying with 'conan-center-v2'...
    Downloading conanfile.py
    Downloading conan_export.tgz
    Downloading conanmanifest.txt
    Decompressing conan_export.tgz
    gcc/12.2.0: Downloaded recipe revision e2965d35782138942e42af8e7da1faf7
    mpc/1.2.0: Not found in local cache, looking in remotes...
    mpc/1.2.0: Checking remote: conan-center-v2
    mpc/1.2.0: Trying with 'conan-center-v2'...
    Downloading conanfile.py
    Downloading conanmanifest.txt
    Downloading conan_export.tgz
    Decompressing conan_export.tgz
    mpc/1.2.0: Downloaded recipe revision db3f79836f2a7a9e15b6c63b7a96649e
    gmp/6.2.1: Not found in local cache, looking in remotes...
    gmp/6.2.1: Checking remote: conan-center-v2
    gmp/6.2.1: Trying with 'conan-center-v2'...
    Downloading conanfile.py
    Downloading conan_export.tgz
    Downloading conanmanifest.txt
    Decompressing conan_export.tgz
    gmp/6.2.1: Downloaded recipe revision 83b8f7a110976a83382f74bab17125fd
    m4/1.4.19: Not found in local cache, looking in remotes...
    m4/1.4.19: Checking remote: conan-center-v2
    m4/1.4.19: Trying with 'conan-center-v2'...
    Downloading conanfile.py
    Downloading conanmanifest.txt
    Downloading conan_export.tgz
    Decompressing conan_export.tgz
    m4/1.4.19: Downloaded recipe revision e7735846d9a3de132d712cb458519daf
    mpfr/4.1.0: Not found in local cache, looking in remotes...
    mpfr/4.1.0: Checking remote: conan-center-v2
    Graph root
        virtual
    Requirements
        gcc/12.2.0#e2965d35782138942e42af8e7da1faf7 - Downloaded (conan-center-v2)
        gmp/6.2.1#83b8f7a110976a83382f74bab17125fd - Downloaded (conan-center-v2)
        lapack/3.10.1#f091ad35540050583e26795bcf7b35f7 - Cache
        mpc/1.2.0#db3f79836f2a7a9e15b6c63b7a96649e - Downloaded (conan-center-v2)
        zlib/1.2.12#b1fd071d8a2234a488b3ff74a3526f81 - Downloaded (conan-center-v2)
    Build requirements
        m4/1.4.19#e7735846d9a3de132d712cb458519daf - Downloaded (conan-center-v2)
    Graph error
        Package 'mpfr/4.1.0' not resolved: Unable to find 'mpfr/4.1.0' in remotes
    ERROR: Package 'mpfr/4.1.0' not resolved: Unable to find 'mpfr/4.1.0' in remotes
    

Note: To save resources, CI tries to finish as soon as an error is found. For this reason you might find that not all the references have been launched or not all the configurations for a given reference. Also, take into account that we cannot guarantee the order of execution as it depends on CI workload and workers availability.

@samuel-emrys
Copy link
Contributor Author

samuel-emrys commented Jan 30, 2023

@memsharded @uilianries @jcar87 @prince-chrismc do you have any opinions on whether a recipe like this should/could be possible with conan? That is, a recipe that specifies it's own requirement for a compiler and brings it in. As far as I can see, we currently face a problem in that a recipe can't override the user profile to:

  • properly set the metadata for any package built by a compiler not specified in the profile
  • ensure that the right configuration is used and that the profile doesn't interfere, as it is above

My ultimate goal with this would be to be able to build an entire toolchain within the conan ecosystem - bootstrap gcc and use it to build glibc and binutils to retain control over binary compatibility without having to resort to using ancient build environments.

This is also somewhat related to conan-io/docs#3960

@samuel-emrys samuel-emrys marked this pull request as draft January 30, 2023 13:28
* Add gfortran component dependency
* Remove duplication of test package source
@uilianries
Copy link
Member

@samuel-emrys I have to say it's an unique scenario. Indeed GCC package is the only one which provides fortran libraries so far and creating a separated package it be possible, but requires much more effort, as gcc is a direct dependency of gfortran. Since GCC 11, we are using new Docker images with gfortran already installed, but still, it's fragile as users would not have it installed in their system.

My suggestion is keeping a minimal configuration for now, like only supporting with GCC 10 to get aligned with the gcc package. Then we can think getting gfortran separated from GCC package.

@stale
Copy link

stale bot commented Mar 11, 2023

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Mar 11, 2023
@samuel-emrys
Copy link
Contributor Author

I've been meaning to come back to this - but is there a reason that a draft PR is being marked stale?

@prince-chrismc
Copy link
Contributor

The third-party bot is not very smart :)

@stale
Copy link

stale bot commented May 1, 2023

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label May 1, 2023
@samuel-emrys
Copy link
Contributor Author

bump for activation. still in progress. Plan is probably to remove gcc as a dependency at the moment as I don't think this holds the solution for providing libgfortran or a fortran compiler. conan-io/conan#13533 discusses some of the complexity here.

#17173 has demonstrated that there might be a way to bypass the fortran requirement and compile with C instead, so that's probably the avenue that I'll explore here.

@stale stale bot removed the stale label May 1, 2023
@stale
Copy link

stale bot commented Jun 9, 2023

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Jun 9, 2023
@samuel-emrys
Copy link
Contributor Author

Bump for activation. still in progress.

@stale stale bot removed the stale label Jun 10, 2023
@stale
Copy link

stale bot commented Jul 15, 2023

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@stale stale bot added the stale label Jul 15, 2023
@stale
Copy link

stale bot commented Oct 15, 2023

This pull request has been automatically closed because it has not had recent activity. Thank you for your contributions.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants