Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Flex: Conan v2 support #14013

Closed
wants to merge 16 commits into from

Conversation

datalogics-kam
Copy link
Contributor

Specify library name and version: flex/2.4.1

Add v2 support to flex.


  • I've read the guidelines for contributing.
  • I've followed the PEP8 style guides for Python code in the recipes.
  • I've used the latest Conan client version.
  • I've tried at least one configuration locally with the conan-center hook activated.

@ghost
Copy link

ghost commented Nov 3, 2022

I detected other pull requests that are modifying flex/all recipe:

This message is automatically generated by https://github.com/ericLemanissier/conan-center-conflicting-prs so don't hesitate to report issues/improvements there.

@ghost ghost mentioned this pull request Nov 3, 2022
4 tasks
@datalogics-kam
Copy link
Contributor Author

The test_package currently gets an error:

CMake Error at CMakeLists.txt:6 (flex_target):
  Unknown CMake command "flex_target".

CMake has a built in module for finding flex, which defines a flex_target macro. I think what's happening here is that whatever Conan is making is not including that target.

Any advice on what to do for this? I suspect that bison is going to have the same kind of problem.

@conan-center-bot

This comment has been minimized.

- Doesn't look like there's a replacement for tools.which() in Conan
@datalogics-kam
Copy link
Contributor Author

Related issue about flex_target: #1813

@conan-center-bot

This comment has been minimized.

@github-actions
Copy link
Contributor

github-actions bot commented Nov 3, 2022

Hooks produced the following warnings for commit b40b1bf
flex/2.6.4
post_package(): WARN: [MISSING SYSTEM LIBS (KB-H043)] Library './lib/libfl.so' links to system library 'm' but it is not in cpp_info.system_libs.

Comment on lines -4 to -5
# Find FLEX before `conanbuildinfo.cmake` because that file will let `find_program`
# look for executables in host packages (let's hope conan 2.0 fixes this)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

https://cmake.org/cmake/help/latest/module/FindFLEX.html 🤔 Usually it's just variables so it's say #13511 (comment)

If this was an output of flex's CMake install we could use build_modules but I am not sure if thats the case?

Just a note this recipe is missing the cmake target definitions so you'll want to add those

Copy link
Contributor

@jwillikers jwillikers Nov 4, 2022

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You should be able to inject the necessary variables in the CMakeToolchain via conf. Generate a small CMake script that sets the necessary Flex variables and add it to tools.cmake.cmaketoolchain:user_toolchain so that anything that tool_requires Flex will inherit those CMake variables. You probably will only need to set FLEX_EXECUTABLE, FLEX_LIBRARIES, and FLEX_INCLUDE_DIRS. Then maybe see what happens if you rename Conan's generated Config module to something like _Flex so that nothing will actually use it.

Alternatively, if you can add a build module that includes CMake's FindFlex module, that may also work. You'd probably have to set the same variables before including it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks to @prince-chrismc and @jwillikers for the insights. The suggestion to set up an environment variable gave me an idea. I went and looked at the code for FindFLEX. It's internally using find_program, find_library and find_path, all of which will use FLEX_ROOT as the first place to search, for CMake >= 3.12. See also the doc for find_library.

Turns out the following works, at least in my local testing:

  • Point FLEX_ROOT at the package.
  • Keep Conan from replacing CMake's built-in FindFLEX. For CMakeDeps, have the recipe produce nothing for CMake, and for the old generators, make the package with a different name.

- Based on the imports from conan.tools
Replacing the internal find module causes CMake to not supply the
flex_target macro.

- For CMakeDeps, don't export a dependency at all.

- For cmake_find_package and cmake_find_package_multi, name the package
  _FLEX to not conflict with the built-in one.

- Set FLEX_ROOT to help CMake find the flex binaries, libraries, and
  headers in the Conan package. This works with CMake 3.12 or newer.
@conan-center-bot

This comment has been minimized.

@conan-center-bot

This comment has been minimized.

@datalogics-kam datalogics-kam marked this pull request as ready for review December 1, 2022 17:04
@datalogics-kam
Copy link
Contributor Author

I need some help with the CI. It's failing the arm64 build. I'm at a loss because I can build, and even cross-build, on an M1 machine locally.

The root cause seems to be the architecture of the library is wrong, in this log.

[ 20%] [FLEX][flex_scanner] Building scanner with flex 2.6.4
[ 40%] Building C object CMakeFiles/test_yywrap.dir/test_yywrap.c.o
[ 60%] Building CXX object CMakeFiles/test_package.dir/basic_nr.cpp.o
[ 80%] Linking C executable test_yywrap
[100%] Linking CXX executable test_package
[100%] Built target test_package
ld: warning: ignoring file /Users/jenkins/w/prod/BuildSingleReference@3/.conan/data/flex/2.6.4/_/_/package/61053b74780e15957d09b787cf51ee4e6d72e07d/lib/libfl.a, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
Undefined symbols for architecture arm64:
  "_yywrap", referenced from:
      _main in test_yywrap.c.o
ld: symbol(s) not found for architecture arm64
clang: error: linker command failed with exit code 1 (use -v to see invocation)
make[2]: *** [test_yywrap] Error 1
make[1]: *** [CMakeFiles/test_yywrap.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
ld: warning: ignoring file /Users/jenkins/w/prod/BuildSingleReference@3/.conan/data/flex/2.6.4/_/_/package/61053b74780e15957d09b787cf51ee4e6d72e07d/lib/libfl.a, building for macOS-arm64 but attempting to link with file built for macOS-x86_64

I got onto a MacBook Air M1 and it builds fine. I even tried to invoke the cross-building that seems to be going on in CI, and that also worked.

$ conan create recipes/flex/all flex/2.6.4@ -pr:h default -pr:b default-x86_64 --build flex
...lots of building...
[ 20%] Linking C executable test_yywrap
[ 40%] Building CXX object CMakeFiles/test_package.dir/basic_nr.cpp.o
[ 60%] Built target test_yywrap
[ 80%] Linking CXX executable test_package
[100%] Built target test_package
flex/2.6.4 (test package): Running test()
$ conan profile show default
Configuration for profile default:

[settings]
os=Macos
os_build=Macos
arch=armv8
arch_build=armv8
compiler=apple-clang
compiler.version=14
compiler.libcxx=libc++
build_type=Release
[options]
[conf]
[build_requires]
[env]
$ conan profile show default-x86_64
Configuration for profile default-x86_64:

[settings]
os=Macos
os_build=Macos
arch=x86_64
arch_build=x86_64
compiler=apple-clang
compiler.version=14
compiler.libcxx=libc++
build_type=Release
[options]
[conf]
[build_requires]
[env]

@datalogics-kam
Copy link
Contributor Author

I'm kind of dead in the water on this one. I tried to reproduce the problem locally again today, and as mentioned above, I can't produce the build problem locally on an M1 machine.

@conan-center-bot

This comment has been minimized.

Copy link
Contributor

@jwillikers jwillikers left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@datalogics-kam Thanks for working so hard on this. It's a really big update. I'm not exactly sure the root cause of your error, but I'm pretty certain that the test_package is attempting to link with the Flex library from a different architecture. I recommend seeing if it's possible to update things to align as much as possible with the Autootools Template Package. That might help solve the problem, or at least reduce the number of potential problems.

recipes/flex/all/test_package/conanfile.py Outdated Show resolved Hide resolved
recipes/flex/all/test_package/CMakeLists.txt Show resolved Hide resolved
recipes/flex/all/conanfile.py Outdated Show resolved Hide resolved
recipes/flex/all/conanfile.py Outdated Show resolved Hide resolved
autotools.configure(args=configure_args, configure_dir=self._source_subfolder)
return autotools
tc = VirtualBuildEnv(self)
tc.generate(scope="build")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You can omit scope="build" here as it is the default.

Comment on lines 59 to 60
f"--enable-shared={yes_no(self.options.shared)}",
f"--enable-static={yes_no(not self.options.shared)}",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AutotoolsToolchain should set these two variables automatically: https://docs.conan.io/en/latest/reference/conanfile/tools/gnu/autotoolstoolchain.html#attributes

Co-authored-by: Jordan Williams <jordan@jwillikers.com>
@conan-center-bot

This comment has been minimized.

- Use tool_requires instead of build_requires

- Don't define --enable-shared or --disable-shared; they're automatic.
- Idea copied from the CMake recipe
- Defining C just caused a build error; there's C++ code in the
  CMakeLists.txt.

- While checking the doc for project(), the default for LANGUAGES is C
  and CXX anyway, which is what is needed.
  https://cmake.org/cmake/help/v3.25/command/project.html#options
@datalogics-kam
Copy link
Contributor Author

I did a bit more cleaning up.

The root of the problem seems to be the failure when cross-building with x86_64 build environment and armv8 host, which is what the M1 CI builder is doing.

  • This package can't let Conan create a FindFLEX module. If the module from Conan replaces the one in CMake, then the flex_target macro isn't there, and programs that depend on it won't build.
  • To get around the above, I define the FLEX_ROOT variable, which CMake should respect, but when cross-building, the FLEX_ROOT from the build environment gets used, when it needs the one from the host environment.

If there's another way to get a CMake variable defined, or a way to extend or wrap a built-in module, I'd be interested in hearing about it.

@conan-center-bot
Copy link
Collaborator

Conan v1 pipeline

Failure in build 7 (0b0160d75da0e0034194bf1b0f7d2018aef98763):

  • flex/2.6.4@:
    CI failed to create some packages (All logs)

    Logs for packageID c91d4b6cfc498f9716c80c25fab1004bd6cc4962:
    [settings]
    arch=x86_64
    build_type=Release
    compiler=apple-clang
    compiler.libcxx=libc++
    compiler.version=12.0
    os=Macos
    [options]
    flex:shared=True
    
    [...]
    gcc -DHAVE_CONFIG_H -I.  -DLOCALEDIR=\"//share/locale\" -DNDEBUG   -m64 -O3 -MT flex-sym.o -MD -MP -MF .deps/flex-sym.Tpo -c -o flex-sym.o `test -f 'sym.c' || echo './'`sym.c
    mv -f .deps/flex-scanopt.Tpo .deps/flex-scanopt.Po
    gcc -DHAVE_CONFIG_H -I.  -DLOCALEDIR=\"//share/locale\" -DNDEBUG   -m64 -O3 -MT flex-tables.o -MD -MP -MF .deps/flex-tables.Tpo -c -o flex-tables.o `test -f 'tables.c' || echo './'`tables.c
    mv -f .deps/flex-sym.Tpo .deps/flex-sym.Po
    gcc -DHAVE_CONFIG_H -I.  -DLOCALEDIR=\"//share/locale\" -DNDEBUG   -m64 -O3 -MT flex-tables_shared.o -MD -MP -MF .deps/flex-tables_shared.Tpo -c -o flex-tables_shared.o `test -f 'tables_shared.c' || echo './'`tables_shared.c
    mv -f .deps/flex-parse.Tpo .deps/flex-parse.Po
    gcc -DHAVE_CONFIG_H -I.  -DLOCALEDIR=\"//share/locale\" -DNDEBUG   -m64 -O3 -MT flex-tblcmp.o -MD -MP -MF .deps/flex-tblcmp.Tpo -c -o flex-tblcmp.o `test -f 'tblcmp.c' || echo './'`tblcmp.c
    mv -f .deps/flex-tables_shared.Tpo .deps/flex-tables_shared.Po
    gcc -DHAVE_CONFIG_H -I.  -DLOCALEDIR=\"//share/locale\" -DNDEBUG   -m64 -O3 -MT flex-yylex.o -MD -MP -MF .deps/flex-yylex.Tpo -c -o flex-yylex.o `test -f 'yylex.c' || echo './'`yylex.c
    mv -f .deps/flex-tables.Tpo .deps/flex-tables.Po
    sed 's|^\(#line .*\)"'`printf %s scan.c | sed 's|[][\\\\.*]|\\\\&|g'`'"|\1"stage1scan.c"|g' scan.c > stage1scan.c
    /bin/sh ../libtool  --tag=CC   --mode=link gcc  -m64 -O3 -version-info 2:0:0 -m64 -o libfl.la -rpath //lib libmain.lo libyywrap.lo  -lm 
    mv -f .deps/flex-yylex.Tpo .deps/flex-yylex.Po
    gcc -DHAVE_CONFIG_H -I.  -DLOCALEDIR=\"//share/locale\" -DNDEBUG   -m64 -O3 -MT flex-stage1scan.o -MD -MP -MF .deps/flex-stage1scan.Tpo -c -o flex-stage1scan.o `test -f 'stage1scan.c' || echo './'`stage1scan.c
    libtool: link: gcc -dynamiclib  -o .libs/libfl.2.dylib  .libs/libmain.o .libs/libyywrap.o   -lm  -m64 -O3 -m64   -install_name  //lib/libfl.2.dylib -compatibility_version 3 -current_version 3.0 -Wl,-single_module
    mv -f .deps/flex-tblcmp.Tpo .deps/flex-tblcmp.Po
    mv -f .deps/flex-gen.Tpo .deps/flex-gen.Po
    mv -f .deps/flex-stage1scan.Tpo .deps/flex-stage1scan.Po
    flex/2.6.4: 
    configure: WARNING: /usr/bin/indent does not appear to be GNU indent; 'make indent' may not function properly
    misc.c:515:40: warning: passing 'unsigned char *' to parameter of type 'const char *' converts between pointers to integer types with different sign [-Wpointer-sign]
                            esc_char = (unsigned char) strtoul (array + 1, NULL, 8);
                                                                ^~~~~~~~~
    /Applications/conan/xcode/12.0/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk/usr/include/stdlib.h:175:23: note: passing argument to parameter '__str' here
             strtoul(const char *__str, char **__endptr, int __base);
                                 ^
    misc.c:537:40: warning: passing 'unsigned char *' to parameter of type 'const char *' converts between pointers to integer types with different sign [-Wpointer-sign]
                            esc_char = (unsigned char) strtoul (array + 2, NULL, 16);
                                                                ^~~~~~~~~
    /Applications/conan/xcode/12.0/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX11.3.sdk/usr/include/stdlib.h:175:23: note: passing argument to parameter '__str' here
             strtoul(const char *__str, char **__endptr, int __base);
                                 ^
    2 warnings generated.
    ld: warning: double slash removed from -install_name (//lib/libfl.2.dylib)
    Undefined symbols for architecture x86_64:
      "_yylex", referenced from:
          _main in libmain.o
    ld: symbol(s) not found for architecture x86_64
    clang: error: linker command failed with exit code 1 (use -v to see invocation)
    make[2]: *** [libfl.la] Error 1
    make[2]: *** Waiting for unfinished jobs....
    make[1]: *** [all] Error 2
    make: *** [all-recursive] Error 1
    flex/2.6.4: WARN: Using the new toolchains and generators without specifying a build profile (e.g: -pr:b=default) is discouraged and might cause failures and unexpected behavior
    flex/2.6.4: WARN: Using the new toolchains and generators without specifying a build profile (e.g: -pr:b=default) is discouraged and might cause failures and unexpected behavior
    flex/2.6.4: ERROR: Package 'c91d4b6cfc498f9716c80c25fab1004bd6cc4962' build failed
    flex/2.6.4: WARN: Build folder /Users/jenkins/w/prod/BuildSingleReference/.conan/data/flex/2.6.4/_/_/build/c91d4b6cfc498f9716c80c25fab1004bd6cc4962
    ERROR: flex/2.6.4: Error in build() method, line 76
    	autotools.make()
    	ConanException: Error 2 while executing make -j4
    

Note: To save resources, CI tries to finish as soon as an error is found. For this reason you might find that not all the references have been launched or not all the configurations for a given reference. Also, take into account that we cannot guarantee the order of execution as it depends on CI workload and workers availability.

@jwillikers
Copy link
Contributor

jwillikers commented Dec 14, 2022

@datalogics-kam I've had to do something similar with Meson / PkgConfigDeps for the Wayland package, see here. You might be able to make this work by using the CMakeDeps features build_context_activated, build-context-suffix, and build_context_build_modules in the generate method of the test_package.

Ideally, it would be possible to more cleanly separate the CMake modules generated by Conan into different directories and point CMake to the correct ones based on the build / host contexts. This wouldn't require renaming the CMake modules for the build context, which requires changes to the underlying build system. There's an open an issue regarding this behavior: conan-io/conan#12342.

@ghost ghost mentioned this pull request Jan 12, 2023
3 tasks
@ghost ghost mentioned this pull request Jan 23, 2023
3 tasks
@datalogics-kam
Copy link
Contributor Author

Superseded by #15234 and #15419

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants