Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

More issues with lief #5594

Open
2 tasks done
h-vetinari opened this issue Jan 16, 2025 · 8 comments · May be fixed by #5595
Open
2 tasks done

More issues with lief #5594

h-vetinari opened this issue Jan 16, 2025 · 8 comments · May be fixed by #5595
Labels
type::bug describes erroneous operation, use severity::* to classify the type

Comments

@h-vetinari
Copy link
Contributor

h-vetinari commented Jan 16, 2025

Checklist

  • I added a descriptive title
  • I searched open reports and couldn't find a duplicate

What happened?

It seems lief 0.16 broke something else than #5564. logs from conda-forge/pyarrow-feedstock#144

Adding support to 'conda-build' for parsing these in 'liefldd.py' would be easy and useful:
['System/Library/PrivateFrameworks/Swift/libswiftGameplayKit.tbd', 'System/Library/PrivateFrameworks/CorePhoneNumbers.framework/CorePhoneNumbers.tbd']...
Traceback (most recent call last):
  File "/Users/runner/miniforge3/bin/conda-build", line 11, in <module>
    sys.exit(execute())
             ^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/cli/main_build.py", line 622, in execute
    api.build(
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/api.py", line 211, in build
    return build_tree(
           ^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/build.py", line 3656, in build_tree
    packages_from_this = build(
                         ^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/build.py", line 2767, in build
    newly_built_packages = bundlers[pkg_type](
                           ^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/build.py", line 1870, in bundle_conda
    files = post_process_files(metadata, initial_files)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/build.py", line 1650, in post_process_files
    post_build(m, new_files, build_python=python)
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/post.py", line 1764, in post_build
    check_overlinking(m, files, host_prefix)
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/post.py", line 1589, in check_overlinking
    return check_overlinking_impl(
           ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/post.py", line 1449, in check_overlinking_impl
    all_needed_dsos, needed_dsos_for_file = _collect_needed_dsos(
                                            ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/post.py", line 864, in _collect_needed_dsos
    needed = get_linkages_memoized(
             ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/os_utils/liefldd.py", line 1199, in __call__
    value = self.func(*args, **kw)
            ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/os_utils/liefldd.py", line 1228, in get_linkages_memoized
    return get_linkages(
           ^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/os_utils/liefldd.py", line 635, in get_linkages
    result_lief = inspect_linkages_lief(
                  ^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/os_utils/liefldd.py", line 532, in inspect_linkages_lief
    rpaths_by_binary[filename2] = get_rpaths(
                                  ^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/os_utils/liefldd.py", line 263, in get_rpaths
    rpaths, rpaths_type, binary_format, binary_type = get_runpaths_or_rpaths_raw(file)
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/os_utils/liefldd.py", line 241, in get_runpaths_or_rpaths_raw
    rpaths, _, _ = get_rpaths_raw(file)
                   ^^^^^^^^^^^^^^^^^^^^
  File "/Users/runner/miniforge3/lib/python3.12/site-packages/conda_build/os_utils/liefldd.py", line 210, in get_rpathy_thing_raw_partial
    if command.command == lief.MachO.LOAD_COMMAND_TYPES.RPATH
                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
AttributeError: module 'lief._lief.MachO' has no attribute 'LOAD_COMMAND_TYPES'

##[error]Bash exited with code '1'.
##[section]Finishing: Run OSX build

Conda Info

++ conda info
++ local cmd=info
++ case "$cmd" in
++ __conda_exe info
++ /Users/runner/miniforge3/bin/conda info

     active environment : base
    active env location : /Users/runner/miniforge3
            shell level : 1
       user config file : /Users/runner/.condarc
 populated config files : /Users/runner/miniforge3/.condarc
          conda version : 24.11.3
    conda-build version : 25.1.1
         python version : 3.12.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=skylake
                          __conda=24.11.3=0
                          __osx=13.7.2=0
                          __unix=0=0
       base environment : /Users/runner/miniforge3  (writable)
      conda av data dir : /Users/runner/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-64
                          https://conda.anaconda.org/conda-forge/noarch
          package cache : /Users/runner/miniforge3/pkgs
                          /Users/runner/.conda/pkgs
       envs directories : /Users/runner/miniforge3/envs
                          /Users/runner/.conda/envs
               platform : osx-64
             user-agent : conda/24.11.3 requests/2.32.3 CPython/3.12.8 Darwin/22.6.0 OSX/13.7.2 solver/libmamba conda-libmamba-solver/24.11.1 libmambapy/2.0.5
                UID:GID : 501:20
             netrc file : None
           offline mode : False

Conda Config

++ conda config --env --show-sources
++ local cmd=config
++ case "$cmd" in
++ __conda_exe config --env --show-sources
++ /Users/runner/miniforge3/bin/conda config --env --show-sources
==> /Users/runner/miniforge3/.condarc <==
add_pip_as_python_dependency: False
auto_update_conda: False
aggressive_update_packages:
  - ca-certificates
  - certifi
channel_priority: strict
channels:
  - conda-forge
show_channel_urls: True
conda_build:
  pkg_format: 2
  zstd_compression_level: 19

==> envvars <==
allow_softlinks: False
solver: libmamba
bld_path: /Users/runner/miniforge3/conda-bld

Conda list

++ conda list --show-channel-urls
++ local cmd=list
++ case "$cmd" in
++ __conda_exe list --show-channel-urls
++ /Users/runner/miniforge3/bin/conda list --show-channel-urls
# packages in environment at /Users/runner/miniforge3:
#
# Name                    Version                   Build  Channel
anaconda-client           1.12.3             pyhd8ed1ab_1    conda-forge
archspec                  0.2.3              pyhd8ed1ab_0    conda-forge
attrs                     24.3.0             pyh71513ae_0    conda-forge
beautifulsoup4            4.12.3             pyha770c72_1    conda-forge
boltons                   24.0.0             pyhd8ed1ab_1    conda-forge
brotli-python             1.1.0           py312h5861a67_2    conda-forge
bzip2                     1.0.8                hfdf4475_7    conda-forge
c-ares                    1.34.4               hf13058a_0    conda-forge
ca-certificates           2024.12.14           h8857fd0_0    conda-forge
cctools                   1010.6               h5b2de21_2    conda-forge
cctools_osx-64            1010.6               hea4301f_2    conda-forge
certifi                   2024.12.14         pyhd8ed1ab_0    conda-forge
cffi                      1.17.1          py312hf857d28_0    conda-forge
chardet                   5.2.0           py312hb401068_2    conda-forge
charset-normalizer        3.4.1              pyhd8ed1ab_0    conda-forge
click                     8.1.8              pyh707e725_0    conda-forge
colorama                  0.4.6              pyhd8ed1ab_1    conda-forge
conda                     24.11.3         py312hb401068_0    conda-forge
conda-build               25.1.1          py312hb401068_0    conda-forge
conda-env                 2.6.0                         1    conda-forge
conda-forge-ci-setup      4.14.3          py312hd35cd3b_100    conda-forge
conda-forge-metadata      0.11.0             pyhd8ed1ab_1    conda-forge
conda-index               0.5.0              pyhd8ed1ab_0    conda-forge
conda-libmamba-solver     24.11.1            pyhd8ed1ab_0    conda-forge
conda-oci-mirror          0.2.1              pyhd8ed1ab_1    conda-forge
conda-package-handling    2.4.0              pyh7900ff3_2    conda-forge
conda-package-streaming   0.11.0             pyhd8ed1ab_0    conda-forge
cpp-expected              1.1.0                hb8565cd_0    conda-forge
defusedxml                0.7.1              pyhd8ed1ab_0    conda-forge
deprecated                1.2.15             pyhd8ed1ab_1    conda-forge
distro                    1.9.0              pyhd8ed1ab_1    conda-forge
filelock                  3.16.1             pyhd8ed1ab_1    conda-forge
fmt                       11.0.2               h3c5361c_0    conda-forge
frozendict                2.4.6           py312h3d0f464_0    conda-forge
git                       2.47.1          pl5321h0e333bc_0    conda-forge
h2                        4.1.0              pyhd8ed1ab_1    conda-forge
hpack                     4.0.0              pyhd8ed1ab_1    conda-forge
hyperframe                6.0.1              pyhd8ed1ab_1    conda-forge
idna                      3.10               pyhd8ed1ab_1    conda-forge
importlib_resources       6.5.2              pyhd8ed1ab_0    conda-forge
jinja2                    3.1.5              pyhd8ed1ab_0    conda-forge
joblib                    1.4.2              pyhd8ed1ab_1    conda-forge
jq                        1.7.1                h10d778d_0    conda-forge
jsonpatch                 1.33               pyhd8ed1ab_1    conda-forge
jsonpointer               3.0.0           py312hb401068_1    conda-forge
jsonschema                4.23.0             pyhd8ed1ab_1    conda-forge
jsonschema-specifications 2024.10.1          pyhd8ed1ab_1    conda-forge
jupyter_core              5.7.2              pyh31011fe_1    conda-forge
krb5                      1.21.3               h37d8d59_0    conda-forge
ld64                      951.9                h0a3eb4e_2    conda-forge
ld64_osx-64               951.9                h5ffbe8e_2    conda-forge
libarchive                3.7.7                h1a33361_3    conda-forge
libcurl                   8.11.1               h5dec5d8_0    conda-forge
libcxx                    19.1.7               hf95d169_0    conda-forge
libedit                   3.1.20240808    pl5321ha958ccf_0    conda-forge
libev                     4.33                 h10d778d_2    conda-forge
libexpat                  2.6.4                h240833e_0    conda-forge
libffi                    3.4.2                h0d85af4_5    conda-forge
libiconv                  1.17                 hd75f5a5_2    conda-forge
libintl                   0.22.5               hdfe23c8_3    conda-forge
liblief                   0.16.0               h240833e_1    conda-forge
libllvm17                 17.0.6               hbedff68_1    conda-forge
liblzma                   5.6.3                hd471939_1    conda-forge
libmamba                  2.0.5                h415aaf8_1    conda-forge
libmambapy                2.0.5           py312h4e4b3da_1    conda-forge
libnghttp2                1.64.0               hc7306c3_0    conda-forge
libsolv                   0.7.30               h69d5d9b_0    conda-forge
libsqlite                 3.47.2               hdb6dae5_0    conda-forge
libssh2                   1.11.1               h3dc7d44_0    conda-forge
libxml2                   2.13.5               he8ee3e7_1    conda-forge
libzlib                   1.3.1                hd23fc13_2    conda-forge
llvm-tools                17.0.6               hbedff68_1    conda-forge
lz4-c                     1.10.0               h240833e_1    conda-forge
lzo                       2.10              h10d778d_1001    conda-forge
mamba                     2.0.5                h82323bd_1    conda-forge
markupsafe                3.0.2           py312h3520af0_1    conda-forge
mbedtls                   3.5.1                h240833e_1    conda-forge
menuinst                  2.2.0           py312hb401068_0    conda-forge
more-itertools            10.6.0             pyhd8ed1ab_0    conda-forge
nbformat                  5.10.4             pyhd8ed1ab_1    conda-forge
ncurses                   6.5                  h0622a9a_2    conda-forge
nlohmann_json             3.11.3               hf036a51_1    conda-forge
oniguruma                 6.9.10               h6e16a3a_0    conda-forge
openssl                   3.4.0                hc426f3f_1    conda-forge
oras-py                   0.1.14             pyhd8ed1ab_0    conda-forge
packaging                 24.2               pyhd8ed1ab_2    conda-forge
patch                     2.7.6             hbcf498f_1002    conda-forge
pcre2                     10.44                h7634a1b_2    conda-forge
perl                      5.32.1          7_h10d778d_perl5    conda-forge
pip                       24.3.1             pyh8b19718_2    conda-forge
pkginfo                   1.12.0             pyhd8ed1ab_1    conda-forge
pkgutil-resolve-name      1.3.10             pyhd8ed1ab_2    conda-forge
platformdirs              4.3.6              pyhd8ed1ab_1    conda-forge
pluggy                    1.5.0              pyhd8ed1ab_1    conda-forge
psutil                    6.1.1           py312h01d7ebd_0    conda-forge
py-lief                   0.16.0          py312haafddd8_1    conda-forge
pybind11-abi              4                    hd8ed1ab_3    conda-forge
pycosat                   0.6.6           py312h01d7ebd_2    conda-forge
pycparser                 2.22               pyh29332c3_1    conda-forge
pysocks                   1.7.1              pyha55dd90_7    conda-forge
python                    3.12.8          h9ccd52b_1_cpython    conda-forge
python-dateutil           2.9.0.post0        pyhff2d567_1    conda-forge
python-fastjsonschema     2.21.1             pyhd8ed1ab_0    conda-forge
python-libarchive-c       5.1             py312hb401068_1    conda-forge
python_abi                3.12                    5_cp312    conda-forge
pytz                      2024.2             pyhd8ed1ab_1    conda-forge
pyyaml                    6.0.2           py312hb553811_1    conda-forge
rattler-build             0.34.1               h113f492_0    conda-forge
rattler-build-conda-compat 1.3.3              pyhd8ed1ab_0    conda-forge
readline                  8.2                  h9e318b2_1    conda-forge
referencing               0.35.1             pyhd8ed1ab_1    conda-forge
reproc                    14.2.5.post0         h6e16a3a_0    conda-forge
reproc-cpp                14.2.5.post0         h240833e_0    conda-forge
requests                  2.32.3             pyhd8ed1ab_1    conda-forge
requests-toolbelt         1.0.0              pyhd8ed1ab_1    conda-forge
ripgrep                   14.1.1               h926acf8_0    conda-forge
rpds-py                   0.22.3          py312h0d0de52_0    conda-forge
ruamel.yaml               0.18.10         py312h01d7ebd_0    conda-forge
ruamel.yaml.clib          0.2.8           py312h3d0f464_1    conda-forge
setuptools                75.8.0             pyhff2d567_0    conda-forge
shyaml                    0.6.2              pyhd3deb0d_0    conda-forge
sigtool                   0.1.3                h88f4db0_0    conda-forge
simdjson                  3.11.6               h9275861_0    conda-forge
six                       1.17.0             pyhd8ed1ab_0    conda-forge
soupsieve                 2.5                pyhd8ed1ab_1    conda-forge
spdlog                    1.15.0               h0ec5880_0    conda-forge
tapi                      1300.6.5             h390ca13_0    conda-forge
tk                        8.6.13               h1abcd95_1    conda-forge
tomli                     2.2.1              pyhd8ed1ab_1    conda-forge
tqdm                      4.67.1             pyhd8ed1ab_1    conda-forge
traitlets                 5.14.3             pyhd8ed1ab_1    conda-forge
truststore                0.10.0             pyhd8ed1ab_0    conda-forge
typing-extensions         4.12.2               hd8ed1ab_1    conda-forge
typing_extensions         4.12.2             pyha770c72_1    conda-forge
tzdata                    2024b                hc8b5060_0    conda-forge
urllib3                   2.3.0              pyhd8ed1ab_0    conda-forge
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge
wrapt                     1.17.2          py312h01d7ebd_0    conda-forge
yaml                      0.2.5                h0d85af4_2    conda-forge
yaml-cpp                  0.8.0                he965462_0    conda-forge
zipp                      3.21.0             pyhd8ed1ab_1    conda-forge
zstandard                 0.23.0          py312h7122b0e_1    conda-forge
zstd                      1.5.6                h915ae27_0    conda-forge

Additional Context

No response

@h-vetinari h-vetinari added the type::bug describes erroneous operation, use severity::* to classify the type label Jan 16, 2025
@github-project-automation github-project-automation bot moved this to 🆕 New in 🧭 Planning Jan 16, 2025
@kenodegard
Copy link
Contributor

kenodegard commented Jan 16, 2025

@h-vetinari sounds like we need to rebuild conda-forge::conda-build=25.1.1 with py-lief<0.16, is this already in the works or should I open a PR?

@h-vetinari
Copy link
Contributor Author

I have a PR upcoming

@tkelman
Copy link
Contributor

tkelman commented Jan 16, 2025

Does it work against lief 0.15 though? It might be more an issue of missing osx ci coverage and #5565 being incomplete, https://lief.re/doc/stable/changelog.html#july-21th-2024 says "Re-scope LOAD_COMMAND_TYPES into lief.MachO.LoadCommand.TYPE" was a change in 0.15.

@jeongseok-meta
Copy link

Does it work against lief 0.15 though?

From my observation, it still fails with 0.15 but works fine with 0.14.

@tkelman
Copy link
Contributor

tkelman commented Jan 16, 2025

Thanks for looking into it. I think the code that fails might only be hit in an osx build? Either way, we need to add another or adjust the existing repodata patches, and I've got a proposed fix #5595 for the next conda-build release.

@duncanmmacleod
Copy link
Contributor

Sorry if this is the wrong place to post, but I'm seeing issues with lief 0.15 on Linux:

# >>>>>>>>>>>>>>>>>>>>>> ERROR REPORT <<<<<<<<<<<<<<<<<<<<<<

    Traceback (most recent call last):
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda/exception_handler.py", line 18, in __call__
        return func(*args, **kwargs)
               ^^^^^^^^^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda/cli/main.py", line 61, in main_subshell
        exit_code = do_call(args, parser)
                    ^^^^^^^^^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda/cli/conda_argparse.py", line 181, in do_call
        result = plugin_subcommand.action(getattr(args, "_args", args))
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/plugin.py", line 17, in build
        return execute(args)
               ^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/cli/main_build.py", line 622, in execute
        api.build(
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/api.py", line 211, in build
        return build_tree(
               ^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/build.py", line 3656, in build_tree
        packages_from_this = build(
                             ^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/build.py", line 2767, in build
        newly_built_packages = bundlers[pkg_type](
                               ^^^^^^^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/build.py", line 1870, in bundle_conda
        files = post_process_files(metadata, initial_files)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/build.py", line 1650, in post_process_files
        post_build(m, new_files, build_python=python)
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/post.py", line 1764, in post_build
        check_overlinking(m, files, host_prefix)
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/post.py", line 1589, in check_overlinking
        return check_overlinking_impl(
               ^^^^^^^^^^^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/post.py", line 1449, in check_overlinking_impl
        all_needed_dsos, needed_dsos_for_file = _collect_needed_dsos(
                                                ^^^^^^^^^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/post.py", line 864, in _collect_needed_dsos
        needed = get_linkages_memoized(
                 ^^^^^^^^^^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/os_utils/liefldd.py", line 1199, in __call__
        value = self.func(*args, **kw)
                ^^^^^^^^^^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/os_utils/liefldd.py", line 1228, in get_linkages_memoized
        return get_linkages(
               ^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/os_utils/liefldd.py", line 635, in get_linkages
        result_lief = inspect_linkages_lief(
                      ^^^^^^^^^^^^^^^^^^^^^^
      File "/home/duncan/opt/conda/lib/python3.12/site-packages/conda_build/os_utils/liefldd.py", line 486, in inspect_linkages_lief
        if binary.type == lief.ELF.ELF_CLASS.CLASS64:
                          ^^^^^^^^^^^^^^^^^^
    AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'

A simple test can reproduce:

$ python3 -c "import lief; print(lief.ELF.ELF_CLASS)"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
AttributeError: module 'lief._lief.ELF' has no attribute 'ELF_CLASS'

According to the release notes this is now

$ python3 -c "import lief; print(lief.ELF.Header.CLASS)"
<class 'lief._lief.ELF.Header.CLASS'>

@tkelman
Copy link
Contributor

tkelman commented Jan 17, 2025

Good find, I'll handle that too. What package build hit that, and why did it exercise a liefldd.py code path that none of the tests on CI did?

@duncanmmacleod
Copy link
Contributor

Good find, I'll handle that too. What package build hit that, and why did it exercise a liefldd.py code path that none of the tests on CI did?

This was a manual build of conda-forge/lal-feedstock@4c08ede. I am trying to reproduce in a PR for visibility, but the Linux builds aren't installing the latest conda-build (see conda-forge/lal-feedstock#107). The macOS builds are, and are falling over the bugs reported above.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
type::bug describes erroneous operation, use severity::* to classify the type
Projects
Status: 🆕 New
Development

Successfully merging a pull request may close this issue.

5 participants