Repaired wheel can't find some of its included libraries

[OKaluza]

Hi, thanks for this really useful software. Unfortunately some of my wheel builds using cibuildwheel 2.12.1 with auditwheel 5.3.0 are broken and I can't find a way to fix them without manually patching the wheel or requiring LD_LIBRARY_PATH to be set:

Testing the package:

$ pip install lavavu-osmesa
...
$ python -m lavavu
Traceback (most recent call last):
  File "/usr/lib/python3.10/runpy.py", line 187, in _run_module_as_main
    mod_name, mod_spec, code = _get_module_details(mod_name, _Error)
  File "/usr/lib/python3.10/runpy.py", line 146, in _get_module_details
    return _get_module_details(pkg_main_name, error)
  File "/usr/lib/python3.10/runpy.py", line 110, in _get_module_details
    __import__(pkg_name)
  File "/home/owen/python-default/lib/python3.10/site-packages/lavavu/__init__.py", line 10, in <module>
    from .lavavu import *
  File "/home/owen/python-default/lib/python3.10/site-packages/lavavu/lavavu.py", line 77, in <module>
    import LavaVuPython
  File "/home/owen/python-default/lib/python3.10/site-packages/lavavu/LavaVuPython.py", line 15, in <module>
    import _LavaVuPython
ImportError: libffi-1c6807d3.so.6.0.4: cannot open shared object file: No such file or directory

Running ldd on the module shows three libraries not found:

~/python-default/lib/python3.10/site-packages/lavavu$ ldd _LavaVuPython.cpython-310-x86_64-linux-gnu.so
        linux-vdso.so.1 (0x00007fffa2eeb000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f7445f13000)
        libtiff-e2a5092b.so.5.2.6 => /home/owen/python-default/lib/python3.10/site-packages/lavavu/./../lavavu_osmesa.libs/libtiff-e2a5092b.so.5.2.6 (0x00007f7445400000)
(...cut)
        libffi-1c6807d3.so.6.0.4 => not found
        libedit-7f8577df.so.2.0.55 => not found
        libtinfo-33626a82.so.5.9 => not found
(...cut)

Yet, inspecting rpath is correct and ls shows the libraries exist there:

$ readelf -d _LavaVuPython.cpython-310-x86_64-linux-gnu.so |head -20

Dynamic section at offset 0x2b90000 contains 37 entries:
  Tag        Type                         Name/Value
 0x000000000000000f (RPATH)              Library rpath: [$ORIGIN:$ORIGIN/../lavavu_osmesa.libs]
(...cut)
$ ls ../lavavu_osmesa.libs/{libtinfo*,libffi*,libedit*}
../lavavu_osmesa.libs/libedit-7f8577df.so.2.0.55  ../lavavu_osmesa.libs/libffi-1c6807d3.so.6.0.4  ../lavavu_osmesa.libs/libtinfo-33626a82.so.5.9

None of these libraries are direct dependencies of my module, they seem to be dependencies of other dependencies (probably from libOSMesa as my main package/module without this library works fine). It seems those libraries aren't able to find their own dependencies in the same directory.
I can get around the issue by setting LD_LIBRARY_PATH=${HOME}/python-default/lib/python3.10/site-packages/lavavu_osmesa.libs or running patchelf --force-rpath --set-rpath '$ORIGIN' *.so* in the lavavu_osmesa.libs directory but I'd like a permanent fix in the wheel.
Am I doing something wrong or is this a bug? Any help would be greatly appreciated.

[lkollar]

Could you upload the bad wheel somewhere, or provide instructions to reproduce the issue?

[OKaluza]

If you try with the previous version $ pip install lavavu-osmesa==1.8.44 the issue above appears exactly as described, I tested just now with lavavu_osmesa-1.8.44-cp310-cp310-manylinux_2_24_x86_64.whl but if I remember correctly, all the manylinux wheels of the 1.8.44 release are broken.
The current release is working because I added a script to fix the broken wheels (https://github.com/lavavu/LavaVu/blob/master/patchwheel.py)

[lkollar]

auditwheel lddtree _LavaVuPython.cpython-310-x86_64-linux-gnu.so reveals that the library which needs the missing 3 .so files is libLLVM-3-6d00db57.8.so.1:

"libLLVM-3-6d00db57.8.so.1": {
            "realpath": "/tmp/lavavu/lavavu_osmesa.libs/libLLVM-3-6d00db57.8.so.1",
            "path": "/tmp/lavavu/lavavu_osmesa.libs/libLLVM-3-6d00db57.8.so.1",
            "needed": [
                "libffi-1c6807d3.so.6.0.4",
                "libedit-7f8577df.so.2.0.55",
                "librt.so.1",
                "libdl.so.2",
                "libtinfo-33626a82.so.5.9",
                "libpthread.so.0",
                "libz.so.1",
                "libstdc++.so.6",
                "libm.so.6",
                "libgcc_s.so.1",
                "libc.so.6",
                "ld-linux-x86-64.so.2"
            ]
        },

It seems that the RUNPATH in this library hasn't been changed by auditwheel:

❮ readelf -d /tmp/lavavu/lavavu_osmesa.libs/libLLVM-3-6d00db57.8.so.1 | grep RUNPATH
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../lib]

Stripping the RUNPATH entry fixes the issue and the missing libraries get picked up from the right place:

libffi-1c6807d3.so.6.0.4 => /tmp/lavavu/lavavu/./../lavavu_osmesa.libs/libffi-1c6807d3.so.6.0.4 (0x00007f2f8af6c000)
libedit-7f8577df.so.2.0.55 => /tmp/lavavu/lavavu/./../lavavu_osmesa.libs/libedit-7f8577df.so.2.0.55 (0x00007f2f8ad2e000)
libtinfo-33626a82.so.5.9 => /tmp/lavavu/lavavu/./../lavavu_osmesa.libs/libtinfo-33626a82.so.5.9 (0x00007f2f8ab02000)

I thought auditwheel stripped the RUNPATH entries in dependencies, but I might be wrong. We should probably update the RUNPATH in dependencies as well and point it to the vendor libs path.

@OKaluza I tried building the project here but it failed with a compiler error. I can give it another go when I get more time, but if you could upload the unrepaired wheel somewhere, that would help.

[OKaluza]

Ah sorry about that, attempting to prepare for distutils being deprecated and broke things, build should be fixed now.
Here is an unrepaired build from my machine: https://github.com/lavavu/LavaVu/releases/download/1.8.45/lavavu_osmesa-1.8.45-cp311-cp311-linux_x86_64.whl

[lkollar]

Thanks. I just checked this and I cannot reproduce the issue when repairing this wheel. I am repairing it for manylinux_2_35_x86_64 (this seems to be the lowest version for the required ABIs) and get a seemingly correctly repaired wheel with the three .so correctly located by ldd:

root@fb3d28a7ec3e unzip]# ldd lavavu/_LavaVuPython.cpython-311-x86_64-linux-gnu.so
...
        libGL.so.1 => /usr/lib/libGL.so.1 (0x00007fbe670b1000)
        libX11.so.6 => /usr/lib/libX11.so.6 (0x00007fbe66f6e000)
        libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x00007fbe66cf1000)
        libm.so.6 => /usr/lib/libm.so.6 (0x00007fbe66c04000)
        libgcc_s.so.1 => /usr/lib/libgcc_s.so.1 (0x00007fbe66bdf000)
        libc.so.6 => /usr/lib/libc.so.6 (0x00007fbe669f5000)
        libzstd-7da67a97.so.1.5.5 => /tmp/wheelhouse/unzip/lavavu/../lavavu_osmesa.libs/libzstd-7da67a97.so.1.5.5 (0x00007fbe6691c000)
        liblzma-57832d06.so.5.4.3 => /tmp/wheelhouse/unzip/lavavu/../lavavu_osmesa.libs/liblzma-57832d06.so.5.4.3 (0x00007fbe668e7000)
        libjpeg-77ed9821.so.8.2.2 => /tmp/wheelhouse/unzip/lavavu/../lavavu_osmesa.libs/libjpeg-77ed9821.so.8.2.2 (0x00007fbe6685f000)

However, libLLVM is not in the wheel. How exactly do you run auditwheel to get the libLLVM dependency vendored? I took a look at the GH actions in the project but couldn't see anything relevant.

I have the following files in lavavu_osmesa.libs:

libjpeg-77ed9821.so.8.2.2
liblzma-57832d06.so.5.4.3
libpng16-3b625471.so.16.39.0
libtiff-e33d88cd.so.6.0.0
libzstd-7da67a97.so.1.5.5

This is a much smaller set of libraries than in the wheel on PyPI so there is probably a difference in library dependencies in the build environment.

[OKaluza]

Sorry my mistake, it seems libOSMesa wasn't installed anymore on the machine I built it on, I rebuilt the wheel and it definitely links to libLLVM now: https://github.com/lavavu/LavaVu/releases/download/1.8.45/lavavu_osmesa-1.8.45-cp311-cp311-linux_x86_64.whl

$ ldd _LavaVuPython.cpython-311-x86_64-linux-gnu.so 
        linux-vdso.so.1 (0x00007ffd53861000)
        libpng16.so.16 => /lib/x86_64-linux-gnu/libpng16.so.16 (0x00007f97ffa70000)
        libz.so.1 => /lib/x86_64-linux-gnu/libz.so.1 (0x00007f97ffa52000)
        libtiff.so.6 => /lib/x86_64-linux-gnu/libtiff.so.6 (0x00007f97ff9c4000)
        libOSMesa.so.8 => /lib/x86_64-linux-gnu/libOSMesa.so.8 (0x00007f97fea00000)
        libstdc++.so.6 => /lib/x86_64-linux-gnu/libstdc++.so.6 (0x00007f97fe600000)
        libm.so.6 => /lib/x86_64-linux-gnu/libm.so.6 (0x00007f97fe917000)
        libgcc_s.so.1 => /lib/x86_64-linux-gnu/libgcc_s.so.1 (0x00007f97ff99e000)
        libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f97fe200000)
        libwebp.so.7 => /lib/x86_64-linux-gnu/libwebp.so.7 (0x00007f97ff591000)
        libzstd.so.1 => /lib/x86_64-linux-gnu/libzstd.so.1 (0x00007f97fe54c000)
        liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f97ff96c000)
        libLerc.so.4 => /lib/x86_64-linux-gnu/libLerc.so.4 (0x00007f97fe892000)
        libjbig.so.0 => /lib/x86_64-linux-gnu/libjbig.so.0 (0x00007f97ff959000)
        libjpeg.so.8 => /lib/x86_64-linux-gnu/libjpeg.so.8 (0x00007f97fe4c9000)
        libdeflate.so.0 => /lib/x86_64-linux-gnu/libdeflate.so.0 (0x00007f97ff57e000)
        libglapi.so.0 => /lib/x86_64-linux-gnu/libglapi.so.0 (0x00007f97fe48f000)
        libLLVM-15.so.1 => /lib/x86_64-linux-gnu/libLLVM-15.so.1 (0x00007f97f7000000)
        /lib64/ld-linux-x86-64.so.2 (0x00007f97ffac5000)
        libffi.so.8 => /lib/x86_64-linux-gnu/libffi.so.8 (0x00007f97fe887000)
        libedit.so.2 => /lib/x86_64-linux-gnu/libedit.so.2 (0x00007f97fe455000)
        libtinfo.so.6 => /lib/x86_64-linux-gnu/libtinfo.so.6 (0x00007f97fe423000)
        libxml2.so.2 => /lib/x86_64-linux-gnu/libxml2.so.2 (0x00007f97f6e16000)
        libbsd.so.0 => /lib/x86_64-linux-gnu/libbsd.so.0 (0x00007f97fe872000)
        libicuuc.so.72 => /lib/x86_64-linux-gnu/libicuuc.so.72 (0x00007f97f6c17000)
        libmd.so.0 => /lib/x86_64-linux-gnu/libmd.so.0 (0x00007f97fe416000)
        libicudata.so.72 => /lib/x86_64-linux-gnu/libicudata.so.72 (0x00007f97f4e00000)

The build command was:
LV_PACKAGE=lavavu-osmesa LV_OSMESA=1 LV_LIB_DIRS=/usr/local/lib python setup.py bdist_wheel
However unless libOSMesa is found it will still build but will skip this library and won't have the issue (can set LV_LIB_DIRS and LV_INC_DIRS to point to a specific build)