- Linux
- OSX
- Windows
When multiple wheels depend on each other and share some .so files,
it is possible to rely on auditwheel --exclude to make sure the .so
is included in only one of the wheels to avoid duplicating the file in every package.
That allows Python wheels to share the same .so as far as at least one of them
is loading them in memory. See pypa/auditwheel#76 and
pypa/auditwheel#368 for discussions on the topic.
The problem with this approach is that the package where the .so is included
will have the library name mangled by auditwheel, while the ones where it's
excluded will reference to the library with its original name.
Suppose you have libone.whl and libtwo.whl both depending on libfoo.so,
and libone is a dependency of libtwo so you already know you can include
libfoo.so only in libone.whl, you could do:
auditwheel repair dist/libone.whl auditwheel repair dist/libtwo.whl --exclude libfoo.so
In such case you would end up with the following wheels:
venv/lib/python3.10/site-packages/libone ├── __init__.py ├── _libone.cpython-310-aarch64-linux-gnu.so venv/lib/python3.10/site-packages/libone.libs └── libfoo-ef63151d.so venv/lib/python3.10/site-packages/libtwo ├── __init__.py ├── _libtwo.cpython-310-aarch64-linux-gnu.so
The problem would be that while _libone.cpython-310-aarch64-linux-gnu.so
was patched by auditwheel to know about the libfoo-ef63151d.so name,
_libtwo.cpython-310-aarch64-linux-gnu.so was not, and so still refers to the
original name:
$ ldd venv/lib/python3.10/site-packages/libone/_libone.cpython-310-aarch64-linux-gnu.so
libfoo-ef63151d.so => ../libone.libs/libfoo-ef63151d.so (0x0000ffff8f8f0000)
$ ldd venv/lib/python3.10/site-packages/libtwo/_libtwo.cpython-310-aarch64-linux-gnu.so
libfoo.so => not found
Which means that trying to import libtwo will fail with:
ImportError: libfoo.so: cannot open shared object file: No such file or directory
Which makes sense, because we actually provided libfoo-ef63151d.so and not libfoo.so.
To solve this problem consolidatewheels will patch all provided wheels to make sure that they
share a single naming convention for libraries that were mangled.
After consolidatewheels is used, the final result would be:
$ ldd venv/lib/python3.10/site-packages/libone/_libone.cpython-310-aarch64-linux-gnu.so
libfoo-ef63151d.so => ../libone.libs/libfoo-ef63151d.so (0x0000ffff8f8f0000)
$ ldd venv/lib/python3.10/site-packages/libtwo/_libtwo.cpython-310-aarch64-linux-gnu.so
libfoo-ef63151d.so => not found
which would work correctly as far as libone is imported _before_ libtwo as they will
both look for libfoo-ef63151d.so which was loaded already by libone.
consolidatewheels works also in conjunction with auditwheel, consolidating all libraries
embedded by auditwheel. But on Linux duplicates won't be removed automatically, so
you need to make sure to use auditwheel --exclude to ensure libraries are not embedded twice.
consolidatewheels works also in conjunction with delocate, consolidating all libraries
embedded by delocate and removing duplicates of the embedded libraries when they are provided
in multiple wheels.
consolidatewheels works also in conjunction with delvewheel, consolidating all libraries
embedded by delvewheel and removing duplicates of the embedded libraries when they are provided
in multiple wheels.
Install with:
$ pip install consolidatewheels
Note that consolidatewheels requires patchelf to be available in the system,
and it only works on Linux systems. But those are the same requirements that
auditwheel has, so you are probably already satisfying them if you use auditwheel.
Usage instructions:
consolidatewheels --help
Example:
consolidatewheels libone.whl libtwo.whl --dest=./consolidated_wheels
For a more complex example and a testing environment, you can take
a look at https://github.com/amol-/wheeldeps which uses consolidatewheels