pytorch and abseil builds are incompatible on macOS #311

fxcoudert · 2025-01-03T15:35:14Z

Solution to issue cannot be found in the documentation.

I checked the documentation.

Issue

Latest PyTorch is currently unusable on macOS due to conda-forge/abseil-cpp-feedstock#87

% python   
Python 3.12.8 | packaged by conda-forge | (main, Dec  5 2024, 14:19:53) [Clang 18.1.8 ] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import torch
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/Users/fx/miniforge3/envs/test8/lib/python3.12/site-packages/torch/__init__.py", line 367, in <module>
    from torch._C import *  # noqa: F403
    ^^^^^^^^^^^^^^^^^^^^^^
ImportError: dlopen(/Users/fx/miniforge3/envs/test8/lib/python3.12/site-packages/torch/_C.cpython-312-darwin.so, 0x0002): Symbol not found: __ZN4absl12lts_2024072212log_internal10LogMessagelsIiLi0EEERS2_RKT_
  Referenced from: <7DD6F527-7A4A-3649-87B6-D68B25F8B594> /Users/fx/miniforge3/envs/test8/lib/libprotobuf.28.2.0.dylib
  Expected in:     <D623F952-8116-35EC-859D-F7F8D5DD7699> /Users/fx/miniforge3/envs/test8/lib/libabsl_log_internal_message.2407.0.0.dylib

The environment is created with:

conda create --name test8 python=3.12
conda activate test8
conda install pytorch

all with default (latest) versions.

Installed packages

# packages in environment at /Users/fx/miniforge3/envs/test8:
#
# Name                    Version                   Build  Channel
bzip2                     1.0.8                h99b78c6_7    conda-forge
ca-certificates           2024.12.14           hf0a4a13_0    conda-forge
cpython                   3.12.8          py312hd8ed1ab_1    conda-forge
filelock                  3.16.1             pyhd8ed1ab_1    conda-forge
fsspec                    2024.12.0          pyhd8ed1ab_0    conda-forge
gmp                       6.3.0                h7bae524_2    conda-forge
gmpy2                     2.1.5           py312h524cf62_3    conda-forge
jinja2                    3.1.5              pyhd8ed1ab_0    conda-forge
libabseil                 20240722.0      cxx17_h07bc746_2    conda-forge
libblas                   3.9.0           26_osxarm64_openblas    conda-forge
libcblas                  3.9.0           26_osxarm64_openblas    conda-forge
libcxx                    19.1.6               ha82da77_1    conda-forge
libexpat                  2.6.4                h286801f_0    conda-forge
libffi                    3.4.2                h3422bc3_5    conda-forge
libgfortran               5.0.0           13_2_0_hd922786_3    conda-forge
libgfortran5              13.2.0               hf226fd6_3    conda-forge
liblapack                 3.9.0           26_osxarm64_openblas    conda-forge
liblzma                   5.6.3                h39f12f2_1    conda-forge
libopenblas               0.3.28          openmp_hf332438_1    conda-forge
libprotobuf               5.28.2               h8f0b736_0    conda-forge
libsqlite                 3.47.2               h3f77e49_0    conda-forge
libtorch                  2.5.1           cpu_generic_h44abd74_7    conda-forge
libuv                     1.49.2               h7ab814d_0    conda-forge
libzlib                   1.3.1                h8359307_2    conda-forge
llvm-openmp               19.1.6               hdb05f8b_0    conda-forge
markupsafe                3.0.2           py312h998013c_1    conda-forge
mpc                       1.3.1                h8f1351a_1    conda-forge
mpfr                      4.2.1                hb693164_3    conda-forge
mpmath                    1.3.0              pyhd8ed1ab_1    conda-forge
ncurses                   6.5                  h7bae524_1    conda-forge
networkx                  3.4.2              pyh267e887_2    conda-forge
nomkl                     1.0                  h5ca1d4c_0    conda-forge
numpy                     2.2.1           py312h41c6370_0    conda-forge
openssl                   3.4.0                h39f12f2_0    conda-forge
pip                       24.3.1             pyh8b19718_2    conda-forge
python                    3.12.8          hc22306f_1_cpython    conda-forge
python_abi                3.12                    5_cp312    conda-forge
pytorch                   2.5.1           cpu_generic_py312_h89af673_7    conda-forge
readline                  8.2                  h92ec313_1    conda-forge
setuptools                75.6.0             pyhff2d567_1    conda-forge
sleef                     3.7                  h8391f65_2    conda-forge
sympy                     1.13.3           pyh2585a3b_104    conda-forge
tk                        8.6.13               h5083fa2_1    conda-forge
typing_extensions         4.12.2             pyha770c72_1    conda-forge
tzdata                    2024b                hc8b5060_0    conda-forge
wheel                     0.45.1             pyhd8ed1ab_1    conda-forge

Environment info

active environment : test8
    active env location : /Users/fx/miniforge3/envs/test8
            shell level : 2
       user config file : /Users/fx/.condarc
 populated config files : /Users/fx/miniforge3/.condarc
                          /Users/fx/.condarc
          conda version : 24.11.2
    conda-build version : not installed
         python version : 3.12.8.final.0
                 solver : libmamba (default)
       virtual packages : __archspec=1=m1
                          __conda=24.11.2=0
                          __osx=15.2=0
                          __unix=0=0
       base environment : /Users/fx/miniforge3  (writable)
      conda av data dir : /Users/fx/miniforge3/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/osx-arm64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/osx-arm64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/osx-arm64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /Users/fx/miniforge3/pkgs
                          /Users/fx/.conda/pkgs
       envs directories : /Users/fx/miniforge3/envs
                          /Users/fx/.conda/envs
               platform : osx-arm64
             user-agent : conda/24.11.2 requests/2.32.3 CPython/3.12.8 Darwin/24.2.0 OSX/15.2 solver/libmamba conda-libmamba-solver/24.11.1 libmambapy/2.0.5
                UID:GID : 502:20
             netrc file : None
           offline mode : False

The text was updated successfully, but these errors were encountered:

fxcoudert · 2025-01-03T15:37:50Z

I think this PR #310 could help by providing new builds (maybe with another re-render to update everything). But it was not run through CI: linux tests say "This request was automatically failed because there were no enabled runners online to process the request for more than 1 days."

mgorny · 2025-01-03T15:40:08Z

This should be addressed by #310.

hmaarrfk · 2025-01-03T15:45:07Z

what should OSX users do in the meantime?

matthewfeickert · 2025-01-03T19:57:09Z

what should OSX users do in the meantime?

A temporary workaround is to restrict libabseil (20240722.0) to build_number 1. With pixi this can be done with

pixi add pytorch "libabseil [build_number=1]"

and with conda this can be done with

conda install pytorch libabseil=20240722.0=cxx17_hf9b8971_1

(I know that the people in this thread already know how to restrict build numbers, but I'm posting for the benefit of people arriving here who don't and found their builds broken.)

* Restrict libabseil on macOS to 20240722.0 build_number 1 given bug. - c.f. conda-forge/pytorch-cpu-feedstock#311

hmaarrfk · 2025-01-03T20:12:55Z

perhaps we can write a repodata patch to add the constraint so that others get this automatically...

Tobias-Fischer · 2025-01-03T20:18:48Z

Note: this goes beyond pytorch, I've been contacted by a colleague that OpenCV fails:

E ImportError: dlopen(/Users/runner/micromamba/envs/myenv/lib/python3.10/site-packages/cv2/python-3.10/cv2.cpython-310-darwin.so, 0x0002): Symbol not found: __ZN4absl12lts_2024072212log_internal10LogMessagelsIiLi0EEERS2_RKT_ 
E Referenced from: <ABD8BED3-9FF3-30AE-9F2C-FB6F306CF934> /Users/runner/micromamba/envs/myenv/lib/libprotobuf.28.2.0.dylib 
E Expected in: <E99B267F-7144-31DD-A5D4-F5B0B82894DA> /Users/runner/micromamba/envs/myenv/lib/libabsl_log_internal_message.2407.0.0.dylib

matthewfeickert · 2025-01-03T21:44:27Z

perhaps we can write a repodata patch to add the constraint so that others get this automatically...

@hmaarrfk Would the repodata patch just be adding a disallow on libabseil build_number 2? I guess that's a patch like

if:
  name: pytorch-cpu
  subdir_in: ["osx-64", "osx-arm64"]
  timestamp_lt: 1735940588000
then:
  - replace_depends:
      old: libabseil
      new: libabseil !=20240722.0=cxx17_*_2

?
(I have something wrong with the syntax there, or conda doesn't allow that, as

micromamba install "libabseil !=20240722.0=cxx17_*_2"

installs 20240116.1.)

My understanding was that conda-forge/abseil-cpp-feedstock#88 fixes all of this and so for build_number 1 and build_numbers 3+ things will be fine, so disallowing a build is probably the way to go, correct?

peterbygrave · 2025-01-04T09:02:40Z

With conda-forge/abseil-cpp-feedstock#88 merged, I can confirm that fresh environment with simple import torch now works on osx-arm64. No need to add the libabseil pin anymore.

fxcoudert added the bug Something isn't working label Jan 3, 2025

sarthakpati mentioned this issue Jan 3, 2025

added recipe for gradient-agreement-filtering conda-forge/staged-recipes#28728

Open

10 tasks

matthewfeickert mentioned this issue Jan 3, 2025

macOS PyTorch build are broken matthewfeickert/pytorch-gpu-pixi-example#3

Closed

matthewfeickert mentioned this issue Jan 3, 2025

fix: Restrict libabseil on macOS given bug matthewfeickert/pytorch-gpu-pixi-example#4

Merged

matthewfeickert added a commit to matthewfeickert/pytorch-gpu-pixi-example that referenced this issue Jan 3, 2025

fix: Restrict libabseil on macOS given bug (#4)

c869743

* Restrict libabseil on macOS to 20240722.0 build_number 1 given bug. - c.f. conda-forge/pytorch-cpu-feedstock#311

fxcoudert closed this as completed Jan 4, 2025

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

pytorch and abseil builds are incompatible on macOS #311

pytorch and abseil builds are incompatible on macOS #311

fxcoudert commented Jan 3, 2025

fxcoudert commented Jan 3, 2025 •

edited

Loading

mgorny commented Jan 3, 2025

hmaarrfk commented Jan 3, 2025 •

edited

Loading

matthewfeickert commented Jan 3, 2025 •

edited

Loading

hmaarrfk commented Jan 3, 2025

Tobias-Fischer commented Jan 3, 2025

matthewfeickert commented Jan 3, 2025 •

edited

Loading

peterbygrave commented Jan 4, 2025

pytorch and abseil builds are incompatible on macOS #311

pytorch and abseil builds are incompatible on macOS #311

Comments

fxcoudert commented Jan 3, 2025

Solution to issue cannot be found in the documentation.

Issue

Installed packages

Environment info

fxcoudert commented Jan 3, 2025 • edited Loading

mgorny commented Jan 3, 2025

hmaarrfk commented Jan 3, 2025 • edited Loading

matthewfeickert commented Jan 3, 2025 • edited Loading

hmaarrfk commented Jan 3, 2025

Tobias-Fischer commented Jan 3, 2025

matthewfeickert commented Jan 3, 2025 • edited Loading

peterbygrave commented Jan 4, 2025

fxcoudert commented Jan 3, 2025 •

edited

Loading

hmaarrfk commented Jan 3, 2025 •

edited

Loading

matthewfeickert commented Jan 3, 2025 •

edited

Loading

matthewfeickert commented Jan 3, 2025 •

edited

Loading