Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[v2.5.x] Fix stray bracket breaking pytest; fix include-patch for cross-compilation #346

Merged
merged 15 commits into from
Feb 11, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions recipe/bld.bat
Original file line number Diff line number Diff line change
Expand Up @@ -93,6 +93,7 @@ if not "%cuda_compiler_version%" == "None" (
set MAGMA_HOME=%LIBRARY_PREFIX%
set "PATH=%CUDA_BIN_PATH%;%PATH%"
set CUDNN_INCLUDE_DIR=%LIBRARY_PREFIX%\include
set "CUDA_VERSION=%cuda_compiler_version%"
@REM turn off very noisy nvcc warnings
set "CUDAFLAGS=-w --ptxas-options=-w"
) else (
Expand Down
1 change: 1 addition & 0 deletions recipe/build.sh
Original file line number Diff line number Diff line change
Expand Up @@ -219,6 +219,7 @@ elif [[ ${cuda_compiler_version} != "None" ]]; then
export USE_STATIC_CUDNN=0
export MAGMA_HOME="${PREFIX}"
export USE_MAGMA=1
export CUDA_VERSION=$cuda_compiler_version
# turn off noisy nvcc warnings
export CMAKE_CUDA_FLAGS="-w --ptxas-options=-w"
else
Expand Down
47 changes: 13 additions & 34 deletions recipe/meta.yaml
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# if you wish to build release candidate number X, append the version string with ".rcX"
{% set version = "2.5.1" %}
{% set build = 12 %}
{% set build = 13 %}

# Use a higher build number for the CUDA variant, to ensure that it's
# preferred by conda's solver, and it's preferentially
Expand Down Expand Up @@ -70,8 +70,7 @@ source:
- patches/0017-Add-conda-prefix-to-inductor-include-paths.patch
- patches/0018-make-ATEN_INCLUDE_DIR-relative-to-TORCH_INSTALL_PREF.patch
- patches/0019-remove-DESTINATION-lib-from-CMake-install-TARGETS-di.patch # [win]
- patches/0020-make-library-name-in-test_mutable_custom_op_fixed_la.patch
- patches/0021-avoid-deprecated-find_package-CUDA-in-caffe2-CMake-m.patch
- patches/0020-avoid-deprecated-find_package-CUDA-in-caffe2-CMake-m.patch
- patches_submodules/fbgemm/0001-remove-DESTINATION-lib-from-CMake-install-directives.patch # [win]
- patches_submodules/tensorpipe/0001-switch-away-from-find_package-CUDA.patch

Expand Down Expand Up @@ -397,7 +396,6 @@ outputs:
# tools/ is needed to optimise test run
# as of pytorch=2.0.0, there is a bug when trying to run tests without the tools
- tools
#- .ci/pytorch/smoke_test/smoke_test.py
commands:
# Run pip check so as to ensure that all pytorch packages are installed
# https://github.com/conda-forge/pytorch-cpu-feedstock/issues/24
Expand All @@ -409,10 +407,11 @@ outputs:
- python -c "import torch; import numpy"
- python -c "import numpy; import torch"
# distributed support is enabled by default on linux; for mac, we enable it manually in build.sh
- python -c "import torch; assert torch.distributed.is_available()" # [linux or osx]
- python -c "import torch; assert torch.backends.cuda.is_built()" # [linux64 and (cuda_compiler_version != "None")]
- python -c "import torch; assert torch.backends.cudnn.is_available()" # [linux64 and (cuda_compiler_version != "None")]
- python -c "import torch; assert torch.backends.cudnn.enabled" # [linux64 and (cuda_compiler_version != "None")]
- python -c "import torch; assert torch.distributed.is_available()" # [linux or osx]
- python -c "import torch; assert torch.backends.cuda.is_built()" # [cuda_compiler_version != "None"]
- python -c "import torch; assert torch.backends.cudnn.is_available()" # [cuda_compiler_version != "None"]
- python -c "import torch; assert torch.backends.cudnn.enabled" # [cuda_compiler_version != "None"]
- python -c "import torch; assert torch.version.cuda is not None" # [cuda_compiler_version != "None"]
# At conda-forge, we target versions of OSX that are too old for MPS support
# But if users install a newer version of OSX, they will have MPS support
# https://github.com/conda-forge/pytorch-cpu-feedstock/pull/123#issuecomment-1186355073
Expand All @@ -423,30 +422,6 @@ outputs:
- if not exist %LIBRARY_BIN%\torch_python.dll exit 1 # [win]
- if not exist %LIBRARY_LIB%\torch_python.lib exit 1 # [win]

# See here for environment variables needed by the smoke test script
# https://github.com/pytorch/pytorch/blob/266fd35c5842902f6304aa8e7713b252cbfb243c/.ci/pytorch/smoke_test/smoke_test.py#L16
- set MATRIX_GPU_ARCH_VERSION="{{ '.'.join((cuda_compiler_version or "").split('.')[:2]) }}" # [(cuda_compiler_version != "None") and (win)]
- set MATRIX_GPU_ARCH_TYPE="cuda" # [(cuda_compiler_version != "None") and (win)]
- set MATRIX_GPU_ARCH_VERSION="none" # [(cuda_compiler_version == "None") and (win)]
- set MATRIX_GPU_ARCH_TYPE="none" # [(cuda_compiler_version == "None") and (win)]
- set MATRIX_CHANNEL="defaults" # [win]
- set MATRIX_STABLE_VERSION={{ version }} # [win]
- set MATRIX_PACKAGE_TYPE="conda" # [win]
- set TARGET_OS="windows" # [win]
- set OMP_NUM_THREADS=4 # [win]
- export MATRIX_GPU_ARCH_VERSION="{{ '.'.join((cuda_compiler_version or "").split('.')[:2]) }}" # [(cuda_compiler_version != "None") and (linux and x86_64)]
- export MATRIX_GPU_ARCH_TYPE="cuda" # [(cuda_compiler_version != "None") and (linux and x86_64)]
- export MATRIX_GPU_ARCH_VERSION="none" # [(cuda_compiler_version == "None") and (not win)]
- export MATRIX_GPU_ARCH_TYPE="none" # [(cuda_compiler_version == "None") and (not win)]
- export MATRIX_CHANNEL="defaults" # [not win]
- export MATRIX_STABLE_VERSION="{{ version }}" # [not win]
- export MATRIX_PACKAGE_TYPE="conda" # [not win]
- export TARGET_OS="linux" # [linux]
- export TARGET_OS="macos-arm64" # [(osx and arm64)]
- export TARGET_OS="macos-x86_64" # [(osx and x86_64)]
- export OMP_NUM_THREADS=4 # [not win]
#- python ./smoke_test/smoke_test.py --package torchonly

# a reasonably safe subset of tests that should run under 15 minutes
{% set tests = " ".join([
"test/test_autograd.py",
Expand Down Expand Up @@ -485,7 +460,7 @@ outputs:
{% set skips = skips ~ " or (TestAutograd and test_profiler_propagation)" %}
# tests that fail due to resource clean-up issues (non-unique temporary libraries), see
# https://github.com/conda-forge/pytorch-cpu-feedstock/pull/318#issuecomment-2620080859
{% set skips = skips ~ " or test_mutable_custom_op_fixed_layout" %} # [cuda_compiler_version != "None"]
{% set skips = skips ~ " or test_mutable_custom_op_fixed_layout" %}
# trivial accuracy problems
{% set skips = skips ~ " or test_BCELoss_weights_no_reduce_cuda" %} # [unix and cuda_compiler_version != "None"]
{% set skips = skips ~ " or test_ctc_loss_cudnn_tensor_cuda " %} # [unix and cuda_compiler_version != "None"]
Expand All @@ -499,14 +474,18 @@ outputs:
{% set skips = skips ~ " or (GPUTests and test_scatter_reduce2)" %} # [linux and cuda_compiler_version != "None"]
# MKL problems
{% set skips = skips ~ " or (TestLinalgCPU and test_inverse_errors_large_cpu)" %} # [linux and blas_impl == "mkl" and cuda_compiler_version != "None"]
{% set skips = skips ~ " or test_reentrant_parent_error_on_cpu_cuda)" %} # [linux and blas_impl == "mkl" and cuda_compiler_version != "None"]
{% set skips = skips ~ " or test_reentrant_parent_error_on_cpu_cuda" %} # [linux and blas_impl == "mkl" and cuda_compiler_version != "None"]
# non-MKL problems
{% set skips = skips ~ " or test_cross_entropy_loss_2d_out_of_bounds_class_index_cuda" %} # [linux and blas_impl != "mkl" and cuda_compiler_version != "None"]
{% set skips = skips ~ " or test_cublas_config_nondeterministic_alert_cuda " %} # [linux and blas_impl != "mkl" and cuda_compiler_version != "None"]
# these tests are failing with low -n values
{% set skips = skips ~ " or test_base_does_not_require_grad_mode_nothing" %}
{% set skips = skips ~ " or test_base_does_not_require_grad_mode_warn" %}
{% set skips = skips ~ " or test_composite_registered_to_cpu_mode_nothing" %}
# torchinductor failures on osx
{% set skips = skips ~ " or test_AllenaiLongformerBase_repro_cpu" %} # [osx]
{% set skips = skips ~ " or (test_torchinductor and test_full_dtype)" %} # [osx]
{% set skips = skips ~ " or (test_torchinductor and test_neg_index_cpu)" %} # [osx]
# these tests are failing on windows
{% set skips = skips ~ " or (TestNN and test_Conv1d_dilated)" %} # [win]
{% set skips = skips ~ " or (TestNN and test_Conv1d_pad_same_dilated)" %} # [win]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From f3a0f9aab6dce56eea590b946f60256014b61bf7 Mon Sep 17 00:00:00 2001
From: Mark Harfouche <mark.harfouche@gmail.com>
Date: Sun, 1 Sep 2024 17:35:40 -0400
Subject: [PATCH 01/21] Force usage of python 3 and error without numpy
Subject: [PATCH 01/20] Force usage of python 3 and error without numpy

---
cmake/Dependencies.cmake | 6 +++---
Expand Down
2 changes: 1 addition & 1 deletion recipe/patches/0002-Help-find-numpy.patch
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 21c30036b5b86f403c0cf4426165d9a6a50edb1a Mon Sep 17 00:00:00 2001
From: Mark Harfouche <mark.harfouche@gmail.com>
Date: Tue, 1 Oct 2024 00:28:40 -0400
Subject: [PATCH 02/21] Help find numpy
Subject: [PATCH 02/20] Help find numpy

---
tools/setup_helpers/cmake.py | 6 ++++++
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From d1826af525db41eda5020a1404f5d5521d67a5dc Mon Sep 17 00:00:00 2001
From: Jeongseok Lee <jeongseok@meta.com>
Date: Sat, 19 Oct 2024 04:26:01 +0000
Subject: [PATCH 03/21] Add USE_SYSTEM_NVTX option (#138287)
Subject: [PATCH 03/20] Add USE_SYSTEM_NVTX option (#138287)

## Summary

Expand Down
2 changes: 1 addition & 1 deletion recipe/patches/0004-Update-sympy-version.patch
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From e3219c5fe8834753b0cf9e92be4d1ef1e874f370 Mon Sep 17 00:00:00 2001
From: Jeongseok Lee <jeongseok@meta.com>
Date: Thu, 17 Oct 2024 15:04:05 -0700
Subject: [PATCH 04/21] Update sympy version
Subject: [PATCH 04/20] Update sympy version

---
setup.py | 2 +-
Expand Down
2 changes: 1 addition & 1 deletion recipe/patches/0005-Fix-duplicate-linker-script.patch
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 08a1f44fbc81324aa98d720dfb7b87a261923ac2 Mon Sep 17 00:00:00 2001
From: Jeongseok Lee <jeongseok@meta.com>
Date: Sun, 3 Nov 2024 01:12:36 -0700
Subject: [PATCH 05/21] Fix duplicate linker script
Subject: [PATCH 05/20] Fix duplicate linker script

---
setup.py | 4 +++-
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 15df314a41c69a31c0443254d5552aa1b39d708d Mon Sep 17 00:00:00 2001
From: William Wen <williamwen@meta.com>
Date: Fri, 13 Sep 2024 13:02:33 -0700
Subject: [PATCH 06/21] fix 3.13 pickle error in serialization.py (#136034)
Subject: [PATCH 06/20] fix 3.13 pickle error in serialization.py (#136034)

Error encountered when adding dynamo 3.13 support.
Pull Request resolved: https://github.com/pytorch/pytorch/pull/136034
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 655f694854c3eafdd631235b60bc6c1b279218ed Mon Sep 17 00:00:00 2001
From: Mark Harfouche <mark.harfouche@gmail.com>
Date: Thu, 3 Oct 2024 22:49:56 -0400
Subject: [PATCH 07/21] Allow users to overwrite ld with environment variables
Subject: [PATCH 07/20] Allow users to overwrite ld with environment variables

This should help in the case of cross compilation.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From f03bf82d9da9cccb2cf4d4833c1a6349622dc37d Mon Sep 17 00:00:00 2001
From: =?UTF-8?q?Micha=C5=82=20G=C3=B3rny?= <mgorny@gentoo.org>
Date: Wed, 27 Nov 2024 13:47:23 +0100
Subject: [PATCH 08/21] Allow overriding CUDA-related paths
Subject: [PATCH 08/20] Allow overriding CUDA-related paths

---
cmake/Modules/FindCUDAToolkit.cmake | 2 +-
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 4b1faf6ba142953ce2730766db44f8d98d161ef0 Mon Sep 17 00:00:00 2001
From: Haifeng Jin <haifeng-jin@users.noreply.github.com>
Date: Tue, 1 Oct 2024 07:53:24 +0000
Subject: [PATCH 09/21] Fix test/test_linalg.py for NumPy 2 (#136800)
Subject: [PATCH 09/20] Fix test/test_linalg.py for NumPy 2 (#136800)

Related to #107302.

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 032b9be9ca7f9ae174e75554cecc82600ea3ef54 Mon Sep 17 00:00:00 2001
From: Haifeng Jin <haifeng-jin@users.noreply.github.com>
Date: Sat, 12 Oct 2024 02:40:17 +0000
Subject: [PATCH 10/21] Fixes NumPy 2 test failures in test_torch.py (#137740)
Subject: [PATCH 10/20] Fixes NumPy 2 test failures in test_torch.py (#137740)

Related to #107302

Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 56f1528fa072023fb2724d5abf8790f2f6cc3aaa Mon Sep 17 00:00:00 2001
From: Isuru Fernando <ifernando@quansight.com>
Date: Wed, 18 Dec 2024 03:59:00 +0000
Subject: [PATCH 11/21] Use BLAS_USE_CBLAS_DOT for OpenBLAS builds
Subject: [PATCH 11/20] Use BLAS_USE_CBLAS_DOT for OpenBLAS builds

There are two calling conventions for *dotu functions

Expand Down
2 changes: 1 addition & 1 deletion recipe/patches/0012-fix-issue-142484.patch
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From beba58d724cc1bd7ca73660b0a5ad9e61ae0c562 Mon Sep 17 00:00:00 2001
From: "Zheng, Zhaoqiong" <zhaoqiong.zheng@intel.com>
Date: Fri, 27 Dec 2024 13:49:36 +0800
Subject: [PATCH 12/21] fix issue 142484
Subject: [PATCH 12/20] fix issue 142484

From https://github.com/pytorch/pytorch/pull/143894
---
Expand Down
2 changes: 1 addition & 1 deletion recipe/patches/0013-Fix-FindOpenBLAS.patch
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 816a248a4425a97350959e412666e6db9012a52e Mon Sep 17 00:00:00 2001
From: Bas Zalmstra <bas@prefix.dev>
Date: Thu, 16 May 2024 10:46:49 +0200
Subject: [PATCH 13/21] Fix FindOpenBLAS
Subject: [PATCH 13/20] Fix FindOpenBLAS

---
cmake/Modules/FindOpenBLAS.cmake | 15 +++++++++------
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From db896f927403f55a18f931b18a6469cb4e37d322 Mon Sep 17 00:00:00 2001
From: atalman <atalman@fb.com>
Date: Tue, 12 Nov 2024 12:28:10 +0000
Subject: [PATCH 14/21] CD Enable Python 3.13 on windows (#138095)
Subject: [PATCH 14/20] CD Enable Python 3.13 on windows (#138095)

Adding CD windows. Part of: https://github.com/pytorch/pytorch/issues/130249
Builder PR landed with smoke test: https://github.com/pytorch/builder/pull/2035
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 33790dfbf966e7d8ea4ff6798d2ff92474d84079 Mon Sep 17 00:00:00 2001
From: "H. Vetinari" <h.vetinari@gmx.com>
Date: Thu, 23 Jan 2025 22:46:58 +1100
Subject: [PATCH 15/21] simplify torch.utils.cpp_extension.include_paths; use
Subject: [PATCH 15/20] simplify torch.utils.cpp_extension.include_paths; use
it in cpp_builder

The /TH headers have not existed since pytorch 1.11
Expand Down
23 changes: 16 additions & 7 deletions recipe/patches/0016-point-include-paths-to-PREFIX-include.patch
Original file line number Diff line number Diff line change
@@ -1,24 +1,33 @@
From 799f6fa59dac93dabbbcf72d46f4e1334e3d65d9 Mon Sep 17 00:00:00 2001
From 5266d900595715b58ab394a8102e722e17a6dd54 Mon Sep 17 00:00:00 2001
From: "H. Vetinari" <h.vetinari@gmx.com>
Date: Thu, 23 Jan 2025 22:58:14 +1100
Subject: [PATCH 16/21] point include paths to $PREFIX/include
Subject: [PATCH 16/20] point include paths to $PREFIX/include

---
torch/utils/cpp_extension.py | 9 +++++++++
1 file changed, 9 insertions(+)
torch/utils/cpp_extension.py | 18 ++++++++++++++++++
1 file changed, 18 insertions(+)

diff --git a/torch/utils/cpp_extension.py b/torch/utils/cpp_extension.py
index 3f584ef5598..4210f62b6db 100644
index 3f584ef5598..22da37092aa 100644
--- a/torch/utils/cpp_extension.py
+++ b/torch/utils/cpp_extension.py
@@ -1155,10 +1155,19 @@ def include_paths(cuda: bool = False) -> List[str]:
@@ -1155,10 +1155,28 @@ def include_paths(cuda: bool = False) -> List[str]:
A list of include path strings.
"""
lib_include = os.path.join(_TORCH_PATH, 'include')
+ if os.environ.get("CONDA_BUILD", None) is not None:
+ if (os.environ.get("CONDA_BUILD", None) is not None
+ and os.environ.get("CONDA_BUILD_CROSS_COMPILATION", None) not in (None, "", "0")):
+ # to avoid problems in cross-compilation, we need to point to the same environment
+ # where the currently running pytorch is -- i.e. the BUILD_PREFIX. See
+ # https://github.com/conda-forge/pytorch-cpu-feedstock/issues/349
+ pieces = [os.environ["BUILD_PREFIX"]] + IS_WINDOWS * ["Library"] + ["include"]
+ lib_include = os.path.join(*pieces)
+ elif os.environ.get("CONDA_BUILD", None) is not None:
+ # regular build (& testing) phase --> PREFIX is set
+ pieces = [os.environ["PREFIX"]] + IS_WINDOWS * ["Library"] + ["include"]
+ lib_include = os.path.join(*pieces)
+ elif os.environ.get("CONDA_PREFIX", None) is not None:
+ # final environment
+ pieces = [os.environ["CONDA_PREFIX"]] + IS_WINDOWS * ["Library"] + ["include"]
+ lib_include = os.path.join(*pieces)
paths = [
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 9f73a02bacf9680833ac64657fde6762d33ab200 Mon Sep 17 00:00:00 2001
From e8e76bd1733e1496ddb6818e26962964a508d39f Mon Sep 17 00:00:00 2001
From: Daniel Petry <dpetry@anaconda.com>
Date: Tue, 21 Jan 2025 17:45:23 -0600
Subject: [PATCH 17/21] Add conda prefix to inductor include paths
Subject: [PATCH 17/20] Add conda prefix to inductor include paths

Currently inductor doesn't look in conda's includes and libs. This results in
errors when it tries to compile, if system versions are being used of
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From b0cfa0f728e96a3a9d6f7434e2c02d74d6daa9a9 Mon Sep 17 00:00:00 2001
From 2b263d48975028456c3676ddf06ded74779e4a14 Mon Sep 17 00:00:00 2001
From: "H. Vetinari" <h.vetinari@gmx.com>
Date: Tue, 28 Jan 2025 14:15:34 +1100
Subject: [PATCH 18/21] make ATEN_INCLUDE_DIR relative to TORCH_INSTALL_PREFIX
Subject: [PATCH 18/20] make ATEN_INCLUDE_DIR relative to TORCH_INSTALL_PREFIX

we cannot set CMAKE_INSTALL_PREFIX without the pytorch build complaining, but we can
use TORCH_INSTALL_PREFIX, which is set correctly relative to our CMake files already:
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From f7db4cbfb0af59027ed8bdcd0387dba6fbcb1192 Mon Sep 17 00:00:00 2001
From 63329f38fc6d2d29eb88802f40db51f7080e6d67 Mon Sep 17 00:00:00 2001
From: "H. Vetinari" <h.vetinari@gmx.com>
Date: Tue, 28 Jan 2025 10:58:29 +1100
Subject: [PATCH 19/21] remove `DESTINATION lib` from CMake `install(TARGETS`
Subject: [PATCH 19/20] remove `DESTINATION lib` from CMake `install(TARGETS`
directives

Suggested-By: Silvio Traversaro <silvio@traversaro.it>
Expand Down
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
From 1780879024ea952f8591aa175a9787f93e697368 Mon Sep 17 00:00:00 2001
From fe088e2690abd96fcbc7b8f1190b343b16c6e9ee Mon Sep 17 00:00:00 2001
From: "H. Vetinari" <h.vetinari@gmx.com>
Date: Thu, 30 Jan 2025 08:33:44 +1100
Subject: [PATCH 21/21] avoid deprecated `find_package(CUDA)` in caffe2 CMake
Subject: [PATCH 20/20] avoid deprecated `find_package(CUDA)` in caffe2 CMake
metadata

vendor the not-available-anymore function torch_cuda_get_nvcc_gencode_flag from CMake
Expand Down
Loading