Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use the same CUDNN version on Windows as PyTorch #7635

Merged
merged 3 commits into from
May 27, 2023

Conversation

huydhn
Copy link
Contributor

@huydhn huydhn commented May 26, 2023

11.7 uses 8.5.0; 11.8 uses 8.7.0; 12.1 uses 8.8.1. Otherwise, Windows vision job (8.5.0) would overwrite the CUDNN version setup by PyTorch (8.7.0) leading to this flaky failures https://github.com/pytorch/pytorch/actions/runs/5088860652/jobs/9146641450

RuntimeError: cuDNN version incompatibility: PyTorch was compiled  against (8, 7, 0) but found runtime version (8, 5, 0). PyTorch already comes bundled with cuDNN. One option to resolving this error is to ensure PyTorch can find the bundled cuDNN.

11.7 uses 8.5.0; 11.8 uses 8.7.0; 12.1 uses 8.8.1
@huydhn huydhn requested a review from atalman May 26, 2023 19:00
@pytorch-bot
Copy link

pytorch-bot bot commented May 26, 2023

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/vision/7635

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 1 Pending

As of commit 2441e59:

NEW FAILURE - The following job has failed:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

Copy link
Collaborator

@pmeier pmeier left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The only time we are hitting this script is here:

# FIXME: Basically, we are reinstalling CUDA here. We only need this, because we need to copy some files that
# can be extracted from the CUDA installer, but are not available on our Windows AMI.
# See https://github.com/pytorch/test-infra/pull/4189
if [[ ${{ matrix.gpu-arch-type }} == cuda ]]; then
export CU_VERSION=cu$(echo ${{ matrix.gpu-arch-version }} | sed 's/\.//')
echo CU_VERSION="${CU_VERSION}"
packaging/windows/internal/cuda_install.bat
fi

@atalman and I are in the process to act on the FIXME so that TorchVision can just rely on the files already present in the AMI. See pytorch/test-infra#4227.

Stamp to unblock in case that takes too long.

@huydhn
Copy link
Contributor Author

huydhn commented May 26, 2023

The only time we are hitting this script is here:

# FIXME: Basically, we are reinstalling CUDA here. We only need this, because we need to copy some files that
# can be extracted from the CUDA installer, but are not available on our Windows AMI.
# See https://github.com/pytorch/test-infra/pull/4189
if [[ ${{ matrix.gpu-arch-type }} == cuda ]]; then
export CU_VERSION=cu$(echo ${{ matrix.gpu-arch-version }} | sed 's/\.//')
echo CU_VERSION="${CU_VERSION}"
packaging/windows/internal/cuda_install.bat
fi

@atalman and I are in the process to act on the FIXME so that TorchVision can just rely on the files already present in the AMI. See pytorch/test-infra#4227.

Stamp to unblock in case that takes too long.

Thank you for the context. It's indeed nicer to not need to do the installation step during CI.

@huydhn huydhn merged commit 01b9faa into pytorch:main May 27, 2023
@github-actions
Copy link

Hey @huydhn!

You merged this PR, but no labels were added. The list of valid labels is available at https://github.com/pytorch/vision/blob/main/.github/process_commit.py

facebook-github-bot pushed a commit that referenced this pull request May 31, 2023
Reviewed By: vmoens

Differential Revision: D46314037

fbshipit-source-id: a5744835186bd6199f7b11e65c70f04107d587fd
@huydhn huydhn deleted the windows-cudnn-8.7.0 branch August 30, 2023 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants