Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

FFT/iFFT pytest cases fails on Win32+CUDA11.4 Cuda backend #413

Closed
TomonobuTsujikawa opened this issue Aug 9, 2022 · 0 comments
Closed

Comments

@TomonobuTsujikawa
Copy link
Contributor

We found that CUDA11.4 environment fails our pytest as follwoing:

nnabla\python\test\function\test_ifft.py:113: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
nnabla\python\test\nbla_test_utils.py:1064: in backward_function_tester
    sum_ograd.forward(clear_no_need_grad=True)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _
>   ???
E   RuntimeError: target_specific error in nbla::exec_cufft
E   C:/gl/builds/qXMMCsJN/1/nnabla/builders/all/nnabla-ext-cuda/include\nbla/cuda/function/utils/fft.cuh:294
E   `cufftXtMakePlanMany(plan, rank, n.data(), inembed.data(), istride, idist, input_type, onembed.data(), ostride, odist, output_type, batch, &work_size, execution_type)` failed with CUFFT_INVALID_PLAN.
_variable.pyx:582: RuntimeError

We got CUFFT_INVALID_PLAN error, but we have not found any suspicious points in our code.
So, now we're checking to see what's causing the problem, including external libraries.
One possibility is related to cufft library:
https://docs.nvidia.com/cuda/cuda-toolkit-release-notes/index.html#title-cufft-library

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant