Testing with CuPy 9

[jakirkham]

Currently CuPy is doing pre-releases for 9. Both prebuilt wheels and Conda packages exist. It would be good to try things out with the pre-releases and make sure things are looking ok.

Should add a lot of functionality has been filled out in cupyx.scipy.ndimage. So it might be worth checking if this unlocks anything that we were blocked on here previously. For example morphological operations ( #157 ) come to mind

cc @grlee77 (for awareness)

[GenevieveBuckley]

Are there CI services that will let us test with GPU? Or are you suggesting that we try and do some local testing?

I can merge #157 if you like. The stable release of cupy 9.0.0 is scheduled for April 2021, so I wasn't sure if it'd be confusing for people if that wasn't out yet.

[jakirkham]

Not really no. I think testing locally would make the most sense

We don't need to rush things. Though making sure those changes work with the current CuPy would be good. What happens if a user has an older version of CuPy installed and those functions are not present?

[GenevieveBuckley]

What happens if a user has an older version of CuPy installed and those functions are not present?

You get an error. I can't remember off the top of my head whether you have to wait until executing the task graph to find out about the problem or not.

[GenevieveBuckley]

I went to do this today, but I'm having a lot of trouble getting CUDA & cupy set up correctly. Would you maybe be able to help with that @jakirkham ? (Maybe email or whereby is a better format for that discussion, rather than here though)

[jakirkham]

Yep happy to help. Feel free to propose a time and we can go from there

[jni]

@jakirkham has nvidia open sourced its CI code/bots? I know you guys have CI testing integrated with github using your own machines. If you could make it easy for projects to set up their own CI on their own boxes, that might help adoption and testing throughout the ecosystem. Not to mention sell some extra GPUs on top. ;) Personally I have zero experience setting up something like that.

[GenevieveBuckley]

Emailed you a link re: scheduling @jakirkham

[jakirkham]

There should be a Conda package built with CuPy 9b3 with CUDA 11.2 support shortly (~1hr). Assuming conda-forge is already in one’s .condarc, can just do the following to install

conda install -c conda-forge/label/cupy_rc cudatoolkit=11.2 cupy=9.0.0b3

conda-forge/cupy-feedstock#106

[GenevieveBuckley]

Just tried it now & all the cupy tests we have pass locally on my machine with cupy=9.0.0b3 and the dask-image main branch.

============================= test session starts ==============================
platform linux -- Python 3.8.8, pytest-6.2.2, py-1.10.0, pluggy-0.13.1
rootdir: /home/genevieve/GitHub/dask-image, configfile: pytest.ini
collected 1422 items                                                           

tests/test_dask_image/test_imread/test_core.py ......................... [  1%]
........................................................................ [  6%]
..............                                                           [  7%]
tests/test_dask_image/test_imread/test_cupy_imread.py .                  [  7%]
tests/test_dask_image/test_ndfilters/test__conv.py ..................... [  9%]
.............................                                            [ 11%]
tests/test_dask_image/test_ndfilters/test__diff.py ..                    [ 11%]
tests/test_dask_image/test_ndfilters/test__edge.py ....................  [ 12%]
tests/test_dask_image/test_ndfilters/test__gaussian.py ................. [ 14%]
..s.......s............................................................. [ 19%]
........                                                                 [ 19%]
tests/test_dask_image/test_ndfilters/test__generic.py .................. [ 21%]
..............                                                           [ 22%]
tests/test_dask_image/test_ndfilters/test__order.py .................... [ 23%]
........................................................................ [ 28%]
....................................................................     [ 33%]
tests/test_dask_image/test_ndfilters/test__smooth.py ...............     [ 34%]
tests/test_dask_image/test_ndfilters/test__threshold.py ................ [ 35%]
....                                                                     [ 35%]
tests/test_dask_image/test_ndfilters/test__utils.py .................... [ 37%]
..........................                                               [ 38%]
tests/test_dask_image/test_ndfilters/test_cupy_ndfilters.py ............ [ 39%]
.....                                                                    [ 40%]
tests/test_dask_image/test_ndfilters/test_cupy_threshold.py ............ [ 40%]
........                                                                 [ 41%]
tests/test_dask_image/test_ndfourier/test__utils.py ..                   [ 41%]
tests/test_dask_image/test_ndfourier/test_core.py ...................... [ 43%]
........s.s..s.s..................................................       [ 47%]
tests/test_dask_image/test_ndinterp/test_affine_transformation.py ...... [ 48%]
........................................................................ [ 53%]
........................................................................ [ 58%]
........................................................................ [ 63%]
........................................................................ [ 68%]
........................................................................ [ 73%]
..                                                                       [ 73%]
tests/test_dask_image/test_ndmeasure/test__utils.py ...........          [ 74%]
tests/test_dask_image/test_ndmeasure/test_core.py ...................... [ 76%]
........................................................................ [ 81%]
.............s.........s................................................ [ 86%]
........................................................................ [ 91%]
................................                                         [ 93%]
tests/test_dask_image/test_ndmorph/test__utils.py ...................... [ 95%]
..............                                                           [ 96%]
tests/test_dask_image/test_ndmorph/test_ndmorph.py ..................... [ 97%]
...................................                                      [100%]

[GenevieveBuckley]

I'm going to close this issue (we have now tested dask-image locally with cupy version 9 and it all looks ok on the main branch).

Annoyingly, the tests I wrote for #157 are failing. That's an unmerged branch, so I'm going to add more details over there since that will need to be resolved before merging.

[jakirkham]

Thanks for doing that Genevieve! 😄 Glad you were able to test that out locally 🙂

[GenevieveBuckley]

dask/community#138 - Ben says:

We've been chatting with folks from the ops teams within RAPIDS about getting access to the gpuCI infrastructure. gpuCI is the GPU based CI platform used for testing throughout the RAPIDS ecosystem.