Distributed Training not starting

[bluesky314]

My code works and training starts with the progress bar with 1 gpu. However when I switch to 2, either using ddp or ddp_spawn the process gets stuck at:

Nothing happens but around 1.5 gb is taken on each gpu(the process takes around 8 on a single gpu)

Here is my code:

Confidential Code

System

Python 3.6 
Pytorch 1.6
Linux
pip installed lightning
CUDA Version: 10.1
Nvidia-V100

[Borda]

the example seems fine to me, mind share complete example even with dummy data, eg colab notebook?

[bluesky314]

Colab does not allow ddp I read in the docs. How can I give you a workable example then?

[Borda]

true, so jut share script so we can run it...

[awaelchli]

@bluesky314 yes please, if you can share the script. I will try to run it and help debug.
how do you launch your script? do you have CUDA_VISIBLE_DEVICES environment variable set?
Also, could you run with NCCL_DEBUG=INFO and post the info that is printed