Dropout speed issues

[telmop]

Hi there!

I have been experiencing some issues using dropout. The thing is that it makes my system many times slower to train. Before dropout it took ~6h in GPU to run a full epoch. Now, it takes ~24h. That's a 4 fold increase. The only change in my code was do use dropout in the hidden layers of the net. Also, GPU usage is now way down, and a lot of time it actually reaches 0, which didn't happen before.

Any clues?

[pmichel31415]

Hmm that is interesting, I certainly never had such slowdown with dropout. Maybe this issue is linked to this one #438 ?

[telmop]

That seems a plausible explanation.

[pmichel31415]

I will see if I can address this this week

[pmichel31415]

Could you provide some minimal reproducible code so that I can test the changes?

[telmop]

I just used the standard dropout. i.e., something like this:

import dynet as dy
...
y_dropped = dy.dropout(y, dropout_prob)

[pmichel31415]

So upon more discussion, it seems that this problem needs some work upstream in eigen, so it'll probably take more time to fix.

[telmop]

Ok, thanks.