[WIP] Use tensor random generator

[msperber]

Use tensor random generator instead of the standard one. This should fix speed issues on GPUs due to creating random numbers on CPU and then transferring to GPU, as was done previously.

I implemented this carefully, but it would be good if someone could review the code, as this involves core functionality and is hard to unit-test.

Some numbers on a CPU machine and a GPU machine:

e = dy.random_normal((1000,1000), batch_size=40)
e2 = dy.dropout(e, 0.5)
v = e2.value()

old, CPU: 8.45s
new, CPU: 8.08s
old, GPU: 6.33s
new, GPU: 3.48s

I suspect that most of the 3.48s were dynet overhead because in my particular use case dropout went from consuming 90% of computation time to <1%.

fixes #542, #438
related to #1059

• Initial implementation using Tensor random generator
• fix strange behavior on some GPUs

[neubig]

Thanks a lot! This is long-needed. I have a few comments though.

[neubig]

Do we need *1.f here?

[msperber]

This is fixed now.

[neubig]

I'm not a huge fan of this code being copied all the time. If this is something that we'll want to use frequently, perhaps it can be made a global variable in the DyNet namespace or something?

Also, tangentially, there are some random number generation functions in tensor.h/tensor.cc that it might be reasonable to move to their own rand.h/rand.cc files, along with this one. What do you think?

[msperber]

That sounds good, I moved the random helpers to their own files and added a draw_random_seed() helper to avoid code duplication.

[neubig]

Also, is there a reason why you didn't just implement all of this in TensorTools? It seems like that would work as well. If there's a reason that's no problem though.

[msperber]

I didn't do that partly because the TensorTools are also used for parameter initialization which I didn't want to touch, and partly because I didn't know how to properly move over the device-related code.

[neubig]

OK, I tested this on CPU and GPU with the current master branch and this PR using the attached script. It looks like master CPU/GPU and PR CPU are consistent, but the PR on GPU gives weird results. Could you try running the script and seeing what's up?

dynet-random.py.zip

[msperber]

Thanks for taking a look, that is strange indeed. random_normal() and random_uniform() look fine to me but bernoulli and dropout do not, which would hint at this line being broken on GPU:

m.tvec().device(*dev.edevice) = (m.tvec() < m.tvec().constant((1.f-p))).cast<float>() / (1.f-p);

The result of operator< is a Tensor<bool> which might potentially be a problem?

[neubig]

OK, I'm getting very strange behavior even on RandomNormal. It looks like almost exactly half the time it's sampling the exact same value, while half the time it's sampling from the appropriate distribution. This looks like a relatively major bug in the upstream Eigen implementation. I'll try to reproduce it only in Eigen and see if they're interested in fixing it.

[neubig]

FYI, we're having some discussions offline about how to implement this directly in cuRAND, which will presumably be more stable than relying on Eigen and potentially easier.

[neubig]

Closed in favor of #1154.