Implement Xavier distributions

[FL03]

XavierNormal

Each weight is drawn from a normal distribution where the mean $\left({\mu}\right)$ is set to 0 and the standard deviation $\left({\sigma}\right)$ is computed as follows:

$$\sigma = \sqrt{\frac{2}{d_{in} + d_{out}}}$$

XavierUniform

Each weight is drawn from a uniform distribution in $\left[{-x, x}\right]$

$$x = \sqrt{ \frac{ 6 }{ d_{in} + d_{out} } }$$

Resources

• Understanding the difficulty of training deep feedforward neural networks