FP32 training provides blurry and incorrect resulting images

[enrique-torres]

I've been trying to train the models on the wojtek data, on FP32, since I want to gather data for a research project and fully fused MLPs obscure weight data. However, when I train on FP32, the resulting images after evaluation and testing are blurry and are not the same as the images produced by the TCNN, FP16 and fully fused model. Is this behaviour expected, or is something wrong with the code?