bf16 & tf32，can they be used together

[DSXiangLi]

In the hyperparameter, I saw the bf16 and tf32 are both set to true. However I don't they can be used together?

[hujunchao]

I have the same question

[dotsnangles]

I was searching for the internet to find the answer. and then I found this thread.
is it possible though? I think the latter arg probably overwrites the former.
if someone could clarity this, I would be grateful.

this thread might help: huggingface/transformers#14608

[yxchng]

any new update on this?

[tdolega]

By setting bf16, the model weights are saved in bf16, but gradients (computed in half-precision but converted to full-precision for the update) and optimizations (optimizer states) are still done in full 32-bit precision. By additionally setting tf32, you speed up that very calculations at the expense of precision, but since ultimately those weights are saved as bf16 anyway, it probably has no effect on accuracy.