The optimizer may track some values across different training routines

[DavdGao]

• Sine the optimizer is initialized within context, different training routines will use the same optimizer.
• Some optimizers, like Adam, will track the past momentum.

Therefore, the optimizer may track some state variables across different training routines. Considering the initialized model of each training routine is broadcast by the server, maybe it is unnecessary or even wrong to track past variables.