Aborts at epoch 37

[LSnyd]

Hi all,
I am running a FL training with 3 clients and everything works well, until the server sends out a random abort signal to the clients every time at around epoch 37 (global). The log files of the clients and server do not show any errors beforehand.
I've seen in a different issue that a timeout may cause such aborts. However, the custom files that I am using (e.g. CifarLearner for instance) do not include any timeouts. Does anybody have an idea where the aborts may come from otherwise?

[nvkevlu]

Hi,

How are you running the FL training? The timeout is not in the custom files, but if you are using the FLAdminAPIRunner, which is used in https://github.com/NVIDIA/NVFlare/blob/main/examples/cifar10/run_fl.py#L60 for example, the timeout will cause the admin API to send the abort command to end the training.

[LSnyd]

Thanks so much, that solved my problem!