batch_norm layer isn't affected by the updated phase (train/test)

[yossibiton]

Issue summary

The parameter use_global_stats_ in batch_norm layer is defined automatically in layer setup, based on the current phase (train/test). see batch_norm_layer.cpp line 14 (false for train phase, true for test phase).

However the layer phase is dynamic and can be changed during train/test iterations.
For that reason, the correct behavior is to change use_global_stats_ value in forward & backward methods (based on the current phase_ value) instead of doing it only once in setup method.
This behavior is similar to what is done in dropout layer, which acts differently in train/test phase.

I will explain why this issue bothers me :
In my application, i use one prototxt network definition file and all layers are initialized in train mode. When i change the network phase from train the test i expect all layers to act respectively. However, the batch_norm layer doesn't change use_global_stats_ from false to true, so actually continue to act like it's in train phase.

[williford]

This is the behavior when use_global_stats_ is not specified. Users do not always want use_global_stats_ to be true during training and false during testing, for example, when you are fine-tuning a model and you only want use_global_stats_ to be true for part of the network.

It sounds like dynamically changing the phase is not supported and generally should not be done. You can create a training network and testing network and share the weights (https://groups.google.com/d/msg/caffe-users/pjqs-DE0eWw/VrM-dFvZLNMJ):

solver.test_nets[0].share_with(solver.net)

Quotes from the Caffe Canon* (#192):

Nets are now defined with a phase at instantiation so there isn't confusion and there is no switching during operation.

and (#1250):

The interface issue is addressed by #1728 but Net should still own phase and it should be set at initialization and not altered #1500.

Caffe Canon - is the standard collection of texts in the BVLC tradition that can be ascribed to the original BVLC members, as recorded by the Octocat.

[yossibiton]

Thanks a lot !
I didn't know the option of sharing weights, and using it will solve my problem.
If dynamic change of phase is not supported then my suggestion is irrelevant, your links helped me understand that.

[shelhamer]

Caffe Canon

@williford thanks for helping to collect and share the lore ☕