Skip unneeded forward computation during learning?

[longjon]

We have a pretty general mechanism to perform only the necessary backward steps during learning. Perhaps we should also skip unnecessary forward steps. It's easy enough to remove layers or use phases or stages to accomplish the same thing, but it might still be convenient to know that unnecessary computation is never being performed, without having to do anything.

This comes up, for example, when one has a net with multiple losses, and disables some of those losses (e.g., with loss_weight).

[jeffdonahue]

I sometimes use loss-less layers to display info like accuracy, so I'm not sure I'm in favor of disabling computation of paths that don't contribute to a loss by default. But if a branch of the net is used in neither to a loss nor a displayed output (i.e. it's probably capped with a SILENCE layer), it could be skipped. In that case it might be good to have a force_forward setting to compute the entire net activations, even if they're not needed. (And it would also be good to have per-layer granularity settings for force_forward and force_backward.)

[bhack]

How could be implemented also a "conditional" forwarding? I.e. It could be interesting to "unlock" the forward for a branch only if accuracy or other output layers (from other branch in the dag) values are greater than a threshold.

EDIT:
We have created with @mtamburrano a proof of concept of a conditional layer at #1448

[mtamburrano]

hi @longjon, we just committed on #1448 a way to block subsequent forward passes (and backward passes too) just setting the num of a Blob to 0.
Of course this is not exactly what you were looking for, but I think this is something similar, I'd like to hear your thoughts about that

[hyojinie]

@mtamburrano @longjon Was there any progress afterwards other than #2053? CuDNN layers do not seem to handle 0 batch size bottom blobs.

tatus == CUDNN_STATUS_SUCCESS (3 vs. 0) CUDNN_STATUS_BAD_PARAM
*** Check failure stack trace: ***
@ 0x7fd13693edaa (unknown)
@ 0x7fd13693ece4 (unknown)
@ 0x7fd13693e6e6 (unknown)
@ 0x7fd136941687 (unknown)
@ 0x7fd13700fb31 caffe::CuDNNSigmoidLayer<>::Reshape()
@ 0x7fd13712cbaa caffe::Net<>::ForwardFromTo()
@ 0x7fd13712cf47 caffe::Net<>::Forward()
@ 0x7fd136fbd1dc caffe::Solver<>::Test()
@ 0x7fd136fbdb0e caffe::Solver<>::TestAll()
@ 0x7fd136fbdc00 caffe::Solver<>::Step()
@ 0x7fd136fbe5c9 caffe::Solver<>::Solve()
@ 0x40873b train()
@ 0x405b3c main
@ 0x7fd13594af45 (unknown)
@ 0x4063ab (unknown)
@ (nil) (unknown)