Implement full graph version (for tensorflow) of NAS-Interface

[Crysple]

Related issue #1159, used to run ENAS

Overview

In tensorflow, users need to build graph first and then create a session to run that graph. In the previous version, the graph of a trial will be determined (to be a sub-graph) once it receives a parameter configuration from the tuner. That is to say, the graph of this trial will not change even if it receives other parameter configurations in the future.

So in this version we will create and use tensorflow variable as signals, and tensorflow conditional functions to control the search space (full-graph) to be more flexible, which means it can be changed into different sub-graphs (multiple times) depending on these signals.

API Changed

Originally, users can insert annotation into their codes like this:

def conv(input, size=3, ch=64):
      ... ...
def pool(input):
      ... ...

out1 = conv(image, 3, 64)
out2 = conv(out1, 3, 64)
out3 = conv(out2, 3, 128)

#NNI Annotation
"""@nni.mutable_layers(
{
    layer_choice: [conv(ch=128), conv(ch=256), pool()],
    optional_inputs: [out3],
    optional_input_size: 1,
    layer_output: layer_out
}
)"""
Out = fc_1000(layer_out)

Now if their want to use tensorflow version, two changes should be made:

1. They need to specify the mode as 'oneshot-tf' in the mutable_layers function like this:

#NNI Annotation
"""@nni.mutable_layers(
{
    layer_choice: [conv(ch=128), conv(ch=256), pool()],
    optional_inputs: [out3],
    optional_input_size: 1,
    layer_output: layer_out
},
+ mode=oneshot-tf
)"""

2. They need to add nni.get_next_parameter(session) before they invoke the session.run function:

Note that they need to pass their tensorflow session as an arg into this function. An example might be:

for _ in range(num):
    """@nni.get_next_parameter(self.session)"""
    loss, _ = self.session.run([loss_op, train_op])
    """@nni.report_final_result(loss)"""

[xuehui1991]

please add more unit-test to make sure the function will get the expected result.