a simple debug tool for general nas programming interface (#1147)

* Dev nas interface -- document (#1049)

* nas interface doc

* Dev nas compile -- code generator (#1067)

* finish code for parsing mutable_layers annotation and testcode

* Dev nas interface -- update figures (#1070)

 update figs

* update searchspace_generator (#1071)

* GeneralNasInterfaces.md: Fix a typo (#1079)

Signed-off-by: Ce Gao <gaoce@caicloud.io>

* add NAS example and fix bugs (#1083)

update searchspace_generator, add example, update NAS example

* fix bugs (#1108)

* nas example

* fix bugs

* remove

* update

* debug

* fix bug

* remove previous mnist.py

* rename

* code gen for specific trial

* fix conflict

* remove print

* add print warning

* update doc

* update doc

* update doc

* remove comment

* update doc

* remove unnecessary global

docs/en_US/GeneralNasInterfaces.md

@@ -1,4 +1,6 @@
-# General Programming Interface for Neural Architecture Search
+# General Programming Interface for Neural Architecture Search (experimental feature)
+
+_*This is an experimental feature, currently, we only implemented the general NAS programming interface. Weight sharing and one-shot NAS based on this programming interface will be supported in the following releases._
 
 Automatic neural architecture search is taking an increasingly important role on finding better models. Recent research works have proved the feasibility of automatic NAS, and also found some models that could beat manually designed and tuned models. Some of representative works are [NASNet][2], [ENAS][1], [DARTS][3], [Network Morphism][4], and [Evolution][5]. There are new innovations keeping emerging. However, it takes great efforts to implement those algorithms, and it is hard to reuse code base of one algorithm for implementing another.
 
@@ -24,6 +26,8 @@ When designing the following model there might be several choices in the fourth
 
 There are two ways to write annotation for this example. For the upper one, `input` of the function calls is `[[],[out3]]`. For the bottom one, `input` is `[[out3],[]]`.
 
+__Debugging__: We provided an `nnictl trial codegen` command to help debugging your code of NAS programming on NNI. If your trial with trial_id `XXX` in your experiment `YYY` is failed, you could run `nnictl trial codegen YYY --trial_id XXX` to generate an executable code for this trial under your current directory. With this code, you can directly run the trial command without NNI to check why this trial is failed. Basically, this command is to compile your trial code and replace the NNI NAS code with the real chosen layers and inputs.
+
 ### Example: choose input connections for a layer
 
 Designing connections of layers is critical for making a high performance model. With our provided interface, users could annotate which connections a layer takes (as inputs). They could choose several ones from a set of connections. Below is an example which chooses two inputs from three candidate inputs for `concat`. Here `concat` always takes the output of its previous layer using `fixed_inputs`.
@@ -92,19 +96,19 @@ NNI's annotation compiler transforms the annotated trial code to the code that c
 
 The above figure shows how the trial code runs on NNI. `nnictl` processes user trial code to generate a search space file and compiled trial code. The former is fed to tuner, and the latter is used to run trials. 
 
-[__TODO__] Simple example of NAS on NNI.
+[Simple example of NAS on NNI](https://github.com/microsoft/nni/tree/v0.8/examples/trials/mnist-nas).
 
-### Weight sharing
+### [__TODO__] Weight sharing
 
 Sharing weights among chosen architectures (i.e., trials) could speedup model search. For example, properly inheriting weights of completed trials could speedup the converge of new trials. One-Shot NAS (e.g., ENAS, Darts) is more aggressive, the training of different architectures (i.e., subgraphs) shares the same copy of the weights in full graph.
 
 ![](../img/nas_weight_share.png)
 
 We believe weight sharing (transferring) plays a key role on speeding up NAS, while finding efficient ways of sharing weights is still a hot research topic. We provide a key-value store for users to store and load weights. Tuners and Trials use a provided KV client lib to access the storage.
 
-[__TODO__] Example of weight sharing on NNI.
+Example of weight sharing on NNI.
 
-### Support of One-Shot NAS
+### [__TODO__] Support of One-Shot NAS
 
 One-Shot NAS is a popular approach to find good neural architecture within a limited time and resource budget. Basically, it builds a full graph based on the search space, and uses gradient descent to at last find the best subgraph. There are different training approaches, such as [training subgraphs (per mini-batch)][1], [training full graph through dropout][6], [training with architecture weights (regularization)][3]. Here we focus on the first approach, i.e., training subgraphs (ENAS).
 
@@ -114,18 +118,18 @@ With the same annotated trial code, users could choose One-Shot NAS as execution
 
 The design of One-Shot NAS on NNI is shown in the above figure. One-Shot NAS usually only has one trial job with full graph. NNI supports running multiple such trial jobs each of which runs independently. As One-Shot NAS is not stable, running multiple instances helps find better model. Moreover, trial jobs are also able to synchronize weights during running (i.e., there is only one copy of weights, like asynchroneous parameter-server mode). This may speedup converge.
 
-[__TODO__] Example of One-Shot NAS on NNI.
+Example of One-Shot NAS on NNI.
 
 
-## General tuning algorithms for NAS
+## [__TODO__] General tuning algorithms for NAS
 
 Like hyperparameter tuning, a relatively general algorithm for NAS is required. The general programming interface makes this task easier to some extent. We have a RL-based tuner algorithm for NAS from our contributors. We expect efforts from community to design and implement better NAS algorithms.
 
-[__TODO__] More tuning algorithms for NAS.
+More tuning algorithms for NAS.
 
-## Export best neural architecture and code
+## [__TODO__] Export best neural architecture and code
 
-[__TODO__] After the NNI experiment is done, users could run `nnictl experiment export --code` to export the trial code with the best neural architecture.
+After the NNI experiment is done, users could run `nnictl experiment export --code` to export the trial code with the best neural architecture.
 
 ## Conclusion and Future work
 

docs/en_US/advanced.rst

@@ -3,4 +3,5 @@ Advanced Features
 
 ..  toctree::
     MultiPhase<MultiPhase>
-    AdvancedNas<AdvancedNas>
+    AdvancedNas<AdvancedNas>
+    NAS Programming Interface<GeneralNasInterfaces>

examples/trials/mnist-nas/mnist.py

@@ -9,7 +9,6 @@
 import tensorflow as tf
 from tensorflow.examples.tutorials.mnist import input_data
 
-import nni
 import operators as op
 
 FLAGS = None
@@ -215,7 +214,7 @@ def main(params):
                                mnist_network.labels: mnist.test.labels,
                                mnist_network.keep_prob: 1.0})
 
-                nni.report_intermediate_result(test_acc)
+                """@nni.report_intermediate_result(test_acc)"""
                 logger.debug('test accuracy %g', test_acc)
                 logger.debug('Pipe send intermediate result done.')
 
@@ -224,7 +223,7 @@ def main(params):
                        mnist_network.labels: mnist.test.labels,
                        mnist_network.keep_prob: 1.0})
 
-        nni.report_final_result(test_acc)
+        """@nni.report_final_result(test_acc)"""
         logger.debug('Final result is %g', test_acc)
         logger.debug('Send final result done.')
 

tools/nni_annotation/__init__.py

@@ -22,9 +22,11 @@
 import os
 import sys
 import shutil
+import json
 
 from . import code_generator
 from . import search_space_generator
+from . import specific_code_generator
 
 
 __all__ = ['generate_search_space', 'expand_annotations']
@@ -74,7 +76,7 @@ def _generate_file_search_space(path, module):
     return search_space
 
 
-def expand_annotations(src_dir, dst_dir):
+def expand_annotations(src_dir, dst_dir, exp_id='', trial_id=''):
     """Expand annotations in user code.
     Return dst_dir if annotation detected; return src_dir if not.
     src_dir: directory path of user code (str)
@@ -93,11 +95,23 @@ def expand_annotations(src_dir, dst_dir):
         dst_subdir = src_subdir.replace(src_dir, dst_dir, 1)
         os.makedirs(dst_subdir, exist_ok=True)
 
+        # generate module name from path
+        if src_subdir == src_dir:
+            package = ''
+        else:
+            assert src_subdir.startswith(src_dir + slash), src_subdir
+            prefix_len = len(src_dir) + 1
+            package = src_subdir[prefix_len:].replace(slash, '.') + '.'
+
         for file_name in files:
             src_path = os.path.join(src_subdir, file_name)
             dst_path = os.path.join(dst_subdir, file_name)
             if file_name.endswith('.py'):
-                annotated |= _expand_file_annotations(src_path, dst_path)
+                if trial_id == '':
+                    annotated |= _expand_file_annotations(src_path, dst_path)
+                else:
+                    module = package + file_name[:-3]
+                    annotated |= _generate_specific_file(src_path, dst_path, exp_id, trial_id, module)
             else:
                 shutil.copyfile(src_path, dst_path)
 
@@ -121,3 +135,22 @@ def _expand_file_annotations(src_path, dst_path):
                 raise RuntimeError(src_path + ' ' + '\n'.join(str(arg) for arg in exc.args))
             else:
                 raise RuntimeError('Failed to expand annotations for %s: %r' % (src_path, exc))
+
+def _generate_specific_file(src_path, dst_path, exp_id, trial_id, module):
+    with open(src_path) as src, open(dst_path, 'w') as dst:
+        try:
+            with open(os.path.expanduser('~/nni/experiments/%s/trials/%s/parameter.cfg'%(exp_id, trial_id))) as fd:
+                para_cfg = json.load(fd)
+            annotated_code = specific_code_generator.parse(src.read(), para_cfg["parameters"], module)
+            if annotated_code is None:
+                shutil.copyfile(src_path, dst_path)
+                return False
+            dst.write(annotated_code)
+            return True
+
+        except Exception as exc:  # pylint: disable=broad-except
+            if exc.args:
+                raise RuntimeError(src_path + ' ' + '\n'.join(str(arg) for arg in exc.args))
+            else:
+                raise RuntimeError('Failed to expand annotations for %s: %r' % (src_path, exc))
+