[tuner] Regularized Evolution

docs/en_US/NAS/ClassicNas.md

@@ -24,10 +24,10 @@ At this point, trial code is ready. Then, we can prepare an NNI experiment, i.e.
 
 A file named `nni_auto_gen_search_space.json` is generated by this command. Then put the path of the generated search space in the field `searchSpacePath` of the experiment config file. The other fields of the config file can be filled by referring [this tutorial](../Tutorial/QuickStart.md).
 
-Currently, we only support [PPO Tuner](../Tuner/BuiltinTuner.md) and [random tuner](https://github.com/microsoft/nni/tree/master/examples/tuners/random_nas_tuner) for classic NAS. More classic NAS algorithms will be supported soon.
+Currently, we only support [PPO Tuner](../Tuner/BuiltinTuner.md), [Evolution NAS Tuner](./EvolutionNasTuner) and [Random Tuner](https://github.com/microsoft/nni/tree/master/examples/tuners/random_nas_tuner) for classic NAS. More classic NAS algorithms will be supported soon.
 
 The complete examples can be found [here](https://github.com/microsoft/nni/tree/master/examples/nas/classic_nas) for PyTorch and [here](https://github.com/microsoft/nni/tree/master/examples/nas/classic_nas-tf) for TensorFlow.
 
 ## Standalone mode for easy debugging
 
-We support a standalone mode for easy debugging, where you can directly run the trial command without launching an NNI experiment. This is for checking whether your trial code can correctly run. The first candidate(s) are chosen for `LayerChoice` and `InputChoice` in this standalone mode.
+We support a standalone mode for easy debugging, where you can directly run the trial command without launching an NNI experiment. This is for checking whether your trial code can correctly run. The first candidate(s) are chosen for `LayerChoice` and `InputChoice` in this standalone mode.

docs/en_US/NAS/EvolutionNasTuner.md

@@ -0,0 +1,11 @@
+# Evolution NAS Tuner on NNI
+
+## EvoNasTuner
+
+This is a tuner geared for NNI’s Neural Architecture Search (NAS) interface. It uses the [evolution algorithm](https://arxiv.org/pdf/1802.01548.pdf).
+
+The tuner first randomly initializes the number of `population` models and evaluates them. After that, every time to produce a new architecture, the tuner randomly chooses the number of `sample` architectures from `population`, then mutates the best model in `sample`, the parent model, to produce the child model. The mutation includes the hidden mutation and the op mutation. The hidden state mutation consists of replacing a hidden state with another hidden state from within the cell, subject to the constraint that no loops are formed. The op mutation behaves like the hidden state mutation as far as replacing one op with another op from the op set. Note that keeping the child model the same as its parent is not allowed. After evaluating the child model, it is added to the tail of the `population`, then pops the front one.
+
+The whole procedure is summarized by the pseudocode below.
+
+![](../../img/EvoNasTuner.png)

docs/en_US/NAS/classic_nas.rst

@@ -0,0 +1,11 @@
+Classic NAS Algorithms
+=======================
+
+Classic NAS algorithms use a controller to ensemble a model from search space. The classic tuner algorithms train one model each time and use the reward of the model to guide the latter models. NNI has supported many popular classic NAS algorithms as following.
+
+
+..  toctree::
+    :maxdepth: 1
+
+    Overview <ClassicNas>
+    Evolution NAS Tuner <EvolutionNasTuner>

docs/en_US/nas.rst

@@ -19,7 +19,7 @@ For details, please refer to the following tutorials:
 
     Overview <NAS/Overview>
     Write A Search Space <NAS/WriteSearchSpace>
-    Classic NAS <NAS/ClassicNas>
+    Classic NAS <NAS/classic_nas>
     One-shot NAS <NAS/one_shot_nas>
     Customize a NAS Algorithm <NAS/Advanced>
     NAS Visualization <NAS/Visualization>

src/sdk/pynni/nni/constants.py

@@ -62,6 +62,11 @@
             'name': 'PBTTuner',
             'class_name': 'nni.pbt_tuner.pbt_tuner.PBTTuner',
             'class_args_validator': 'nni.pbt_tuner.pbt_tuner.PBTClassArgsValidator'
+        },
+        {
+            'name': 'EvolutionNasTuner',
+            'class_name': 'nni.evo_nas_tuner.evo_nas_tuner.EvoNasTuner',
+            'class_args_validator': 'nni.evo_nas_tuner.evo_nas_tuner.EvolutionClassArgsValidator'
         }
     ],
     'assessors': [

src/sdk/pynni/nni/evo_nas_tuner/__init__.py

@@ -0,0 +1 @@
+from .evo_nas_tuner import EvoNasTuner

src/sdk/pynni/nni/evo_nas_tuner/evo_nas_tuner.py

@@ -0,0 +1,169 @@
+import copy
+import logging
+import random
+from collections import deque
+
+import nni
+from schema import Schema, Optional
+from nni.tuner import Tuner
+from nni import ClassArgsValidator
+from nni.utils import OptimizeMode, extract_scalar_reward
+
+logger = logging.getLogger(__name__)
+
+
+class FinishedIndividual:
+    def __init__(self, parameter_id, parameters, result):
+        """
+        Parameters
+        ----------
+        parameter_id: int
+            the index of the parameter
+        parameters : dict
+            chosen architecture and parameters
+        result : float
+            final metric of the chosen one
+        """
+        self.parameter_id = parameter_id
+        self.parameters = parameters
+        self.result = result
+
+
+class EvolutionClassArgsValidator(ClassArgsValidator):
+    def validate_class_args(self, **kwargs):
+        Schema({
+            'optimize_mode': self.choices('optimize_mode', 'maximize', 'minimize'),
+            Optional('population_size'): self.range('population_size', int, 0, 99999),
+            Optional('sample_size'): self.range('sample_size', int, 0, 9999),
+        }).validate(kwargs)
+
+
+class EvoNasTuner(Tuner):
+    """
+    EvoNasTuner is tuner using Evolution NAS Tuner.
+    See ``Regularized Evolution for Image Classifier Architecture Search`` for details.
+
+    Parameters
+    ---
+    optimize_mode: str
+        whether to maximize metric or not. default: 'maximize'
+    population_size: int
+        the maximum number of keeping models
+    sample_size: int
+        the number of models chosen from population each time when evolution
+    """
+    def __init__(self, optimize_mode="maximize", population_size=100, sample_size=25):
+        super(EvoNasTuner, self).__init__()
+        self.optimize_mode = OptimizeMode(optimize_mode)
+        self.population_size = population_size
+        self.sample_size = sample_size
+        self.initial_population = deque()
+        self.population = deque()
+        self.history = {}
+        self.search_space = None
+
+    def generate_parameters(self, parameter_id, **kwargs):
+        """
+        This function will returns a dict of trial (hyper-)parameters, as a serializable object.
+
+        Parameters
+        ---
+        parameter_id: int
+            the index of current set of parameters
+        """
+        if self.initial_population:
+            arch = self.initial_population.popleft()
+            self.history[parameter_id] = arch
+            return arch
+        elif self.population:
+            sample = []
+            while len(sample) < self.sample_size:
+                sample.append(random.choice(list(self.population)))
+
+            candidate = max(sample, key=lambda x: x.result)
+            arch = self._mutate_model(candidate)
+            self.history[parameter_id] = arch
+            return arch
+        else:
+            raise nni.NoMoreTrialError
+
+    def receive_trial_result(self, parameter_id, parameters, value, **kwargs):
+        """
+        Record the result from a trial
+
+        Parameters
+        ----------
+        parameter_id : int
+        parameters : dict
+        value : dict/float
+            if value is dict, it should have "default" key.
+            value is final metrics of the trial.
+        """
+        reward = extract_scalar_reward(value)
+        if parameter_id not in self.history:
+            raise RuntimeError('Received parameter_id not in total_data.')
+        params = self.history[parameter_id]
+
+        if self.optimize_mode == OptimizeMode.Minimize:
+            reward = -reward
+
+        self.population.append(FinishedIndividual(parameter_id, params, reward))
+        if len(self.population) > self.population_size:
+            self.population.popleft()
+
+    def update_search_space(self, search_space):
+        """
+        Update search space.
+        Search_space contains the information that user pre-defined.
+
+        Parameters
+        ----------
+        search_space : dict
+        """
+        logger.info('update search space %s', search_space)
+        assert self.search_space is None
+        self.search_space = search_space
+
+        for _, val in search_space.items():
+            if val['_type'] != 'layer_choice' and val['_type'] != 'input_choice':
+                raise ValueError('Unsupported search space type: %s' % (val['_type']))
+
+        self._generate_initial_population()
+
+    def _random_model(self):
+        individual = {}
+        for key, val in self.search_space.items():
+            if val['_type'] == 'layer_choice':
+                idx = random.randint(0, len(val['_value']) - 1)
+                individual[key] = {'_value': val['_value'][idx], '_idx': idx}
+            elif val['_type'] == 'input_choice':
+                candidates = val['_value']['candidates']
+                n_chosen = val['_value']['n_chosen']
+                if n_chosen == None:
+                    raise RuntimeError('Key n_chosen must be set in InputChoice.')
+                idxs = [random.randint(0, len(candidates) - 1) for _ in range(n_chosen)]
+                vals = [candidates[k] for k in idxs]
+                individual[key] = {'_value': vals, '_idx': idxs}
+        return individual
+
+    def _mutate_model(self, model):
+        new_individual = copy.deepcopy(model.parameters)
+        mutate_key = random.choice(new_individual.keys())
+        mutate_val = self.search_space[mutate_key]
+        if mutate_val['_type'] == 'layer_choice':
+            idx = random.randint(0, len(mutate_val['_value']) - 1)
+            new_individual[mutate_key] = {'_value': mutate_val['_value'][idx], '_idx': idx}
+        elif mutate_val['_type'] == 'input_choice':
+            candidates = mutate_val['_value']['candidates']
+            n_chosen = mutate_val['_value']['n_chosen']
+            idxs = [random.randint(0, len(candidates) - 1) for _ in range(n_chosen)]
+            vals = [candidates[k] for k in idxs]
+            new_individual[mutate_key] = {'_value': vals, '_idx': idxs}
+        else:
+            raise KeyError
+        return new_individual
+
+    def _generate_initial_population(self):
+        while len(self.initial_population) < self.population_size:
+            self.initial_population.append(self._random_model())
+        logger.info('init population done.')

tools/nni_cmd/constants.py

@@ -89,7 +89,8 @@
     'MetisTuner',
     'BOHB',
     'SMAC',
-    'BatchTuner'
+    'BatchTuner',
+    'EvolutionNasTuner'
 }
 
 TUNERS_NO_NEED_TO_IMPORT_DATA = {