Merge pull request #252 from microsoft/master

merge master

docs/en_US/Assessor/BuiltinAssessor.md

@@ -55,13 +55,15 @@ assessor:
 
 It's applicable in a wide range of performance curves, thus, it can be used in various scenarios to speed up the tuning progress. Even better, it's able to handle and assess curves with similar performance. [Detailed Description](./CurvefittingAssessor.md)
 
+**Note**, according to the original paper, only incremental functions are supported. Therefore this assessor can only be used to maximize optimization metrics. For example, it can be used for accuracy, but not for loss.
+
+
 **classArgs requirements:**
 
 * **epoch_num** (*int, **required***) - The total number of epochs. We need to know the number of epochs to determine which points we need to predict.
-* **optimize_mode** (*maximize or minimize, optional, default = maximize*) - If 'maximize', assessor will **stop** the trial with smaller expectation. If 'minimize', assessor will **stop** the trial with larger expectation.
 * **start_step** (*int, optional, default = 6*) - A trial is determined to be stopped or not only after receiving start_step number of reported intermediate results.
-* **threshold** (*float, optional, default = 0.95*) - The threshold that we use to decide to early stop the worst performance curve. For example: if threshold = 0.95, optimize_mode = maximize, and the best performance in the history is 0.9, then we will stop the trial who's predicted value is lower than 0.95 * 0.9 = 0.855.
-* **gap** (*int, optional, default = 1*) - The gap interval between Assesor judgements. For example: if gap = 2, start_step = 6, then we will assess the result when we get 6, 8, 10, 12...intermediate results.
+* **threshold** (*float, optional, default = 0.95*) - The threshold that we use to decide to early stop the worst performance curve. For example: if threshold = 0.95, and the best performance in the history is 0.9, then we will stop the trial who's predicted value is lower than 0.95 * 0.9 = 0.855.
+* **gap** (*int, optional, default = 1*) - The gap interval between Assessor judgements. For example: if gap = 2, start_step = 6, then we will assess the result when we get 6, 8, 10, 12...intermediate results.
 
 **Usage example:**
 
@@ -71,8 +73,7 @@ assessor:
     builtinAssessorName: Curvefitting
     classArgs:
       epoch_num: 20
-      optimize_mode: maximize
       start_step: 6
       threshold: 0.95
       gap: 1
-```
+```

docs/en_US/Assessor/CurvefittingAssessor.md

@@ -1,20 +1,20 @@
-Curve Fitting Assessor on NNI
-===
+# Curve Fitting Assessor on NNI
+
+## Introduction
 
-## 1. Introduction
 The Curve Fitting Assessor is an LPA (learning, predicting, assessing) algorithm. It stops a pending trial X at step S if the prediction of the final epoch's performance is worse than the best final performance in the trial history.
 
 In this algorithm, we use 12 curves to fit the learning curve. The set of parametric curve models are chosen from this [reference paper][1]. The learning curves' shape coincides with our prior knowledge about the form of learning curves: They are typically increasing, saturating functions.
 
-![](../../img/curvefitting_learning_curve.PNG)
+![learning_curve](../../img/curvefitting_learning_curve.PNG)
 
 We combine all learning curve models into a single, more powerful model. This combined model is given by a weighted linear combination:
 
-![](../../img/curvefitting_f_comb.gif)
+![f_comb](../../img/curvefitting_f_comb.gif)
 
 with the new combined parameter vector
 
-![](../../img/curvefitting_expression_xi.gif)
+![expression_xi](../../img/curvefitting_expression_xi.gif)
 
 Assuming additive Gaussian noise and the noise parameter being initialized to its maximum likelihood estimate.
 
@@ -30,44 +30,46 @@ Concretely, this algorithm goes through three stages of learning, predicting, an
 
 The figure below is the result of our algorithm on MNIST trial history data, where the green point represents the data obtained by Assessor, the blue point represents the future but unknown data, and the red line is the Curve predicted by the Curve fitting assessor.
 
-![](../../img/curvefitting_example.PNG)
+![examples](../../img/curvefitting_example.PNG)
+
+## Usage
 
-## 2. Usage
 To use Curve Fitting Assessor, you should add the following spec in your experiment's YAML config file:
 
-```
+```yaml
 assessor:
-    builtinAssessorName: Curvefitting
-    classArgs:
-      # (required)The total number of epoch.
-      #  We need to know the number of epoch to determine which point we need to predict.
-      epoch_num: 20
-      # (optional) choice: maximize, minimize
-      * The default value of optimize_mode is maximize
-      optimize_mode: maximize
-      # (optional) In order to save our computing resource, we start to predict when we have more than only after receiving start_step number of reported intermediate results.
-      * The default value of start_step is 6.
-      start_step: 6
-      # (optional) The threshold that we decide to early stop the worse performance curve.
-      # For example: if threshold = 0.95, optimize_mode = maximize, best performance in the history is 0.9, then we will stop the trial which predict value is lower than 0.95 * 0.9 = 0.855.
-      * The default value of threshold is 0.95.
-      # Kindly reminds that if you choose minimize mode, please adjust the value of threshold >= 1.0 (e.g threshold=1.1)
-      threshold: 0.95
-      # (optional) The gap interval between Assesor judgements.
-      # For example: if gap = 2, start_step = 6, then we will assess the result when we get 6, 8, 10, 12...intermedian result.
-      * The default value of gap is 1.
-      gap: 1
+  builtinAssessorName: Curvefitting
+  classArgs:
+    # (required)The total number of epoch.
+    #  We need to know the number of epoch to determine which point we need to predict.
+    epoch_num: 20
+    # (optional) In order to save our computing resource, we start to predict when we have more than only after receiving start_step number of reported intermediate results.
+    # The default value of start_step is 6.
+    start_step: 6
+    # (optional) The threshold that we decide to early stop the worse performance curve.
+    # For example: if threshold = 0.95, best performance in the history is 0.9, then we will stop the trial which predict value is lower than 0.95 * 0.9 = 0.855.
+    # The default value of threshold is 0.95.
+    threshold: 0.95
+    # (optional) The gap interval between Assesor judgements.
+    # For example: if gap = 2, start_step = 6, then we will assess the result when we get 6, 8, 10, 12...intermedian result.
+    # The default value of gap is 1.
+    gap: 1
 ```
 
-## 3. File Structure
+## Limitation
+
+According to the original paper, only incremental functions are supported. Therefore this assessor can only be used to maximize optimization metrics. For example, it can be used for accuracy, but not for loss.
+
+## File Structure
+
 The assessor has a lot of different files, functions, and classes. Here we briefly describe a few of them.
 
 * `curvefunctions.py` includes all the function expressions and default parameters.
 * `modelfactory.py` includes learning and predicting; the corresponding calculation part is also implemented here.
 * `curvefitting_assessor.py` is the assessor which receives the trial history and assess whether to early stop the trial.
 
-## 4. TODO
-* Further improve the accuracy of the prediction and test it on more models.
+## TODO
 
+* Further improve the accuracy of the prediction and test it on more models.
 
 [1]: http://aad.informatik.uni-freiburg.de/papers/15-IJCAI-Extrapolation_of_Learning_Curves.pdf

docs/en_US/Tuner/PBTTuner.md

@@ -46,7 +46,3 @@ tuner:
     all_checkpoint_dir: /the/path/to/store/checkpoints
     population_size: 10
 ```
-
-### Limitation
-
-Importing data has not been supported yet.

docs/en_US/Tutorial/InstallationLinux.md

@@ -19,7 +19,7 @@ Installation on Linux and macOS follow the same instructions, given below.
   Prerequisites: `python 64-bit >=3.5`, `git`, `wget`
 
   ```bash
-  git clone -b v1.5 https://github.com/Microsoft/nni.git
+  git clone -b v1.6 https://github.com/Microsoft/nni.git
   cd nni
   ./install.sh
   ```
@@ -35,7 +35,7 @@ The following example is built on TensorFlow 1.x. Make sure **TensorFlow 1.x is
 * Download the examples via cloning the source code.
 
   ```bash
-  git clone -b v1.5 https://github.com/Microsoft/nni.git
+  git clone -b v1.6 https://github.com/Microsoft/nni.git
   ```
 
 * Run the MNIST example.

docs/en_US/Tutorial/InstallationWin.md

@@ -29,7 +29,7 @@ If you want to contribute to NNI, refer to [setup development environment](Setup
 * From source code
 
     ```bat
-    git clone -b v1.5 https://github.com/Microsoft/nni.git
+    git clone -b v1.6 https://github.com/Microsoft/nni.git
     cd nni
     powershell -ExecutionPolicy Bypass -file install.ps1
     ```
@@ -41,7 +41,7 @@ The following example is built on TensorFlow 1.x. Make sure **TensorFlow 1.x is
 * Clone examples within source code.
 
     ```bat
-    git clone -b v1.5 https://github.com/Microsoft/nni.git
+    git clone -b v1.6 https://github.com/Microsoft/nni.git
     ```
 
 * Run the MNIST example.

docs/en_US/conf.py

@@ -28,7 +28,7 @@
 # The short X.Y version
 version = ''
 # The full version, including alpha/beta/rc tags
-release = 'v1.5'
+release = 'v1.6'
 
 # -- General configuration ---------------------------------------------------
 

examples/trials/mnist-pbt-tuner-pytorch/mnist.py

@@ -13,12 +13,12 @@
 logger = logging.getLogger('mnist_pbt_tuner_pytorch_AutoML')
 
 class Net(nn.Module):
-    def __init__(self, hidden_size):
+    def __init__(self):
         super(Net, self).__init__()
         self.conv1 = nn.Conv2d(1, 20, 5, 1)
         self.conv2 = nn.Conv2d(20, 50, 5, 1)
-        self.fc1 = nn.Linear(4*4*50, hidden_size)
-        self.fc2 = nn.Linear(hidden_size, 10)
+        self.fc1 = nn.Linear(4*4*50, 512)
+        self.fc2 = nn.Linear(512, 10)
 
     def forward(self, x):
         x = F.relu(self.conv1(x))
@@ -104,9 +104,7 @@ def main(args):
         ])),
         batch_size=1000, shuffle=True, **kwargs)
 
-    hidden_size = args['hidden_size']
-
-    model = Net(hidden_size=hidden_size).to(device)
+    model = Net().to(device)
 
     save_checkpoint_dir = args['save_checkpoint_dir']
     save_checkpoint_path = os.path.join(save_checkpoint_dir, 'model.pth')
@@ -146,11 +144,9 @@ def get_params():
     # Training settings
     parser = argparse.ArgumentParser(description='PyTorch MNIST Example')
     parser.add_argument("--data_dir", type=str,
-                        default='./tmp/pytorch/mnist/input_data', help="data directory")
+                        default='/tmp/pytorch/mnist/input_data', help="data directory")
     parser.add_argument('--batch_size', type=int, default=64, metavar='N',
                         help='input batch size for training (default: 64)')
-    parser.add_argument("--hidden_size", type=int, default=512, metavar='N',
-                        help='hidden layer size (default: 512)')
     parser.add_argument('--lr', type=float, default=0.01, metavar='LR',
                         help='learning rate (default: 0.01)')
     parser.add_argument('--momentum', type=float, default=0.5, metavar='M',

examples/trials/mnist-pbt-tuner-pytorch/search_space.json

@@ -1,6 +1,5 @@
 {
     "batch_size": {"_type":"choice", "_value": [16, 32, 64, 128]},
-    "hidden_size":{"_type":"choice","_value":[128, 256, 512, 1024]},
     "lr":{"_type":"choice","_value":[0.0001, 0.001, 0.01, 0.1]},
     "momentum":{"_type":"uniform","_value":[0, 1]}
 }

examples/trials/mnist-pytorch/config_assessor.yml

@@ -19,8 +19,6 @@ assessor:
   #choice: Medianstop, Curvefitting
   builtinAssessorName: Curvefitting
   classArgs:
-    #choice: maximize, minimize
-    optimize_mode: maximize
     epoch_num: 20
     threshold: 0.9
 trial:

examples/trials/mnist-tfv1/config_assessor.yml

@@ -19,8 +19,6 @@ assessor:
   #choice: Medianstop, Curvefitting
   builtinAssessorName: Curvefitting
   classArgs:
-    #choice: maximize, minimize
-    optimize_mode: maximize
     epoch_num: 20
     threshold: 0.9
 trial:

examples/trials/mnist-tfv2/config_assessor.yml

@@ -19,8 +19,6 @@ assessor:
   #choice: Medianstop, Curvefitting
   builtinAssessorName: Curvefitting
   classArgs:
-    #choice: maximize, minimize
-    optimize_mode: maximize
     epoch_num: 20
     threshold: 0.9
 trial:

src/sdk/pynni/nni/curvefitting_assessor/curvefitting_assessor.py

@@ -9,6 +9,7 @@
 
 logger = logging.getLogger('curvefitting_Assessor')
 
+
 class CurvefittingAssessor(Assessor):
     """CurvefittingAssessor uses learning curve fitting algorithm to predict the learning curve performance in the future.
     It stops a pending trial X at step S if the trial's forecast result at target step is convergence and lower than the
@@ -18,26 +19,17 @@ class CurvefittingAssessor(Assessor):
     ----------
     epoch_num : int
         The total number of epoch
-    optimize_mode : str
-        optimize mode, 'maximize' or 'minimize'
     start_step : int
         only after receiving start_step number of reported intermediate results
     threshold : float
         The threshold that we decide to early stop the worse performance curve.
     """
-    def __init__(self, epoch_num=20, optimize_mode='maximize', start_step=6, threshold=0.95, gap=1):
+
+    def __init__(self, epoch_num=20, start_step=6, threshold=0.95, gap=1):
         if start_step <= 0:
             logger.warning('It\'s recommended to set start_step to a positive number')
         # Record the target position we predict
         self.target_pos = epoch_num
-        # Record the optimize_mode
-        if optimize_mode == 'maximize':
-            self.higher_better = True
-        elif optimize_mode == 'minimize':
-            self.higher_better = False
-        else:
-            self.higher_better = True
-            logger.warning('unrecognized optimize_mode %s', optimize_mode)
         # Start forecasting when historical data reaches start step
         self.start_step = start_step
         # Record the compared threshold
@@ -109,10 +101,12 @@ def assess_trial(self, trial_job_id, trial_history):
             # Predict the final result
             curvemodel = CurveModel(self.target_pos)
             predict_y = curvemodel.predict(scalar_trial_history)
-            logger.info('Prediction done. Trial job id = %s. Predict value = %s', trial_job_id, predict_y)
+            log_message = "Prediction done. Trial job id = {}, Predict value = {}".format(trial_job_id, predict_y)
             if predict_y is None:
-                logger.info('wait for more information to predict precisely')
+                logger.info('%s, wait for more information to predict precisely', log_message)
                 return AssessResult.Good
+            else:
+                logger.info(log_message)
             standard_performance = self.completed_best_performance * self.threshold
 
             end_time = datetime.datetime.now()
@@ -122,14 +116,9 @@ def assess_trial(self, trial_job_id, trial_history):
                     trial_job_id, self.trial_history
                 )
 
-            if self.higher_better:
-                if predict_y > standard_performance:
-                    return AssessResult.Good
-                return AssessResult.Bad
-            else:
-                if predict_y < standard_performance:
-                    return AssessResult.Good
-                return AssessResult.Bad
+            if predict_y > standard_performance:
+                return AssessResult.Good
+            return AssessResult.Bad
 
         except Exception as exception:
             logger.exception('unrecognize exception in curvefitting_assessor %s', exception)