Merge pull request #248 from microsoft/master

merge master

docs/en_US/NAS/NasGuide.md

@@ -176,7 +176,7 @@ For example,
 }
 ```
 
-After applying, the model is then fixed and ready for final training. The model works as a single model, although it might contain more parameters than expected. This comes with pros and cons. The good side is, you can directly load the checkpoint dumped from supernet during the search phase and start retraining from there. However, this is also a model with redundant parameters and this may cause problems when trying to count the number of parameters in the model. For deeper reasons and possible workarounds, see [Trainers](./NasReference.md).
+After applying, the model is then fixed and ready for final training. The model works as a single model, and unused parameters and modules are pruned.
 
 Also, refer to [DARTS](./DARTS.md) for code exemplifying retraining.
 

docs/en_US/TrainingService/PaiMode.md

@@ -7,9 +7,9 @@ Step 1. Install NNI, follow the install guide [here](../Tutorial/QuickStart.md).
 
 Step 2. Get PAI token.   
 Click `My profile` button in the top-right side of PAI's webprotal.
-![](../../img/pai_token_button.jpg)
-Find the token management region, copy one of the token as your account token.
-![](../../img/pai_token_profile.jpg)  
+![](../../img/pai_profile.jpg)  
+Click `copy` button in the page to copy a jwt token.
+![](../../img/pai_token.jpg)  
 
 Step 3. Mount NFS storage to local machine.  
   Click `Submit job` button in PAI's webportal.
@@ -19,7 +19,7 @@ Step 3. Mount NFS storage to local machine.
 The `DEFAULT_STORAGE`field is the path to be mounted in PAI's container when a job is started. The `Preview container paths` is the NFS host and path that PAI provided, you need to mount the corresponding host and path to your local machine first, then NNI could use the PAI's NFS storage.  
 For example, use the following command:
 ```
-sudo mount nfs://gcr-openpai-infra02:/pai/data /local/mnt
+sudo mount -t nfs4 gcr-openpai-infra02:/pai/data /local/mnt
 ```
 Then the `/data` folder in container will be mounted to `/local/mnt` folder in your local machine.  
 You could use the following configuration in your NNI's config file:
@@ -66,15 +66,15 @@ trial:
   virtualCluster: default
   nniManagerNFSMountPath: /home/user/mnt
   containerNFSMountPath: /mnt/data/user
-  paiStoragePlugin: team_wise
+  paiStoragePlugin: teamwise_storage
 # Configuration to access OpenPAI Cluster
 paiConfig:
   userName: your_pai_nni_user
   token: your_pai_token
   host: 10.1.1.1
 ```
 
-Note: You should set `trainingServicePlatform: pai` in NNI config YAML file if you want to start experiment in pai mode.
+Note: You should set `trainingServicePlatform: pai` in NNI config YAML file if you want to start experiment in pai mode. The host field in configuration file is PAI's job submission page uri, like `10.10.5.1`, the default http protocol in NNI is `http`, if your PAI's cluster enabled https, please use the uri in `https://10.10.5.1` format.
 
 Compared with [LocalMode](LocalMode.md) and [RemoteMachineMode](RemoteMachineMode.md), trial configuration in pai mode have these additional keys:
 * cpuNum

docs/en_US/Tutorial/SetupNniDeveloperEnvironment.md

@@ -1,76 +1,59 @@
-**Set up NNI developer environment**
+# Setup NNI development environment
 
-===
+NNI development environment supports Ubuntu 1604 (or above), and Windows 10 with Python3 64bit.
 
-## Best practice for debug NNI source code
+## Installation
 
-For debugging NNI source code, your development environment should be under Ubuntu 16.04 (or above) system with python 3 and pip 3 installed, then follow the below steps.
+The installation steps are similar with installing from source code. But the installation links to code directory, so that code changes can be applied to installation as easy as possible.
 
-### 1. Clone the source code
+### 1. Clone source code
 
-Run the command
-
-```
+```bash
 git clone https://github.com/Microsoft/nni.git
 ```
 
-to clone the source code
+Note, if you want to contribute code back, it needs to fork your own NNI repo, and clone from there.
 
-### 2. Prepare the debug environment and install dependencies
+### 2. Install from source code
 
-Change directory to the source code folder, then run the command
+#### Ubuntu
 
+```bash
+make dev-easy-install
 ```
-make install-dependencies
-```
-
-to install the dependent tools for the environment
-
-### 3. Build source code
 
-Run the command
+#### Windows
 
+```bat
+powershell -ExecutionPolicy Bypass -file install.ps1 -Development
 ```
-make build
-```
-
-to build the source code
 
-### 4. Install NNI to development environment
-
-Run the command
-
-```
-make dev-install
-```
-
-to install the distribution content to development environment, and create cli scripts
-
-### 5. Check if the environment is ready
+### 3. Check if the environment is ready
 
 Now, you can try to start an experiment to check if your environment is ready.
 For example, run the command
 
-```
-nnictl create --config ~/nni/examples/trials/mnist-tfv1/config.yml
+```bash
+nnictl create --config examples/trials/mnist-tfv1/config.yml
 ```
 
 And open WebUI to check if everything is OK
 
-### 6. Redeploy
-
-After the code changes, it may need to redeploy. It depends on what kind of code changed. 
+### 4. Reload changes
 
 #### Python
 
-It doesn't need to redeploy, but the nnictl may need to be restarted.
+Nothing to do, the code is already linked to package folders.
 
 #### TypeScript
 
-* If `src/nni_manager` is changed, run `yarn watch` continually under this folder. It will rebuild code instantly. The nnictl may need to be restarted to reload NNI manager.
+* If `src/nni_manager` is changed, run `yarn watch` under this folder. It will watch and build code continually. The `nnictl` need to be restarted to reload NNI manager.
 * If `src/webui` or `src/nasui` are changed, run `yarn start` under the corresponding folder. The web UI will refresh automatically if code is changed.
 
+### 5. Submit Pull Request
+
+All changes are merged to master branch from your forked repo. The description of Pull Request must be meaningful, and useful.
+
+We will review the changes as soon as possible. Once it passes review, we will merge it to master branch.
 
----
-At last, wish you have a wonderful day.
-For more contribution guidelines on making PR's or issues to NNI source code, you can refer to our [Contributing](Contributing.md) document.
+For more contribution guidelines and coding styles, you can refer to the [contributing document](Contributing.md).

install.ps1

@@ -1,12 +1,14 @@
+param ([Switch] $Development)
 [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
 
+
 $install_node = $true
 $install_yarn = $true
 
-if([Environment]::Is64BitOperatingSystem){
+if ([Environment]::Is64BitOperatingSystem) {
     $OS_VERSION = 'win64'
 }
-else{
+else {
     $OS_VERSION = 'win32'
 }
 # nodejs
@@ -15,86 +17,89 @@ $yarnUrl = "https://yarnpkg.com/latest.tar.gz"
 $unzipNodeDir = "node-v*"
 $unzipYarnDir = "yarn-v*"
 
-$NNI_DEPENDENCY_FOLDER = [System.IO.Path]::GetTempPath()+$env:USERNAME
+$NNI_DEPENDENCY_FOLDER = [System.IO.Path]::GetTempPath() + $env:USERNAME
 
 $WHICH_PYTHON = where.exe python
-if($WHICH_PYTHON -eq $null){
+if ($WHICH_PYTHON -eq $null) {
     throw "Can not find python"
 }
-else{
+else {
     $pyVersion = & python -V 2>&1
-    $pyVersion =  ([string]$pyVersion).substring(7,3)
-    if([double]$pyVersion -lt 3.5){
+    $pyVersion = ([string]$pyVersion).substring(7, 3)
+    if ([double]$pyVersion -lt 3.5) {
         throw "python version should >= 3.5"
     }
 }
 
 $WHICH_PIP = where.exe pip
-if($WHICH_PIP -eq $null){
+if ($WHICH_PIP -eq $null) {
     throw "Can not find pip"
 }
 
 $env:PYTHONIOENCODING = "UTF-8"
-if($env:VIRTUAL_ENV){
+if ($env:VIRTUAL_ENV) {
     $NNI_PYTHON3 = $env:VIRTUAL_ENV + "\Scripts"
     $NNI_PKG_FOLDER = $env:VIRTUAL_ENV + "\nni"
     $NNI_PYTHON_SCRIPTS = $NNI_PYTHON3
 }
-else{
+else {
     $NNI_PYTHON3 = $(python -c 'import site; from pathlib import Path; print(Path(site.getsitepackages()[0]))')
     $NNI_PKG_FOLDER = $NNI_PYTHON3 + "\nni"
-    $NNI_PYTHON_SCRIPTS =  $NNI_PYTHON3 + "\Scripts"
+    $NNI_PYTHON_SCRIPTS = $NNI_PYTHON3 + "\Scripts"
 }
 
-$PIP_INSTALL = """$NNI_PYTHON3\python"" -m pip install ."
+$PIP_INSTALL = """$NNI_PYTHON3\python"" -m pip install "
 
-if(!(Test-Path $NNI_DEPENDENCY_FOLDER)){
+if (!(Test-Path $NNI_DEPENDENCY_FOLDER)) {
     New-Item $NNI_DEPENDENCY_FOLDER -ItemType Directory
 }
-$NNI_NODE_ZIP = $NNI_DEPENDENCY_FOLDER+"\nni-node.zip"
-$NNI_NODE_FOLDER = $NNI_DEPENDENCY_FOLDER+"\nni-node"
-$NNI_YARN_TARBALL = $NNI_DEPENDENCY_FOLDER+"\nni-yarn.tar.gz"
-$NNI_YARN_FOLDER = $NNI_DEPENDENCY_FOLDER+"\nni-yarn"
-$NNI_YARN = $NNI_YARN_FOLDER +"\bin\yarn"
+$NNI_NODE_ZIP = $NNI_DEPENDENCY_FOLDER + "\nni-node.zip"
+$NNI_NODE_FOLDER = $NNI_DEPENDENCY_FOLDER + "\nni-node"
+$NNI_YARN_TARBALL = $NNI_DEPENDENCY_FOLDER + "\nni-yarn.tar.gz"
+$NNI_YARN_FOLDER = $NNI_DEPENDENCY_FOLDER + "\nni-yarn"
+$NNI_YARN = $NNI_YARN_FOLDER + "\bin\yarn"
 
 ## Version number
 $NNI_VERSION_VALUE = $(git describe --tags)
 $NNI_VERSION_TEMPLATE = "999.0.0-developing"
 
-if(!(Test-Path $NNI_NODE_ZIP)){
+if (!(Test-Path $NNI_NODE_ZIP)) {
     Write-Host "Downloading Node..."
     (New-Object Net.WebClient).DownloadFile($nodeUrl, $NNI_NODE_ZIP)
 }
 
-if(!(Test-Path $NNI_YARN_TARBALL)){
+if (!(Test-Path $NNI_YARN_TARBALL)) {
     Write-Host "Downloading Yarn..."
     (New-Object Net.WebClient).DownloadFile($yarnUrl, $NNI_YARN_TARBALL)
 }
 
 $NNI_YARN_TARBALL = $NNI_YARN_TARBALL -split '\\' -join '\\'
 $NNI_DEPENDENCY_FOLDER = $NNI_DEPENDENCY_FOLDER -split '\\' -join '\\'
 $SCRIPT_PATH = $NNI_DEPENDENCY_FOLDER + '\extract.py'
-$SCRIPT =  "import tarfile",
-       ("tar = tarfile.open(""{0}"")" -f $NNI_YARN_TARBALL),
-       ("tar.extractall(""{0}"")" -f $NNI_DEPENDENCY_FOLDER),
+$SCRIPT = "import tarfile",
+        ("tar = tarfile.open(""{0}"")" -f $NNI_YARN_TARBALL),
+        ("tar.extractall(""{0}"")" -f $NNI_DEPENDENCY_FOLDER),
         "tar.close()"
 [System.IO.File]::WriteAllLines($SCRIPT_PATH, $SCRIPT)
 
 Add-Type -AssemblyName System.IO.Compression.FileSystem
-function Unzip{
+function Unzip {
     param([string]$zipfile, [string]$outpath)
     [System.IO.Compression.ZipFile]::ExtractToDirectory($zipfile, $outpath)
 }
 if ($install_node) {
     ### nodejs install
-    if(!(Test-Path $NNI_NODE_FOLDER)){
+    if (!(Test-Path $NNI_NODE_FOLDER)) {
         Unzip $NNI_NODE_ZIP $NNI_DEPENDENCY_FOLDER
         $unzipNodeDir = Get-ChildItem "$NNI_DEPENDENCY_FOLDER\$unzipNodeDir"
         Rename-Item $unzipNodeDir "nni-node"
     }
     Copy-Item "$NNI_NODE_FOLDER\node.exe" $NNI_PYTHON_SCRIPTS -Recurse -Force
+}
+
+if ($install_yarn) {
     ### yarn install
-    if(!(Test-Path $NNI_YARN_FOLDER)){
+    if (!(Test-Path $NNI_YARN_FOLDER)) {
         cmd /C """$NNI_PYTHON3\python""" $SCRIPT_PATH
         $unzipYarnDir = Get-ChildItem "$NNI_DEPENDENCY_FOLDER\$unzipYarnDir"
         Rename-Item $unzipYarnDir "nni-yarn"
@@ -104,10 +109,40 @@ if ($install_node) {
 ## install-python-modules:
 ### Installing Python SDK
 (Get-Content setup.py).replace($NNI_VERSION_TEMPLATE, $NNI_VERSION_VALUE) | Set-Content setup.py
-cmd /c $PIP_INSTALL
+
+if ($Development) {
+    $PYTHON_BUILD = "build"
+    if (Test-Path $PYTHON_BUILD) { 
+        # To compat with file and links.
+        cmd /c rmdir /s /q $PYTHON_BUILD
+    }
+    New-Item $PYTHON_BUILD -ItemType Directory
+    New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nni" -Target "src\sdk\pynni\nni"
+    New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nnicli" -Target "src\sdk\pycli\nnicli"
+    New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nni_annotation" -Target "tools\nni_annotation"
+    New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nni_cmd" -Target "tools\nni_cmd"
+    New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nni_trial_tool" -Target "tools\nni_trial_tool"
+    New-Item -ItemType Junction -Path "$($PYTHON_BUILD)\nni_gpu_tool" -Target "tools\nni_gpu_tool"
+
+    Copy-Item setup.py $PYTHON_BUILD
+    Copy-Item README.md $PYTHON_BUILD
+
+    Push-Location build
+    #update folders in setup file
+    (Get-Content setup.py).replace("src/sdk/pynni/", "") | Set-Content setup.py
+    (Get-Content setup.py).replace("src/sdk/pycli/", "") | Set-Content setup.py
+    (Get-Content setup.py).replace("src/sdk/pynni", ".") | Set-Content setup.py
+    (Get-Content setup.py).replace("tools/", "") | Set-Content setup.py
+    # install current folder.
+    cmd /c $PIP_INSTALL -e .
+    Pop-Location
+}
+else {
+    cmd /c $PIP_INSTALL .
+}
 
 # Building NNI Manager
-$env:PATH=$NNI_PYTHON_SCRIPTS+';'+$env:PATH
+$env:PATH = $NNI_PYTHON_SCRIPTS + ';' + $env:PATH
 cd src\nni_manager
 cmd /c $NNI_YARN
 cmd /c $NNI_YARN build
@@ -124,17 +159,31 @@ cmd /c $NNI_YARN build
 cd ..\..
 
 ## install-node-modules
-if(!(Test-Path $NNI_PKG_FOLDER)){
-    New-Item $NNI_PKG_FOLDER -ItemType Directory
+if (Test-Path $NNI_PKG_FOLDER) {
+    # it needs to remove the whole folder for following copy.
+    cmd /c rmdir /s /q $NNI_PKG_FOLDER
+}
+
+$NNI_PKG_FOLDER_STATIC = $NNI_PKG_FOLDER + "\static"
+$NASUI_PKG_FOLDER = $NNI_PKG_FOLDER + "\nasui"
+
+if ($Development) {
+    New-Item -ItemType Junction -Path $($NNI_PKG_FOLDER) -Target "src\nni_manager\dist"
+    New-Item -ItemType Junction -Path "$($NNI_PKG_FOLDER)\node_modules" -Target "src\nni_manager\node_modules"
+    New-Item -ItemType Junction -Path $($NNI_PKG_FOLDER_STATIC) -Target "src\webui\build"
+    New-Item -ItemType Junction -Path $($NASUI_PKG_FOLDER) -Target "src\nasui\build"
 }
-Remove-Item $NNI_PKG_FOLDER -Recurse -Force
-Copy-Item "src\nni_manager\dist" $NNI_PKG_FOLDER -Recurse
+else {
+    Copy-Item "src\nni_manager\dist" $NNI_PKG_FOLDER -Recurse
+    Copy-Item "src\webui\build" $NNI_PKG_FOLDER_STATIC -Recurse
+    Copy-Item "src\nasui\build" $NASUI_PKG_FOLDER -Recurse
+}
+
 Copy-Item "src\nni_manager\package.json" $NNI_PKG_FOLDER
 $PKG_JSON = $NNI_PKG_FOLDER + "\package.json"
 (Get-Content $PKG_JSON).replace($NNI_VERSION_TEMPLATE, $NNI_VERSION_VALUE) | Set-Content $PKG_JSON
-cmd /c $NNI_YARN --prod --cwd $NNI_PKG_FOLDER
-$NNI_PKG_FOLDER_STATIC = $NNI_PKG_FOLDER + "\static"
-$NASUI_PKG_FOLDER = $NNI_PKG_FOLDER + "\nasui"
-Copy-Item "src\webui\build" $NNI_PKG_FOLDER_STATIC -Recurse
-Copy-Item "src\nasui\build" $NASUI_PKG_FOLDER -Recurse
 Copy-Item "src\nasui\server.js" $NASUI_PKG_FOLDER -Recurse
+
+if (!$Development) {
+    cmd /c $NNI_YARN --prod --cwd $NNI_PKG_FOLDER
+}

src/nni_manager/core/ipcInterface.ts

@@ -23,11 +23,7 @@ const ipcIncomingFd: number = 4;
  */
 function encodeCommand(commandType: string, content: string): Buffer {
     const contentBuffer: Buffer = Buffer.from(content);
-    if (contentBuffer.length >= 1_000_000) {
-        throw new RangeError('Command too long');
-    }
-    const contentLengthBuffer: Buffer = Buffer.from(contentBuffer.length.toString().padStart(6, '0'));
-
+    const contentLengthBuffer: Buffer = Buffer.from(contentBuffer.length.toString().padStart(14, '0'));
     return Buffer.concat([Buffer.from(commandType), contentLengthBuffer, contentBuffer]);
 }
 
@@ -43,12 +39,12 @@ function decodeCommand(data: Buffer): [boolean, string, string, Buffer] {
         return [false, '', '', data];
     }
     const commandType: string = data.slice(0, 2).toString();
-    const contentLength: number = parseInt(data.slice(2, 8).toString(), 10);
-    if (data.length < contentLength + 8) {
+    const contentLength: number = parseInt(data.slice(2, 16).toString(), 10);
+    if (data.length < contentLength + 16) {
         return [false, '', '', data];
     }
-    const content: string = data.slice(8, contentLength + 8).toString();
-    const remain: Buffer = data.slice(contentLength + 8);
+    const content: string = data.slice(16, contentLength + 16).toString();
+    const remain: Buffer = data.slice(contentLength + 16);
 
     return [true, commandType, content, remain];
 }