Skip to content
This repository has been archived by the owner on Sep 18, 2024. It is now read-only.

Fix aml doc #2631

Merged
merged 130 commits into from
Jul 8, 2020
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
130 commits
Select commit Hold shift + click to select a range
b809394
update doc for PAI
SparkSnail May 18, 2020
9fb9eb0
fix typo
SparkSnail May 18, 2020
352b516
fix comments
SparkSnail May 19, 2020
dcd2ffd
Merge pull request #251 from microsoft/master
SparkSnail May 29, 2020
a738331
init changes
squirrelsc Jun 2, 2020
3177aeb
Merge remote-tracking branch 'official/master' into 2391-reuse-job
squirrelsc Jun 2, 2020
2aafac1
refactors
squirrelsc Jun 3, 2020
0435b7f
refactoring
squirrelsc Jun 3, 2020
2e5ef51
minor fix, and take some review comments.
squirrelsc Jun 3, 2020
6d7bc62
move reuse to upper level
squirrelsc Jun 3, 2020
c67b162
support multi nodes
Jun 4, 2020
e13a620
fix eslint errors
Jun 4, 2020
59d4a71
support multi environments better
Jun 5, 2020
eae0540
Merge remote-tracking branch 'official/master' into 2391-reuse-job
Jun 5, 2020
81c49cf
code refactor
Jun 5, 2020
92cab3a
fix openpai yaml format
Jun 5, 2020
0674d88
fix k8s yaml schema
Jun 5, 2020
e5b9665
rename forward training service
Jun 5, 2020
67ef648
Merge remote-tracking branch 'official/master' into 2391-reuse-job
Jun 5, 2020
3b8b6fb
Merge pull request #252 from microsoft/master
SparkSnail Jun 7, 2020
1e626fd
add trialService
Jun 9, 2020
c6b6061
not send stop for single node
Jun 9, 2020
916e444
Merge pull request #253 from microsoft/master
SparkSnail Jun 15, 2020
b8e47be
rename environmentManager to trialDispatcher
Jun 16, 2020
0ee933a
support no central storage service
Jun 16, 2020
c7973be
init
SparkSnail Jun 16, 2020
c094057
improve delopment support
Jun 16, 2020
c2735d3
Merge remote-tracking branch 'official/master' into 2391-reuse-job
Jun 16, 2020
d0b2504
use latest storage component
Jun 16, 2020
c8d4696
add gpu info
Jun 16, 2020
1a9f19f
work version
SparkSnail Jun 16, 2020
3f4c177
separate channel and add gpu collector in runner
Jun 16, 2020
648e0bb
Merge branch '2391-reuse-job' of https://github.com/squirrelsc/nni in…
SparkSnail Jun 16, 2020
2fa4a77
init
SparkSnail Jun 16, 2020
caeffb8
Merge pull request #254 from microsoft/master
SparkSnail Jun 17, 2020
d0768b0
add more GPU information, and improve debugging.
squirrelsc Jun 17, 2020
8dff16f
fix GPU info collector
Jun 17, 2020
bea8ed6
Merge branch '2391-reuse-job' of https://github.com/squirrelsc/nni in…
SparkSnail Jun 18, 2020
e297aa5
update
SparkSnail Jun 18, 2020
500c1cb
channel support single file
Jun 18, 2020
d880512
refine code, and implement command channel
Jun 19, 2020
5c33d11
update
SparkSnail Jun 20, 2020
45424e8
support concurrent trials in runner.
Jun 22, 2020
9ca3444
implement web channel
Jun 24, 2020
5018039
fix eslint errors, and rename rest to web
Jun 24, 2020
283bceb
remove trial service, as it's replaced by channel.
Jun 24, 2020
0c67c5c
Merge remote-tracking branch 'official/master' into 2391-reuse-job
Jun 24, 2020
671f5d8
fix merged problem, and small refine for ut.
Jun 24, 2020
a65a810
fix pylint errors
Jun 24, 2020
6d36ae5
fix lint error
Jun 26, 2020
5e352f7
init
SparkSnail Jun 26, 2020
b9d1aa5
fix conflict
SparkSnail Jun 28, 2020
a3a91d8
format
SparkSnail Jun 28, 2020
57c300e
Merge pull request #255 from microsoft/master
SparkSnail Jun 28, 2020
69a5170
remove useless deferred.
Jun 29, 2020
edc4608
fix package
SparkSnail Jun 29, 2020
c1f0239
fix incorrect check logic
Jun 29, 2020
af97bb1
make license header consistent
Jun 29, 2020
10feb6a
Merge remote-tracking branch 'official/master' into 2391-reuse-job
Jun 29, 2020
c00cd31
add missed await.
Jun 29, 2020
78f1386
add doc and example
SparkSnail Jun 29, 2020
586d6ac
support log level in UT
Jun 29, 2020
2db8ff8
refine interface to support aml better.
Jun 29, 2020
f631e4c
fix runtime error on exit
Jun 29, 2020
5982fb3
Merge remote-tracking branch 'official/master' into 2391-reuse-job
Jun 29, 2020
f687a6e
fix eslint error
Jun 29, 2020
476ffec
send metric data from channel
Jun 29, 2020
0f2367c
support version check
Jun 29, 2020
9d7bd3c
fix pylint errors
Jun 29, 2020
130ed27
fix non-local failed ITs
Jun 29, 2020
ab86080
fix comments
SparkSnail Jun 30, 2020
4b11a53
fix conflict
SparkSnail Jun 30, 2020
15ee064
fix conflict
SparkSnail Jun 30, 2020
7c48610
format
SparkSnail Jun 30, 2020
c0c7d96
format code
SparkSnail Jun 30, 2020
93eefb2
format code
SparkSnail Jun 30, 2020
53cea0f
remove unused code
SparkSnail Jun 30, 2020
34d9351
format code
SparkSnail Jun 30, 2020
25a9dab
fix comments
SparkSnail Jun 30, 2020
cada76a
fix comments
SparkSnail Jun 30, 2020
de7dc7c
fix comments
SparkSnail Jun 30, 2020
428dc3d
add blank line
SparkSnail Jun 30, 2020
2e9c70e
fix comments
SparkSnail Jun 30, 2020
8cf8583
fix comments
SparkSnail Jun 30, 2020
fd5fd9e
fix build
SparkSnail Jun 30, 2020
54a22af
fix comments
SparkSnail Jun 30, 2020
525b961
fix channel async calls
Jun 30, 2020
8ec5e7d
fix comments
SparkSnail Jun 30, 2020
bdd3840
fix comments
SparkSnail Jun 30, 2020
b341dce
fix comments
SparkSnail Jun 30, 2020
ce81c51
Merge remote-tracking branch 'snail/dev-aml' into 2391-improve
Jun 30, 2020
e66dc23
fix comments
SparkSnail Jun 30, 2020
9cf6744
merge code logic
Jun 30, 2020
bd77f5c
Merge remote-tracking branch 'snail/dev-aml' into 2391-improve
Jun 30, 2020
ddfb0cc
Merge branch 'master' of https://github.com/microsoft/nni into dev-aml
SparkSnail Jun 30, 2020
65660e6
Merge pull request #257 from microsoft/master
SparkSnail Jun 30, 2020
fec8a67
Merge branch 'master' of https://github.com/SparkSnail/nni into dev-aml
SparkSnail Jun 30, 2020
5200a3a
Merge remote-tracking branch 'snail/dev-aml' into 2391-improve
Jun 30, 2020
51befa5
fix eslint errors
Jun 30, 2020
478629f
add run fo messages
Jun 30, 2020
c299ce1
Merge pull request #256 from squirrelsc/2391-improve
SparkSnail Jun 30, 2020
0517e13
fix comments
SparkSnail Jun 30, 2020
fc4b978
sort class
SparkSnail Jun 30, 2020
e527743
fix eslint
SparkSnail Jun 30, 2020
b047681
fix eslint
SparkSnail Jun 30, 2020
4acc7e8
fix annotation
SparkSnail Jun 30, 2020
ec1475a
fix import aml
SparkSnail Jun 30, 2020
8eaeebf
fix comments
SparkSnail Jun 30, 2020
56b6818
fix doc build
SparkSnail Jul 1, 2020
e09ff79
fix trial_runner import
SparkSnail Jul 1, 2020
ecf615d
fix doc
SparkSnail Jul 1, 2020
a7a3baf
fix pylint
SparkSnail Jul 1, 2020
9376d6a
Merge pull request #258 from microsoft/master
SparkSnail Jul 1, 2020
870f1e6
Merge branch 'master' of https://github.com/SparkSnail/nni into dev-aml
SparkSnail Jul 1, 2020
7eaa105
add doc for aml
SparkSnail Jul 1, 2020
a0ea554
add content
SparkSnail Jul 1, 2020
972822c
supplement dlts doc
SparkSnail Jul 1, 2020
b5f7f06
add doc content
SparkSnail Jul 1, 2020
f6d9c3f
fix content
SparkSnail Jul 1, 2020
f39036f
fix content
SparkSnail Jul 1, 2020
5e366cf
fix broken link
SparkSnail Jul 1, 2020
a767b79
Fix win32 build (#2623)
chicm-ms Jul 1, 2020
43b12b6
Merge branch 'v1.7' into dev-aml
squirrelsc Jul 3, 2020
5fef3cf
Merge pull request #259 from microsoft/master
SparkSnail Jul 3, 2020
9a864fd
Merge branch 'master' of https://github.com/SparkSnail/nni into dev-aml
SparkSnail Jul 3, 2020
3515185
Merge branch 'dev-aml' of https://github.com/SparkSnail/nni into dev-aml
SparkSnail Jul 3, 2020
b3ec35a
fix aml docker image
SparkSnail Jul 3, 2020
e6fbec2
fix conflict
SparkSnail Jul 3, 2020
c3bc4f5
revert change
SparkSnail Jul 3, 2020
8c5f471
format
SparkSnail Jul 3, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
3 changes: 2 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@

**NNI (Neural Network Intelligence)** is a lightweight but powerful toolkit to help users **automate** <a href="docs/en_US/FeatureEngineering/Overview.md">Feature Engineering</a>, <a href="docs/en_US/NAS/Overview.md">Neural Architecture Search</a>, <a href="docs/en_US/Tuner/BuiltinTuner.md">Hyperparameter Tuning</a> and <a href="docs/en_US/Compressor/Overview.md">Model Compression</a>.

The tool manages automated machine learning (AutoML) experiments, **dispatches and runs** experiments' trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in **different training environments** like <a href="docs/en_US/TrainingService/LocalMode.md">Local Machine</a>, <a href="docs/en_US/TrainingService/RemoteMachineMode.md">Remote Servers</a>, <a href="docs/en_US/TrainingService/PaiMode.md">OpenPAI</a>, <a href="docs/en_US/TrainingService/KubeflowMode.md">Kubeflow</a>, <a href="docs/en_US/TrainingService/FrameworkControllerMode.md">FrameworkController on K8S (AKS etc.)</a>, <a href="docs/en_US/TrainingService/DLTSMode.md">DLWorkspace (aka. DLTS)</a> and other cloud options.
The tool manages automated machine learning (AutoML) experiments, **dispatches and runs** experiments' trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in **different training environments** like <a href="docs/en_US/TrainingService/LocalMode.md">Local Machine</a>, <a href="docs/en_US/TrainingService/RemoteMachineMode.md">Remote Servers</a>, <a href="docs/en_US/TrainingService/PaiMode.md">OpenPAI</a>, <a href="docs/en_US/TrainingService/KubeflowMode.md">Kubeflow</a>, <a href="docs/en_US/TrainingService/FrameworkControllerMode.md">FrameworkController on K8S (AKS etc.)</a>, <a href="docs/en_US/TrainingService/DLTSMode.md">DLWorkspace (aka. DLTS)</a>, <a href="docs/en_US/TrainingService/AMLMode.md">AML (Azure Machine Learning)</a> and other cloud options.

## **Who should consider using NNI**

Expand Down Expand Up @@ -170,6 +170,7 @@ Within the following table, we summarized the current NNI capabilities, we are g
<ul>
<li><a href="docs/en_US/TrainingService/LocalMode.md">Local Machine</a></li>
<li><a href="docs/en_US/TrainingService/RemoteMachineMode.md">Remote Servers</a></li>
<li><a href="docs/en_US/TrainingService/AMLMode.md">AML(Azure Machine Learning)</a></li>
<li><b>Kubernetes based services</b></li>
<ul><li><a href="docs/en_US/TrainingService/PaiMode.md">OpenPAI</a></li>
<li><a href="docs/en_US/TrainingService/KubeflowMode.md">Kubeflow</a></li>
Expand Down
15 changes: 8 additions & 7 deletions docs/en_US/TrainingService/Overview.md
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@

NNI training service is designed to allow users to focus on AutoML itself, agnostic to the underlying computing infrastructure where the trials are actually run. When migrating from one cluster to another (e.g., local machine to Kubeflow), users only need to tweak several configurations, and the experiment can be easily scaled.

Users can use training service provided by NNI, to run trial jobs on [local machine](./LocalMode.md), [remote machines](./RemoteMachineMode.md), and on clusters like [PAI](./PaiMode.md), [Kubeflow](./KubeflowMode.md) and [FrameworkController](./FrameworkControllerMode.md). These are called *built-in training services*.
Users can use training service provided by NNI, to run trial jobs on [local machine](./LocalMode.md), [remote machines](./RemoteMachineMode.md), and on clusters like [PAI](./PaiMode.md), [Kubeflow](./KubeflowMode.md), [FrameworkController](./FrameworkControllerMode.md), [DLTS](./DLTSMode.md) and [AML](./AMLMode.md). These are called *built-in training services*.

If the computing resource customers try to use is not listed above, NNI provides interface that allows users to build their own training service easily. Please refer to "[how to implement training service](./HowToImplementTrainingService)" for details.

Expand All @@ -20,12 +20,13 @@ In case users intend to use large files in their experiment (like large-scaled d

|TrainingService|Brief Introduction|
|---|---|
|[__Local__](./LocalMode.html)|NNI supports running an experiment on local machine, called local mode. Local mode means that NNI will run the trial jobs and nniManager process in same machine, and support gpu schedule function for trial jobs.|
|[__Remote__](./RemoteMachineMode.html)|NNI supports running an experiment on multiple machines through SSH channel, called remote mode. NNI assumes that you have access to those machines, and already setup the environment for running deep learning training code. NNI will submit the trial jobs in remote machine, and schedule suitable machine with enough gpu resource if specified.|
|[__PAI__](./PaiMode.html)|NNI supports running an experiment on [OpenPAI](https://github.com/Microsoft/pai) (aka PAI), called PAI mode. Before starting to use NNI PAI mode, you should have an account to access an [OpenPAI](https://github.com/Microsoft/pai) cluster. See [here](https://github.com/Microsoft/pai#how-to-deploy) if you don't have any OpenPAI account and want to deploy an OpenPAI cluster. In PAI mode, your trial program will run in PAI's container created by Docker.|
|[__Kubeflow__](./KubeflowMode.html)|NNI supports running experiment on [Kubeflow](https://github.com/kubeflow/kubeflow), called kubeflow mode. Before starting to use NNI kubeflow mode, you should have a Kubernetes cluster, either on-premises or [Azure Kubernetes Service(AKS)](https://azure.microsoft.com/en-us/services/kubernetes-service/), a Ubuntu machine on which [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) is setup to connect to your Kubernetes cluster. If you are not familiar with Kubernetes, [here](https://kubernetes.io/docs/tutorials/kubernetes-basics/) is a good start. In kubeflow mode, your trial program will run as Kubeflow job in Kubernetes cluster.|
|[__FrameworkController__](./FrameworkControllerMode.html)|NNI supports running experiment using [FrameworkController](https://github.com/Microsoft/frameworkcontroller), called frameworkcontroller mode. FrameworkController is built to orchestrate all kinds of applications on Kubernetes, you don't need to install Kubeflow for specific deep learning framework like tf-operator or pytorch-operator. Now you can use FrameworkController as the training service to run NNI experiment.|
|[__DLTS__](./DLTSMode.html)|NNI supports running experiment using [DLTS](https://github.com/microsoft/DLWorkspace.git), which is an open source toolkit, developed by Microsoft, that allows AI scientists to spin up an AI cluster in turn-key fashion.|
|[__Local__](./LocalMode.md)|NNI supports running an experiment on local machine, called local mode. Local mode means that NNI will run the trial jobs and nniManager process in same machine, and support gpu schedule function for trial jobs.|
|[__Remote__](./RemoteMachineMode.md)|NNI supports running an experiment on multiple machines through SSH channel, called remote mode. NNI assumes that you have access to those machines, and already setup the environment for running deep learning training code. NNI will submit the trial jobs in remote machine, and schedule suitable machine with enough gpu resource if specified.|
|[__PAI__](./PaiMode.md)|NNI supports running an experiment on [OpenPAI](https://github.com/Microsoft/pai) (aka PAI), called PAI mode. Before starting to use NNI PAI mode, you should have an account to access an [OpenPAI](https://github.com/Microsoft/pai) cluster. See [here](https://github.com/Microsoft/pai#how-to-deploy) if you don't have any OpenPAI account and want to deploy an OpenPAI cluster. In PAI mode, your trial program will run in PAI's container created by Docker.|
|[__Kubeflow__](./KubeflowMode.md)|NNI supports running experiment on [Kubeflow](https://github.com/kubeflow/kubeflow), called kubeflow mode. Before starting to use NNI kubeflow mode, you should have a Kubernetes cluster, either on-premises or [Azure Kubernetes Service(AKS)](https://azure.microsoft.com/en-us/services/kubernetes-service/), a Ubuntu machine on which [kubeconfig](https://kubernetes.io/docs/concepts/configuration/organize-cluster-access-kubeconfig/) is setup to connect to your Kubernetes cluster. If you are not familiar with Kubernetes, [here](https://kubernetes.io/docs/tutorials/kubernetes-basics/) is a good start. In kubeflow mode, your trial program will run as Kubeflow job in Kubernetes cluster.|
|[__FrameworkController__](./FrameworkControllerMode.md)|NNI supports running experiment using [FrameworkController](https://github.com/Microsoft/frameworkcontroller), called frameworkcontroller mode. FrameworkController is built to orchestrate all kinds of applications on Kubernetes, you don't need to install Kubeflow for specific deep learning framework like tf-operator or pytorch-operator. Now you can use FrameworkController as the training service to run NNI experiment.|
|[__DLTS__](./DLTSMode.md)|NNI supports running experiment using [DLTS](https://github.com/microsoft/DLWorkspace.git), which is an open source toolkit, developed by Microsoft, that allows AI scientists to spin up an AI cluster in turn-key fashion.|
|[__AML__](./AMLMode.md)|NNI supports running an experiment on [AML](https://azure.microsoft.com/en-us/services/machine-learning/) , called aml mode.

## What does Training Service do?

Expand Down
3 changes: 1 addition & 2 deletions docs/en_US/TrainingService/PaiMode.md
Original file line number Diff line number Diff line change
Expand Up @@ -28,7 +28,6 @@ For example, use the following command:
```bash
sudo mount -t nfs4 gcr-openpai-infra02:/pai/data /local/mnt
```

Then the `/data` folder in container will be mounted to `/local/mnt` folder in your local machine.
You could use the following configuration in your NNI's config file:

Expand Down Expand Up @@ -87,7 +86,7 @@ paiConfig:
reuse: true
```

Note: You should set `trainingServicePlatform: pai` in NNI config YAML file if you want to start experiment in pai mode.
Note: You should set `trainingServicePlatform: pai` in NNI config YAML file if you want to start experiment in pai mode. The host field in configuration file is PAI's job submission page uri, like `10.10.5.1`, the default http protocol in NNI is `http`, if your PAI's cluster enabled https, please use the uri in `https://10.10.5.1` format.

### Trial configurations

Expand Down
2 changes: 2 additions & 0 deletions docs/en_US/_templates/index.html
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
<a href="{{ pathto('TrainingService/KubeflowMode') }}">Kubeflow</a>,
<a href="{{ pathto('TrainingService/FrameworkControllerMode') }}">FrameworkController on K8S (AKS etc.)</a>
<a href="{{ pathto('TrainingService/DLTSMode') }}">DLWorkspace (aka. DLTS)</a>
<a href="{{ pathto('TrainingService/AMLMode') }}">AML (Azure Machine Learning)</a>
and other cloud options.
</p>
<!-- Who should consider using NNI -->
Expand Down Expand Up @@ -197,6 +198,7 @@ <h1 class="title">NNI capabilities in a glance</h1>
(AKSetc.)</a>
</li>
<li><a href="{{ pathto('TrainingService/DLTSMode') }}">DLWorkspace (aka. DLTS)</a></li>
<li><a href="{{ pathto('TrainingService/AMLMode') }}">AML (Azure Machine Learning)</a></li>
</ul>
</ul>
</td>
Expand Down