This repository has been archived by the owner on Sep 18, 2024. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 1.8k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge master into dev-retiarii (#3178)
- Loading branch information
Showing
327 changed files
with
19,958 additions
and
887 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -81,6 +81,7 @@ typings/ | |
| __pycache__ | ||
| build | ||
| *.egg-info | ||
| .eggs/ | ||
| setup.pye | ||
| **/__init__.pye | ||
| **/.ipynb_checkpoints | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,131 @@ | ||
| # Research and Publications | ||
|
|
||
| We are intensively working on both tool chain and research to make automatic model design and tuning really practical and powerful. On the one hand, our main work is tool chain oriented development. On the other hand, our research works aim to improve this tool chain, rethink challenging problems in AutoML (on both system and algorithm) and propose elegant solutions. Below we list some of our research works, we encourage more research works on this topic and encourage collaboration with us. | ||
|
|
||
|
|
||
| ## System Research | ||
|
|
||
| - [Retiarii: A Deep Learning Exploratory-Training Framework](https://www.usenix.org/system/files/osdi20-zhang_quanlu.pdf) | ||
|
|
||
| ```bibtex | ||
| @inproceedings{zhang2020retiarii, | ||
| title={Retiarii: A Deep Learning Exploratory-Training Framework}, | ||
| author={Zhang, Quanlu and Han, Zhenhua and Yang, Fan and Zhang, Yuge and Liu, Zhe and Yang, Mao and Zhou, Lidong}, | ||
| booktitle={14th $\{$USENIX$\}$ Symposium on Operating Systems Design and Implementation ($\{$OSDI$\}$ 20)}, | ||
| pages={919--936}, | ||
| year={2020} | ||
| } | ||
| ``` | ||
|
|
||
| - [AutoSys: The Design and Operation of Learning-Augmented Systems](https://www.usenix.org/system/files/atc20-liang-chieh-jan.pdf) | ||
|
|
||
| ```bibtex | ||
| @inproceedings{liang2020autosys, | ||
| title={AutoSys: The Design and Operation of Learning-Augmented Systems}, | ||
| author={Liang, Chieh-Jan Mike and Xue, Hui and Yang, Mao and Zhou, Lidong and Zhu, Lifei and Li, Zhao Lucis and Wang, Zibo and Chen, Qi and Zhang, Quanlu and Liu, Chuanjie and others}, | ||
| booktitle={2020 $\{$USENIX$\}$ Annual Technical Conference ($\{$USENIX$\}$$\{$ATC$\}$ 20)}, | ||
| pages={323--336}, | ||
| year={2020} | ||
| } | ||
| ``` | ||
|
|
||
| - [Gandiva: Introspective Cluster Scheduling for Deep Learning](https://www.usenix.org/system/files/osdi18-xiao.pdf) | ||
|
|
||
| ```bibtex | ||
| @inproceedings{xiao2018gandiva, | ||
| title={Gandiva: Introspective cluster scheduling for deep learning}, | ||
| author={Xiao, Wencong and Bhardwaj, Romil and Ramjee, Ramachandran and Sivathanu, Muthian and Kwatra, Nipun and Han, Zhenhua and Patel, Pratyush and Peng, Xuan and Zhao, Hanyu and Zhang, Quanlu and others}, | ||
| booktitle={13th $\{$USENIX$\}$ Symposium on Operating Systems Design and Implementation ($\{$OSDI$\}$ 18)}, | ||
| pages={595--610}, | ||
| year={2018} | ||
| } | ||
| ``` | ||
|
|
||
| ## Algorithm Research | ||
|
|
||
| ### New Algorithms | ||
|
|
||
| - [TextNAS: A Neural Architecture Search Space Tailored for Text Representation](https://arxiv.org/pdf/1912.10729.pdf) | ||
|
|
||
| ```bibtex | ||
| @inproceedings{wang2020textnas, | ||
| title={TextNAS: A Neural Architecture Search Space Tailored for Text Representation.}, | ||
| author={Wang, Yujing and Yang, Yaming and Chen, Yiren and Bai, Jing and Zhang, Ce and Su, Guinan and Kou, Xiaoyu and Tong, Yunhai and Yang, Mao and Zhou, Lidong}, | ||
| booktitle={AAAI}, | ||
| pages={9242--9249}, | ||
| year={2020} | ||
| } | ||
| ``` | ||
|
|
||
| - [Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search](https://papers.nips.cc/paper/2020/file/d072677d210ac4c03ba046120f0802ec-Paper.pdf) | ||
|
|
||
| ```bibtex | ||
| @article{peng2020cream, | ||
| title={Cream of the Crop: Distilling Prioritized Paths For One-Shot Neural Architecture Search}, | ||
| author={Peng, Houwen and Du, Hao and Yu, Hongyuan and Li, Qi and Liao, Jing and Fu, Jianlong}, | ||
| journal={Advances in Neural Information Processing Systems}, | ||
| volume={33}, | ||
| year={2020} | ||
| } | ||
| ``` | ||
|
|
||
| - [Metis: Robustly tuning tail latencies of cloud systems](https://www.usenix.org/system/files/conference/atc18/atc18-li-zhao.pdf) | ||
|
|
||
| ```bibtex | ||
| @inproceedings{li2018metis, | ||
| title={Metis: Robustly tuning tail latencies of cloud systems}, | ||
| author={Li, Zhao Lucis and Liang, Chieh-Jan Mike and He, Wenjia and Zhu, Lianjie and Dai, Wenjun and Jiang, Jin and Sun, Guangzhong}, | ||
| booktitle={2018 $\{$USENIX$\}$ Annual Technical Conference ($\{$USENIX$\}$$\{$ATC$\}$ 18)}, | ||
| pages={981--992}, | ||
| year={2018} | ||
| } | ||
| ``` | ||
|
|
||
| - [OpEvo: An Evolutionary Method for Tensor Operator Optimization](https://arxiv.org/abs/2006.05664) | ||
|
|
||
| ```bibtex | ||
| @article{gao2020opevo, | ||
| title={OpEvo: An Evolutionary Method for Tensor Operator Optimization}, | ||
| author={Gao, Xiaotian and Wei, Cui and Zhang, Lintao and Yang, Mao}, | ||
| journal={arXiv preprint arXiv:2006.05664}, | ||
| year={2020} | ||
| } | ||
| ``` | ||
|
|
||
| ### Measurement and Understanding | ||
|
|
||
| - [Deeper insights into weight sharing in neural architecture search](https://arxiv.org/pdf/2001.01431.pdf) | ||
|
|
||
| ```bibtex | ||
| @article{zhang2020deeper, | ||
| title={Deeper insights into weight sharing in neural architecture search}, | ||
| author={Zhang, Yuge and Lin, Zejun and Jiang, Junyang and Zhang, Quanlu and Wang, Yujing and Xue, Hui and Zhang, Chen and Yang, Yaming}, | ||
| journal={arXiv preprint arXiv:2001.01431}, | ||
| year={2020} | ||
| } | ||
| ``` | ||
|
|
||
| - [How Does Supernet Help in Neural Architecture Search?](https://arxiv.org/abs/2010.08219) | ||
|
|
||
| ```bibtex | ||
| @article{zhang2020does, | ||
| title={How Does Supernet Help in Neural Architecture Search?}, | ||
| author={Zhang, Yuge and Zhang, Quanlu and Yang, Yaming}, | ||
| journal={arXiv preprint arXiv:2010.08219}, | ||
| year={2020} | ||
| } | ||
| ``` | ||
|
|
||
| ### Applications | ||
|
|
||
| - [AutoADR: Automatic Model Design for Ad Relevance](https://arxiv.org/pdf/2010.07075.pdf) | ||
|
|
||
| ```bibtex | ||
| @inproceedings{chen2020autoadr, | ||
| title={AutoADR: Automatic Model Design for Ad Relevance}, | ||
| author={Chen, Yiren and Yang, Yaming and Sun, Hong and Wang, Yujing and Xu, Yu and Shen, Wei and Zhou, Rong and Tong, Yunhai and Bai, Jing and Zhang, Ruofei}, | ||
| booktitle={Proceedings of the 29th ACM International Conference on Information \& Knowledge Management}, | ||
| pages={2365--2372}, | ||
| year={2020} | ||
| } | ||
| ``` |
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
File renamed without changes.
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,101 @@ | ||
| .. role:: raw-html(raw) | ||
| :format: html | ||
|
|
||
|
|
||
| Built-in Assessors | ||
| ================== | ||
|
|
||
| NNI provides state-of-the-art tuning algorithms within our builtin-assessors and makes them easy to use. Below is a brief overview of NNI's current builtin Assessors. | ||
|
|
||
| Note: Click the **Assessor's name** to get each Assessor's installation requirements, suggested usage scenario, and a config example. A link to a detailed description of each algorithm is provided at the end of the suggested scenario for each Assessor. | ||
|
|
||
| Currently, we support the following Assessors: | ||
|
|
||
| .. list-table:: | ||
| :header-rows: 1 | ||
| :widths: auto | ||
|
|
||
| * - Assessor | ||
| - Brief Introduction of Algorithm | ||
| * - `Medianstop <#MedianStop>`__ | ||
| - Medianstop is a simple early stopping rule. It stops a pending trial X at step S if the trial’s best objective value by step S is strictly worse than the median value of the running averages of all completed trials’ objectives reported up to step S. `Reference Paper <https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/46180.pdf>`__ | ||
| * - `Curvefitting <#Curvefitting>`__ | ||
| - Curve Fitting Assessor is an LPA (learning, predicting, assessing) algorithm. It stops a pending trial X at step S if the prediction of the final epoch's performance worse than the best final performance in the trial history. In this algorithm, we use 12 curves to fit the accuracy curve. `Reference Paper <http://aad.informatik.uni-freiburg.de/papers/15-IJCAI-Extrapolation_of_Learning_Curves.pdf>`__ | ||
|
|
||
|
|
||
| Usage of Builtin Assessors | ||
| -------------------------- | ||
|
|
||
| Usage of builtin assessors provided by the NNI SDK requires one to declare the **builtinAssessorName** and **classArgs** in the ``config.yml`` file. In this part, we will introduce the details of usage and the suggested scenarios, classArg requirements, and an example for each assessor. | ||
|
|
||
| Note: Please follow the provided format when writing your ``config.yml`` file. | ||
|
|
||
| :raw-html:`<a name="MedianStop"></a>` | ||
|
|
||
| Median Stop Assessor | ||
| ^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| .. | ||
| Builtin Assessor Name: **Medianstop** | ||
|
|
||
|
|
||
| **Suggested scenario** | ||
|
|
||
| It's applicable in a wide range of performance curves, thus, it can be used in various scenarios to speed up the tuning progress. `Detailed Description <./MedianstopAssessor.rst>`__ | ||
|
|
||
| **classArgs requirements:** | ||
|
|
||
|
|
||
| * **optimize_mode** (*maximize or minimize, optional, default = maximize*\ ) - If 'maximize', assessor will **stop** the trial with smaller expectation. If 'minimize', assessor will **stop** the trial with larger expectation. | ||
| * **start_step** (*int, optional, default = 0*\ ) - A trial is determined to be stopped or not only after receiving start_step number of reported intermediate results. | ||
|
|
||
| **Usage example:** | ||
|
|
||
| .. code-block:: yaml | ||
| # config.yml | ||
| assessor: | ||
| builtinAssessorName: Medianstop | ||
| classArgs: | ||
| optimize_mode: maximize | ||
| start_step: 5 | ||
| :raw-html:`<br>` | ||
|
|
||
| :raw-html:`<a name="Curvefitting"></a>` | ||
|
|
||
| Curve Fitting Assessor | ||
| ^^^^^^^^^^^^^^^^^^^^^^ | ||
|
|
||
| .. | ||
| Builtin Assessor Name: **Curvefitting** | ||
|
|
||
|
|
||
| **Suggested scenario** | ||
|
|
||
| It's applicable in a wide range of performance curves, thus, it can be used in various scenarios to speed up the tuning progress. Even better, it's able to handle and assess curves with similar performance. `Detailed Description <./CurvefittingAssessor.rst>`__ | ||
|
|
||
| **Note**\ , according to the original paper, only incremental functions are supported. Therefore this assessor can only be used to maximize optimization metrics. For example, it can be used for accuracy, but not for loss. | ||
|
|
||
| **classArgs requirements:** | ||
|
|
||
|
|
||
| * **epoch_num** (*int,** required***\ ) - The total number of epochs. We need to know the number of epochs to determine which points we need to predict. | ||
| * **start_step** (*int, optional, default = 6*\ ) - A trial is determined to be stopped or not only after receiving start_step number of reported intermediate results. | ||
| * **threshold** (*float, optional, default = 0.95*\ ) - The threshold that we use to decide to early stop the worst performance curve. For example: if threshold = 0.95, and the best performance in the history is 0.9, then we will stop the trial who's predicted value is lower than 0.95 * 0.9 = 0.855. | ||
| * **gap** (*int, optional, default = 1*\ ) - The gap interval between Assessor judgements. For example: if gap = 2, start_step = 6, then we will assess the result when we get 6, 8, 10, 12...intermediate results. | ||
|
|
||
| **Usage example:** | ||
|
|
||
| .. code-block:: yaml | ||
| # config.yml | ||
| assessor: | ||
| builtinAssessorName: Curvefitting | ||
| classArgs: | ||
| epoch_num: 20 | ||
| start_step: 6 | ||
| threshold: 0.95 | ||
| gap: 1 |
Oops, something went wrong.