diff --git a/README.md b/README.md index fb7148af000..e08512cefe8 100644 --- a/README.md +++ b/README.md @@ -10,13 +10,13 @@ [![Bugs](https://img.shields.io/github/issues/Microsoft/nni/bug.svg)](https://github.com/Microsoft/nni/issues?q=is%3Aissue+is%3Aopen+label%3Abug) [![Pull Requests](https://img.shields.io/github/issues-pr-raw/Microsoft/nni.svg)](https://github.com/Microsoft/nni/pulls?q=is%3Apr+is%3Aopen) [![Version](https://img.shields.io/github/release/Microsoft/nni.svg)](https://github.com/Microsoft/nni/releases) [![Join the chat at https://gitter.im/Microsoft/nni](https://badges.gitter.im/Microsoft/nni.svg)](https://gitter.im/Microsoft/nni?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge) -[![Documentation Status](https://readthedocs.org/projects/nni/badge/?version=latest)](https://nni.readthedocs.io/en/latest/?badge=latest) +[![Documentation Status](https://readthedocs.org/projects/nni/badge/?version=stable)](https://nni.readthedocs.io/en/stable/?badge=stable) [NNI Doc](https://nni.readthedocs.io/) | [简体中文](README_zh_CN.md) -**NNI (Neural Network Intelligence)** is a lightweight but powerful toolkit to help users **automate** Feature Engineering, Neural Architecture Search, Hyperparameter Tuning and Model Compression. +**NNI (Neural Network Intelligence)** is a lightweight but powerful toolkit to help users **automate** Feature Engineering, Neural Architecture Search, Hyperparameter Tuning and Model Compression. -The tool manages automated machine learning (AutoML) experiments, **dispatches and runs** experiments' trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in **different training environments** like Local Machine, Remote Servers, OpenPAI, Kubeflow, FrameworkController on K8S (AKS etc.), DLWorkspace (aka. DLTS), AML (Azure Machine Learning), AdaptDL (aka. ADL) , other cloud options and even Hybrid mode. +The tool manages automated machine learning (AutoML) experiments, **dispatches and runs** experiments' trial jobs generated by tuning algorithms to search the best neural architecture and/or hyper-parameters in **different training environments** like Local Machine, Remote Servers, OpenPAI, Kubeflow, FrameworkController on K8S (AKS etc.), DLWorkspace (aka. DLTS), AML (Azure Machine Learning), AdaptDL (aka. ADL) , other cloud options and even Hybrid mode. ## **Who should consider using NNI** @@ -72,7 +72,7 @@ Within the following table, we summarized the current NNI capabilities, we are g
  • TensorFlow
  • MXNet
  • Caffe2
    • - More...
    + More...
    - Hyperparameter Tuning + Hyperparameter Tuning - Neural Architecture Search + Neural Architecture Search - Model Compression + Model Compression - Feature Engineering (Beta) + Feature Engineering (Beta) - Early Stop Algorithms + Early Stop Algorithms @@ -197,22 +197,22 @@ Within the following table, we summarized the current NNI capabilities, we are g @@ -237,15 +237,15 @@ Windows python -m pip install --upgrade nni ``` -If you want to try latest code, please [install NNI](https://nni.readthedocs.io/en/latest/installation.html) from source code. +If you want to try latest code, please [install NNI](https://nni.readthedocs.io/en/stable/installation.html) from source code. -For detail system requirements of NNI, please refer to [here](https://nni.readthedocs.io/en/latest/Tutorial/InstallationLinux.html#system-requirements) for Linux & macOS, and [here](https://nni.readthedocs.io/en/latest/Tutorial/InstallationWin.html#system-requirements) for Windows. +For detail system requirements of NNI, please refer to [here](https://nni.readthedocs.io/en/stable/Tutorial/InstallationLinux.html#system-requirements) for Linux & macOS, and [here](https://nni.readthedocs.io/en/stable/Tutorial/InstallationWin.html#system-requirements) for Windows. Note: * If there is any privilege issue, add `--user` to install NNI in the user directory. -* Currently NNI on Windows supports local, remote and pai mode. Anaconda or Miniconda is highly recommended to install [NNI on Windows](docs/en_US/Tutorial/InstallationWin.rst). -* If there is any error like `Segmentation fault`, please refer to [FAQ](docs/en_US/Tutorial/FAQ.rst). For FAQ on Windows, please refer to [NNI on Windows](docs/en_US/Tutorial/InstallationWin.rst#faq). +* Currently NNI on Windows supports local, remote and pai mode. Anaconda or Miniconda is highly recommended to install [NNI on Windows](https://nni.readthedocs.io/en/stable/Tutorial/InstallationWin.html). +* If there is any error like `Segmentation fault`, please refer to [FAQ](https://nni.readthedocs.io/en/stable/Tutorial/FAQ.html). For FAQ on Windows, please refer to [NNI on Windows](https://nni.readthedocs.io/en/stable/Tutorial/InstallationWin.html#faq). ### **Verify installation** @@ -297,7 +297,7 @@ You can use these commands to get more information about the experiment ----------------------------------------------------------------------- ``` -* Open the `Web UI url` in your browser, you can view detail information of the experiment and all the submitted trial jobs as shown below. [Here](docs/en_US/Tutorial/WebUI.rst) are more Web UI pages. +* Open the `Web UI url` in your browser, you can view detailed information of the experiment and all the submitted trial jobs as shown below. [Here](https://nni.readthedocs.io/en/stable/Tutorial/WebUI.html) are more Web UI pages. diff --git a/docs/en_US/CommunitySharings/ModelCompressionComparison.rst b/docs/en_US/CommunitySharings/ModelCompressionComparison.rst index 758acabeb3e..de22b863902 100644 --- a/docs/en_US/CommunitySharings/ModelCompressionComparison.rst +++ b/docs/en_US/CommunitySharings/ModelCompressionComparison.rst @@ -13,7 +13,7 @@ The experiments are performed with the following pruners/datasets/models: * - Models: :githublink:`VGG16, ResNet18, ResNet50 ` + Models: :githublink:`VGG16, ResNet18, ResNet50 ` * Datasets: CIFAR-10 @@ -96,14 +96,14 @@ Implementation Details This avoids potential issues of counting them of masked models. * - The experiment code can be found :githublink:`here `. + The experiment code can be found :githublink:`here `. Experiment Result Rendering ^^^^^^^^^^^^^^^^^^^^^^^^^^^ * - If you follow the practice in the :githublink:`example `\ , for every single pruning experiment, the experiment result will be saved in JSON format as follows: + If you follow the practice in the :githublink:`example `\ , for every single pruning experiment, the experiment result will be saved in JSON format as follows: .. code-block:: json @@ -114,8 +114,8 @@ Experiment Result Rendering } * - The experiment results are saved :githublink:`here `. - You can refer to :githublink:`analyze ` to plot new performance comparison figures. + The experiment results are saved :githublink:`here `. + You can refer to :githublink:`analyze ` to plot new performance comparison figures. Contribution ------------ diff --git a/docs/en_US/Compression/Overview.rst b/docs/en_US/Compression/Overview.rst index e08c57e97d5..4cb29533fe7 100644 --- a/docs/en_US/Compression/Overview.rst +++ b/docs/en_US/Compression/Overview.rst @@ -14,7 +14,9 @@ NNI provides a model compression toolkit to help user compress and speed up thei * Provide friendly and easy-to-use compression utilities for users to dive into the compression process and results. * Concise interface for users to customize their own compression algorithms. -*Note that the interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future.* +.. note:: + Since NNI compression algorithms are not meant to compress model while NNI speedup tool can truly compress model and reduce latency. To obtain a truly compact model, users should conduct `model speedup <./ModelSpeedup.rst>`__. The interface and APIs are unified for both PyTorch and TensorFlow, currently only PyTorch version has been supported, TensorFlow version will be supported in future. + Supported Algorithms -------------------- @@ -24,7 +26,7 @@ The algorithms include pruning algorithms and quantization algorithms. Pruning Algorithms ^^^^^^^^^^^^^^^^^^ -Pruning algorithms compress the original network by removing redundant weights or channels of layers, which can reduce model complexity and address the over-fitting issue. +Pruning algorithms compress the original network by removing redundant weights or channels of layers, which can reduce model complexity and address the over-fitting issue. .. list-table:: :header-rows: 1 diff --git a/docs/en_US/Compression/Tutorial.rst b/docs/en_US/Compression/Tutorial.rst index 2cb874620fe..8c60d337fcc 100644 --- a/docs/en_US/Compression/Tutorial.rst +++ b/docs/en_US/Compression/Tutorial.rst @@ -17,30 +17,37 @@ The ``dict``\ s in the ``list`` are applied one by one, that is, the configurati There are different keys in a ``dict``. Some of them are common keys supported by all the compression algorithms: -* **op_types**\ : This is to specify what types of operations to be compressed. 'default' means following the algorithm's default setting. +* **op_types**\ : This is to specify what types of operations to be compressed. 'default' means following the algorithm's default setting. All suported module types are defined in :githublink:`default_layers.py ` for pytorch. * **op_names**\ : This is to specify by name what operations to be compressed. If this field is omitted, operations will not be filtered by it. * **exclude**\ : Default is False. If this field is True, it means the operations with specified types and names will be excluded from the compression. Some other keys are often specific to a certain algorithm, users can refer to `pruning algorithms <./Pruner.rst>`__ and `quantization algorithms <./Quantizer.rst>`__ for the keys allowed by each algorithm. -A simple example of configuration is shown below: +To prune all ``Conv2d`` layers with the sparsity of 0.6, the configuration can be written as: .. code-block:: python - [ - { - 'sparsity': 0.8, - 'op_types': ['default'] - }, - { - 'sparsity': 0.6, - 'op_names': ['op_name1', 'op_name2'] - }, - { - 'exclude': True, - 'op_names': ['op_name3'] - } - ] + [{ + 'sparsity': 0.6, + 'op_types': ['Conv2d'] + }] + +To control the sparsity of specific layers, the configuration can be written as: + +.. code-block:: python + + [{ + 'sparsity': 0.8, + 'op_types': ['default'] + }, + { + 'sparsity': 0.6, + 'op_names': ['op_name1', 'op_name2'] + }, + { + 'exclude': True, + 'op_names': ['op_name3'] + }] It means following the algorithm's default setting for compressed operations with sparsity 0.8, but for ``op_name1`` and ``op_name2`` use sparsity 0.6, and do not compress ``op_name3``. @@ -62,10 +69,10 @@ bits length of quantization, key is the quantization type, value is the quantiza .. code-block:: bash { - quant_bits: { - 'weight': 8, - 'output': 4, - }, + quant_bits: { + 'weight': 8, + 'output': 4, + }, } when the value is int type, all quantization types share same bits length. eg. @@ -73,7 +80,7 @@ when the value is int type, all quantization types share same bits length. eg. .. code-block:: bash { - quant_bits: 8, # weight or output quantization are all 8 bits + quant_bits: 8, # weight or output quantization are all 8 bits } The following example shows a more complete ``config_list``\ , it uses ``op_names`` (or ``op_types``\ ) to specify the target layers along with the quantization bits for those layers. @@ -81,25 +88,26 @@ The following example shows a more complete ``config_list``\ , it uses ``op_name .. code-block:: bash config_list = [{ - 'quant_types': ['weight'], - 'quant_bits': 8, - 'op_names': ['conv1'] - }, { - 'quant_types': ['weight'], - 'quant_bits': 4, - 'quant_start_step': 0, - 'op_names': ['conv2'] - }, { - 'quant_types': ['weight'], - 'quant_bits': 3, - 'op_names': ['fc1'] - }, - { - 'quant_types': ['weight'], - 'quant_bits': 2, - 'op_names': ['fc2'] - } - ] + 'quant_types': ['weight'], + 'quant_bits': 8, + 'op_names': ['conv1'] + }, + { + 'quant_types': ['weight'], + 'quant_bits': 4, + 'quant_start_step': 0, + 'op_names': ['conv2'] + }, + { + 'quant_types': ['weight'], + 'quant_bits': 3, + 'op_names': ['fc1'] + }, + { + 'quant_types': ['weight'], + 'quant_bits': 2, + 'op_names': ['fc2'] + }] In this example, 'op_names' is the name of layer and four layers will be quantized to different quant_bits. diff --git a/examples/model_compress/pruning/.gitignore b/examples/model_compress/pruning/.gitignore new file mode 100644 index 00000000000..b4bd197eaaf --- /dev/null +++ b/examples/model_compress/pruning/.gitignore @@ -0,0 +1,2 @@ +cifar-10-python.tar.gz +cifar-10-batches-py/ \ No newline at end of file
    drawing