Refactor model compression examples #3326

colorjam · 2021-01-22T03:59:43Z

Examples:

Add mnist example mnist_torch.py for quick start.
Merge reproduced codes into basic_pruners_torch.py.
Add knowledge distillation example basic_pruners_kd_torch.py.

Doc:

Remove pruners details.
Refine KD example.

docs/en_US/Compression/Pruner.rst

docs/en_US/TrialExample/KDExample.rst

QuanluZhang · 2021-01-27T05:17:22Z

docs/en_US/Compression/Pruner.rst

-Tensorflow
-""""""""""
-
-..  autoclass:: nni.algorithms.compression.tensorflow.pruning.LevelPruner


@liuzhe-lz do we still support tensorflow, at least provided one pruner?

The tensorflow example was removed here: https://github.com/microsoft/nni/pull/3242/files#diff-8555e1a0ab0c25960a752bdb8741ae4de1d9ab10634740970a657a9ebff38c42
You have reviewed it 🙂

@liuzhe-lz @QuanluZhang Sorry for deleting by mistake 🥺
Upload the TensorFlow example naive_prune_tf.py.

QuanluZhang · 2021-01-27T05:24:33Z

docs/en_US/Compression/Pruner.rst


+This is an one-shot pruner, which prunes filters with the smallest geometric median


better to add a little bit more description

QuanluZhang · 2021-01-27T05:25:14Z

docs/en_US/Compression/Pruner.rst

-   :target: ../../img/l1filter_pruner.png
-   :alt: 
-
+This is an one-shot pruner, which prunes the filters prunes filters in the **convolution layers**.


"prunes the filters prunes filters"?

QuanluZhang · 2021-01-27T05:26:47Z

docs/en_US/Compression/Pruner.rst

-
-.. image:: ../../img/l1filter_pruner.png
-   :target: ../../img/l1filter_pruner.png
-   :alt: 


so you think figure is not helpful?

Yes, figures look redundant.

QuanluZhang · 2021-01-27T05:28:25Z

docs/en_US/Compression/Pruner.rst

@@ -471,7 +418,8 @@ PyTorch code

   pruner.update_epoch(epoch)

-You can view :githublink:`example <examples/model_compress/pruning/model_prune_torch.py>` for more information.
+You can view :githublink:`mnist example <examples/model_compress/pruning/basic_pruners_torch.py>` for more information.
+


there is no command for this one?

QuanluZhang · 2021-01-27T05:29:26Z

docs/en_US/TrialExample/KDExample.rst

@@ -4,39 +4,35 @@ Knowledge Distillation on NNI
 KnowledgeDistill
 ----------------

-Knowledge distillation support, in `Distilling the Knowledge in a Neural Network <https://arxiv.org/abs/1503.02531>`__\ ,  the compressed model is trained to mimic a pre-trained, larger model.  This training setting is also referred to as "teacher-student",  where the large model is the teacher and the small model is the student.
+Knoiwledge Distillation (KD) is proposed in `Distilling the Knowledge in a Neural Network <https://arxiv.org/abs/1503.02531>`__\ ,  the compressed model is trained to mimic a pre-trained, larger model.  This training setting is also referred to as "teacher-student",  where the large model is the teacher and the small model is the student. KD is often used to fine-tune the pruned model.


QuanluZhang · 2021-01-27T05:33:27Z

examples/model_compress/pruning/configure_example.yaml

@@ -1,9 +0,0 @@
-AGPruner: 


what is this file used for?

no used in the examples, looks redundant, so remove it.

QuanluZhang · 2021-01-27T05:33:53Z

examples/model_compress/pruning/auto_pruners_torch.py

@@ -62,30 +62,6 @@ def get_data(dataset, data_dir, batch_size, test_batch_size):
            ])),
            batch_size=batch_size, shuffle=False, **kwargs)
        criterion = torch.nn.CrossEntropyLoss()
-    elif dataset == 'imagenet':


why imagenet is removed?

BTW, what kind of pruners is auto pruner?

suggest to add one more example which combines auto tune and pruner, for example, tuning the sparsity.

suggest to add more detailed description at the top of each of these example code files.

Imagenet is not used in comparison experiment, remove it for simplicity and clarity. Add more description at the top of the file, please review the latest version.

QuanluZhang · 2021-01-27T05:48:07Z

examples/model_compress/pruning/mnist_torch.py

+# Licensed under the MIT license.
+'''
+Examples for level pruner on mnist
+'''


this is a very simple example right? suggest to rename it to "naive_example_torch.py"

do you think "naive_prune_torch" is better?

QuanluZhang · 2021-01-27T05:49:26Z

examples/model_compress/pruning/basic_pruners_torch.py

+# Licensed under the MIT license.
+'''
+Examples for basic pruners
+'''


what is the difference between this file and basic_pruners_kd_torch.py

simplify the kd example, please review the latest version.

QuanluZhang · 2021-02-02T03:14:27Z

docs/en_US/Compression/AutoPruningUsingTuners.rst

+    trialConcurrency: 1
+    trialGpuNumber: 0
+    tuner:
+    name: grid


the indent is strange

better to tell users how to start this experiment

QuanluZhang · 2021-02-02T03:16:30Z

docs/en_US/Compression/AutoPruningUsingTuners.rst

+    pruner.compress()
+
+    # after testing
+    nni.report_final_results(acc)


would be better to add simple code to show how is acc created

QuanluZhang · 2021-02-02T03:18:01Z

docs/en_US/Compression/AutoPruningUsingTuners.rst

-   }
-
-Then we need to modify our codes for few lines
+The previous example manually choosed L2FilterPruner and pruned with a specified sparsity. Different sparsity and different pruners may have different effect on different models. This process can be done with NNI tuners.


"choosed" -> "chose"

QuanluZhang · 2021-02-02T03:20:21Z

docs/en_US/Compression/Pruner.rst

-
-   Slim Pruner **prunes channels in the convolution layers by masking corresponding scaling factors in the later BN layers**\ , L1 regularization on the scaling factors should be applied in batch normalization (BN) layers while training, scaling factors of BN layers are** globally ranked** while pruning, so the sparse model can be automatically found given sparsity.
-
+This is an one-shot pruner, which adds sparsity regularization on the scaling factors of batch normalization (BN) layers durting training to identify unimportant channels. . The channels with small scaling factor values will be pruned. For more details, please refer to `'Learning Efficient Convolutional Networks through Network Slimming' <https://arxiv.org/pdf/1708.06519.pdf>`__\.


typo: "channels. ."

QuanluZhang · 2021-02-02T03:25:45Z

docs/en_US/Compression/QuickStart.rst

@@ -45,7 +45,7 @@ After training, you get accuracy of the pruned model. You can export model weigh

   pruner.export_model(model_path='pruned_vgg19_cifar10.pth', mask_path='mask_vgg19_cifar10.pth')

-The complete code of model compression examples can be found :githublink:`here <examples/model_compress/pruning/model_prune_torch.py>`.
+Please refer :githublink:`mnist example <examples/model_compress/pruning/mnist_torch.py>` for quick start.


there is no such file

Thanks for pointing it out. The link has been updated.

QuanluZhang · 2021-02-02T03:27:38Z

docs/en_US/Compression/Pruner.rst

@@ -471,7 +416,12 @@ PyTorch code

   pruner.update_epoch(epoch)

-You can view :githublink:`example <examples/model_compress/pruning/model_prune_torch.py>` for more information.
+You can view :githublink:`mnist example <examples/model_compress/pruning/naive_example_torch.py>` for a quick start.


it is a little strange, why mention quick start here?

I want to show the simplest example for the usage of pruners.
I will refactor the QuickStart.rst. Remove the quick start in this file.

QuanluZhang · 2021-02-02T03:35:49Z

examples/model_compress/pruning/auto_pruners_torch.py

 '''
-Examples for automatic pruners
+Example for supported automatic pruning algorithms.
+In this example, we present the usage of automatic pruners (NetAdapt, AutoCompressPruner). L1, L2, FPGM pruners are aims for comparsion.


"are aims for comparison" -> "are also executed for comparison purpose"

Thanks, fix it.

QuanluZhang · 2021-02-02T03:36:45Z

examples/model_compress/pruning/basic_pruners_torch.py

+'''
+NNI example for supported basic pruning algorithms.
+In this example, we show the end-to-end pruning process: pre-training -> pruning -> fine-tuning.
+Note that pruners use masks to simiulate the real pruning. In order to obtain a real compressed model, model speed up is required.


-> simulate

QuanluZhang · 2021-02-02T03:37:54Z

examples/model_compress/pruning/finetune_kd_torch.py

+# Licensed under the MIT license.
+
+'''
+NNI exmaple for fine-tuning the pruend model with KD.


'pruend' -> 'pruned'

QuanluZhang · 2021-02-02T03:38:14Z

examples/model_compress/pruning/finetune_kd_torch.py

+
+'''
+NNI exmaple for fine-tuning the pruend model with KD.
+Run basic_pruners_torch.py first to get the pruend model.


'pruend' -> 'pruned'

QuanluZhang · 2021-02-02T03:40:29Z

examples/model_compress/pruning/basic_pruners_torch.py

+
+def get_model_optimizer_scheduler(args, device, train_loader, test_loader, criterion):
+    if args.model == 'lenet':
+        model = LeNet().to(device)


this is not right, we should use the masked model instead of the original model.

sorry, i mean the kd part

J-shang · 2021-02-03T09:03:06Z

docs/en_US/Compression/AutoPruningUsingTuners.rst

-   }
-
-Then we need to modify our codes for few lines
+The previous example manually chose L2FilterPruner and pruned with a specified sparsity. Different sparsity and different pruners may have different effect on different models. This process can be done with NNI tuners.


may have different effects...

J-shang · 2021-02-03T09:05:01Z

docs/en_US/Compression/AutoPruningUsingTuners.rst


-Last, define our task and automatically tuning pruning methods with layers sparsity
+Then, define a ``config`` file in YAML to automatically tuning model, pruning algorithm and sparisty.


sparisty -> sparsity

J-shang · 2021-02-03T09:09:21Z

docs/en_US/Compression/Pruner.rst

@@ -1,16 +1,15 @@
 Supported Pruning Algorithms on NNI
 ===================================

-We provide several pruning algorithms that support fine-grained weight pruning and structural filter pruning. **Fine-grained Pruning** generally results in  unstructured models, which need specialized haredware or software to speed up the sparse network.** Filter Pruning** achieves acceleratation by removing the entire filter.  We also provide an algorithm to control the** pruning schedule**.
+We provide several pruning algorithms that support fine-grained weight pruning and structural filter pruning. **Fine-grained Pruning** generally results in  unstructured models, which need specialized haredware or software to speed up the sparse network. **Filter Pruning** achieves acceleratation by removing the entire filter. Some pruning algorithms use one-shot method that prune weights at once based on an importance metric. Other pruning algorithms control the **pruning schedule** that prune weights during optimization, including some automatic pruning algorithms.


haredware -> hardware
acceleratation -> acceleration

J-shang · 2021-02-03T09:12:17Z

docs/en_US/Compression/Pruner.rst

-
-   Slim Pruner **prunes channels in the convolution layers by masking corresponding scaling factors in the later BN layers**\ , L1 regularization on the scaling factors should be applied in batch normalization (BN) layers while training, scaling factors of BN layers are** globally ranked** while pruning, so the sparse model can be automatically found given sparsity.
-
+This is an one-shot pruner, which adds sparsity regularization on the scaling factors of batch normalization (BN) layers durting training to identify unimportant channels. The channels with small scaling factor values will be pruned. For more details, please refer to `'Learning Efficient Convolutional Networks through Network Slimming' <https://arxiv.org/pdf/1708.06519.pdf>`__\.


durting -> during?

Thanks! Fix typo errors!

colorjam and others added 7 commits October 16, 2020 17:39

update title level

1817648

Merge branch 'master' of https://github.com/microsoft/nni

8c639c4

Merge branch 'master' of https://github.com/microsoft/nni

1fc7083

Update examples & reproduction results of darts

4f73183

Merge branch 'master' of https://github.com/colorjam/nni

c3dfe60

Add simple mnist pruning example

5754b5e

Merge branch 'master' of https://github.com/microsoft/nni

781f3be

colorjam changed the title ~~Refactor compression examples~~ Refactor model compression examples Jan 22, 2021

Merge branch 'master' into refactor-model-compression

03d1bf7

QuanluZhang mentioned this pull request Jan 22, 2021

NNI 2021 Jan~Feb Iteration Planning #3308

Closed

94 tasks

colorjam added 4 commits January 22, 2021 06:08

Update prune files

a8b7862

Add copyright

9fc466a

Add speedup in pipeline

0ee4089

Mege reproduced files into basic pruners

4e847b2

J-shang requested review from QuanluZhang and J-shang January 25, 2021 02:43

Refine kd example

fbccbfc

J-shang reviewed Jan 26, 2021

View reviewed changes

colorjam added 2 commits January 27, 2021 01:44

Update docs

7399581

Update doc

e15730f

J-shang closed this Jan 27, 2021

J-shang reopened this Jan 27, 2021

QuanluZhang reviewed Jan 27, 2021

View reviewed changes

colorjam added 4 commits January 27, 2021 10:01

Update example description

7c474ce

Update kd example and basic pruners

7e1cedf

Add autopruning config

cbc5e66

fix doc error

712738a

QuanluZhang reviewed Feb 2, 2021

View reviewed changes

Fix comments

be4aa1d

J-shang reviewed Feb 3, 2021

View reviewed changes

Fix typo errors

e133cdb

QuanluZhang approved these changes Feb 4, 2021

View reviewed changes

J-shang approved these changes Feb 4, 2021

View reviewed changes

J-shang merged commit a9dcc00 into microsoft:master Feb 4, 2021


		This is an one-shot pruner, which prunes filters with the smallest geometric median


		Slim Pruner prunes channels in the convolution layers by masking corresponding scaling factors in the later BN layers\ , L1 regularization on the scaling factors should be applied in batch normalization (BN) layers while training, scaling factors of BN layers are globally ranked while pruning, so the sparse model can be automatically found given sparsity.

		This is an one-shot pruner, which adds sparsity regularization on the scaling factors of batch normalization (BN) layers durting training to identify unimportant channels. . The channels with small scaling factor values will be pruned. For more details, please refer to `'Learning Efficient Convolutional Networks through Network Slimming' <https://arxiv.org/pdf/1708.06519.pdf>`__\.


		Last, define our task and automatically tuning pruning methods with layers sparsity
		Then, define a ``config`` file in YAML to automatically tuning model, pruning algorithm and sparisty.

Refactor model compression examples #3326

Refactor model compression examples #3326

Conversation

colorjam commented Jan 22, 2021 • edited Loading

Choose a reason for hiding this comment

liuzhe-lz Feb 2, 2021 • edited Loading

Choose a reason for hiding this comment

colorjam Feb 3, 2021 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

colorjam commented Jan 22, 2021 •

edited

Loading

liuzhe-lz Feb 2, 2021 •

edited

Loading

colorjam Feb 3, 2021 •

edited

Loading