From 8f42655b0b1c5326d2d613d1714bcc56695ae96c Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Tue, 23 Jun 2020 10:42:48 +0800 Subject: [PATCH 01/43] Update BuiltinAssessor.md --- docs/en_US/Assessor/BuiltinAssessor.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/en_US/Assessor/BuiltinAssessor.md b/docs/en_US/Assessor/BuiltinAssessor.md index 7374d6665d..ad8a88ad76 100644 --- a/docs/en_US/Assessor/BuiltinAssessor.md +++ b/docs/en_US/Assessor/BuiltinAssessor.md @@ -19,7 +19,7 @@ Note: Please follow the provided format when writing your `config.yml` file. -![](https://placehold.it/15/1589F0/000000?text=+) `Median Stop Assessor` +### Median Stop Assessor > Builtin Assessor Name: **Medianstop** @@ -47,7 +47,7 @@ assessor: -![](https://placehold.it/15/1589F0/000000?text=+) `Curve Fitting Assessor` +### Curve Fitting Assessor > Builtin Assessor Name: **Curvefitting** From eb784d07194d4605994cb6eaf8c7105a97e004e3 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Tue, 23 Jun 2020 10:46:06 +0800 Subject: [PATCH 02/43] Update BuiltinTuner.md --- docs/en_US/Tuner/BuiltinTuner.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/docs/en_US/Tuner/BuiltinTuner.md b/docs/en_US/Tuner/BuiltinTuner.md index 2cff5cded3..507da03f1a 100644 --- a/docs/en_US/Tuner/BuiltinTuner.md +++ b/docs/en_US/Tuner/BuiltinTuner.md @@ -31,7 +31,7 @@ Note: Please follow the format when you write your `config.yml` file. Some built -![](https://placehold.it/15/1589F0/000000?text=+) `TPE` +### TPE > Built-in Tuner Name: **TPE** @@ -60,7 +60,7 @@ tuner: -![](https://placehold.it/15/1589F0/000000?text=+) `Random Search` +### Random Search > Built-in Tuner Name: **Random** @@ -80,7 +80,7 @@ tuner: -![](https://placehold.it/15/1589F0/000000?text=+) `Anneal` +### Anneal > Built-in Tuner Name: **Anneal** @@ -107,7 +107,7 @@ tuner: -![](https://placehold.it/15/1589F0/000000?text=+) `Naïve Evolution` +### Naïve Evolution > Built-in Tuner Name: **Evolution** @@ -136,7 +136,7 @@ tuner: -![](https://placehold.it/15/1589F0/000000?text=+) `SMAC` +### SMAC > Built-in Tuner Name: **SMAC** @@ -173,7 +173,7 @@ tuner: -![](https://placehold.it/15/1589F0/000000?text=+) `Batch Tuner` +### Batch Tuner > Built-in Tuner Name: BatchTuner @@ -212,7 +212,7 @@ The search space file should include the high-level key `combine_params`. The ty -![](https://placehold.it/15/1589F0/000000?text=+) `Grid Search` +### Grid Search > Built-in Tuner Name: **Grid Search** @@ -234,7 +234,7 @@ tuner: -![](https://placehold.it/15/1589F0/000000?text=+) `Hyperband` +### Hyperband > Built-in Advisor Name: **Hyperband** @@ -264,7 +264,7 @@ advisor: -![](https://placehold.it/15/1589F0/000000?text=+) `Network Morphism` +### Network Morphism > Built-in Tuner Name: **NetworkMorphism** @@ -302,7 +302,7 @@ tuner: -![](https://placehold.it/15/1589F0/000000?text=+) `Metis Tuner` +### Metis Tuner > Built-in Tuner Name: **MetisTuner** @@ -330,7 +330,7 @@ tuner: -![](https://placehold.it/15/1589F0/000000?text=+) `BOHB Advisor` +### BOHB Advisor > Built-in Tuner Name: **BOHB** @@ -375,7 +375,7 @@ advisor: -![](https://placehold.it/15/1589F0/000000?text=+) `GP Tuner` +### GP Tuner > Built-in Tuner Name: **GPTuner** @@ -417,7 +417,7 @@ tuner: -![](https://placehold.it/15/1589F0/000000?text=+) `PPO Tuner` +### PPO Tuner > Built-in Tuner Name: **PPOTuner** @@ -453,7 +453,7 @@ tuner: -![](https://placehold.it/15/1589F0/000000?text=+) `PBT Tuner` +### PBT Tuner > Built-in Tuner Name: **PBTTuner** From dfa08175fc9c841532b5b5acea02489da3be9e6d Mon Sep 17 00:00:00 2001 From: tabVersion Date: Mon, 3 Aug 2020 15:28:18 +0800 Subject: [PATCH 03/43] nas bench 201 readme --- docs/en_US/NAS/SearchSpaceZoo.md | 36 ++++++++++++++++++++++++++++++-- docs/img/NAS_Bench_201.svg | 1 + 2 files changed, 35 insertions(+), 2 deletions(-) create mode 100644 docs/img/NAS_Bench_201.svg diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index 5bdb599f72..0cca536208 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -65,7 +65,7 @@ This layer is extracted from the model designed [here](https://github.com/micros ENAS Micro employs a DAG with N nodes in one cell, where the nodes represent local computations, and the edges represent the flow of information between the N nodes. One cell contains two input nodes and a single output node. The following nodes choose two previous nodes as input and apply two operations from [predefined ones](#predefined-operations-enas) then add them as the output of this node. For example, Node 4 chooses Node 1 and Node 3 as inputs then applies `MaxPool` and `AvgPool` on the inputs respectively, then adds and sums them as the output of Node 4. Nodes that are not served as input for any other node are viewed as the output of the layer. If there are multiple output nodes, the model will calculate the average of these nodes as the layer output. -One structure in the ENAS micro search space is shown below. +The ENAS micro search space is shown below. ![](../../img/NAS_ENAS_micro.svg) @@ -172,4 +172,36 @@ All supported operations for ENAS macro search are listed below. .. autoclass:: nni.nas.pytorch.search_space_zoo.enas_ops.PoolBranch ``` - +## NAS Bench 201 + +NAS Bench 201 defines a unified search space, which is algorithm agnostic. The predefined skeleton consists of a stack of cells that share the same architecture. Every cell consists of four nodes and a DAG is formed by connecting edges among them, where the node represents the sum of feature maps and the edge stands for an operation transforming a tensor from the source node to the target node. The predefined candidate operations can be found in [reference](#nas-bench-201-reference). + +The search space of NAS Bench 201 is shown below. + +![](../../img/NAS_Bench_201.svg) + +### Example code + + + +### Reference + +All supported operations for NAS Bench 201 are listed below. + +* MaxPool / AvgPool + * MaxPool: Call `torch.nn.MaxPool2d`. This operation applies a 2D max pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3` and `padding=1`. + * AvgPool: Call `torch.nn.AvgPool2d`. This operation applies a 2D average pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3` and `padding=1`. + +* Conv + * Conv1x1: On behalf of a sequence of ReLU, `nn.Cinv2d` and BatchNorm. The Conv operation's parameter is fixed to `kernal_size=1`, `padding=0`, and `dilation=1`. + * Conv3x3: On behalf of a sequence of ReLU, `nn.Cinv2d` and BatchNorm. The Conv operation's parameter is fixed to `kernal_size=3`, `padding=1`, and `dilation=1`. + +* SkipConnect + + Call `torch.nn.Identity` to connect directly to the next cell. + +* Zeroize + + Generate zero tensor indicating there is no connection from the source node to the target node. + (TODO: Check with Yuge, why not mention zeroize operation in darts operation.) + diff --git a/docs/img/NAS_Bench_201.svg b/docs/img/NAS_Bench_201.svg new file mode 100644 index 0000000000..6b08625b05 --- /dev/null +++ b/docs/img/NAS_Bench_201.svg @@ -0,0 +1 @@ +inputnode 1node 2node 3choose none/one/multiple input(s) then add them as outputthe output of the previous cellchoose one operation from MaxPool, AvgPool,, Conv1x1, Conv3x3, SkipConnect, Zeroize \ No newline at end of file From 7561edb2cf4bd0be79eebff6b0e67d39b717fb1b Mon Sep 17 00:00:00 2001 From: tabVersion Date: Mon, 3 Aug 2020 17:34:26 +0800 Subject: [PATCH 04/43] nas bench 201 darft --- docs/en_US/NAS/SearchSpaceZoo.md | 21 ++- .../nas/pytorch/search_space_zoo/__init__.py | 1 + .../pytorch/search_space_zoo/nas_bench_201.py | 65 +++++++ .../search_space_zoo/nas_bench_201_ops.py | 173 ++++++++++++++++++ 4 files changed, 259 insertions(+), 1 deletion(-) create mode 100644 src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py create mode 100644 src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index 0cca536208..c1b18bfbc2 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -180,6 +180,11 @@ The search space of NAS Bench 201 is shown below. ![](../../img/NAS_Bench_201.svg) +```eval_rst +.. autoclass:: nni.nas.pytorch.search_space_zoo.NASBench201Cell + :members: +``` + ### Example code @@ -192,10 +197,20 @@ All supported operations for NAS Bench 201 are listed below. * MaxPool: Call `torch.nn.MaxPool2d`. This operation applies a 2D max pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3` and `padding=1`. * AvgPool: Call `torch.nn.AvgPool2d`. This operation applies a 2D average pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3` and `padding=1`. + ```eval_rst + .. autoclass:: nni.nas.pytorch.search_space_zoo.nas_bench_201_ops.Pooling + :members: + ``` + * Conv * Conv1x1: On behalf of a sequence of ReLU, `nn.Cinv2d` and BatchNorm. The Conv operation's parameter is fixed to `kernal_size=1`, `padding=0`, and `dilation=1`. * Conv3x3: On behalf of a sequence of ReLU, `nn.Cinv2d` and BatchNorm. The Conv operation's parameter is fixed to `kernal_size=3`, `padding=1`, and `dilation=1`. + ```eval_rst + .. autoclass:: nni.nas.pytorch.search_space_zoo.nas_bench_201_ops.ReLUConvBN + :members: + ``` + * SkipConnect Call `torch.nn.Identity` to connect directly to the next cell. @@ -203,5 +218,9 @@ All supported operations for NAS Bench 201 are listed below. * Zeroize Generate zero tensor indicating there is no connection from the source node to the target node. - (TODO: Check with Yuge, why not mention zeroize operation in darts operation.) + + ```eval_rst + .. autoclass:: nni.nas.pytorch.search_space_zoo.nas_bench_201_ops.Zero + :members: + ``` diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/__init__.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/__init__.py index 59bb3b78d1..6a2de3bbc6 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/__init__.py +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/__init__.py @@ -2,3 +2,4 @@ from .enas_cell import ENASMicroLayer from .enas_cell import ENASMacroLayer from .enas_cell import ENASMacroGeneralModel +from .nas_bench_201 import NASBench201Cell diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py new file mode 100644 index 0000000000..75bbbc7892 --- /dev/null +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py @@ -0,0 +1,65 @@ +import torch.nn as nn +from nni.nas.pytorch.mutables import LayerChoice + +from .nas_bench_201_ops import Pooling, ReLUConvBN, Zero, FactorizedReduce + + +class NASBench201Cell(nn.Module): + """ + Builtin cell structure of NAS Bench 201. One cell contains four nodes. The First node serves as an input node + accepting the output of the previous cell. And other nodes connect to all previous nodes with an edge that + represents an operation chosen from a set to transform the tensor from the source node to the target node. + Every node accepts all its inputs and adds them as its output. + + Parameters + --- + configs: dict + # TODO: repalce this with speific parameters + cell_id: str + the name of this cell + C_in: int + the number of input channels + C_out: int + the number of output channels + stride: int + stride of the convolution + """ + def __init__(self, configs, cell_id, C_in, C_out, stride): + super(NASBench201Cell, self).__init__() + + self.NUM_NODES = 4 + self.ENABLE_VIS = False + self.layers = nn.ModuleList() + + OPS = { + "none": lambda configs, C_in, C_out, stride: Zero(configs, C_in, C_out, stride), + "avg_pool_3x3": lambda configs, C_in, C_out, stride: Pooling(configs, C_in, C_out, stride, "avg"), + "max_pool_3x3": lambda configs, C_in, C_out, stride: Pooling(configs, C_in, C_out, stride, "max"), + "nor_conv_3x3": lambda configs, C_in, C_out, stride: ReLUConvBN(configs, C_in, C_out, (3, 3), (stride, stride), (1, 1), (1, 1)), + "nor_conv_1x1": lambda configs, C_in, C_out, stride: ReLUConvBN(configs, C_in, C_out, (1, 1), (stride, stride), (0, 0), (1, 1)), + "skip_connect": lambda configs, C_in, C_out, stride: nn.Identity() if stride == 1 and C_in == C_out else FactorizedReduce(configs, C_in, C_out, stride), + } + PRIMITIVES = ["none", "skip_connect", "nor_conv_1x1", "nor_conv_3x3", "avg_pool_3x3"] + + for i in range(self.NUM_NODES): + node_ops = nn.ModuleList() + for j in range(0, i): + op_choices = [OPS[op](configs, C_in, C_out, stride if j == 0 else 1) for op in PRIMITIVES] + node_ops.append(LayerChoice(op_choices, key="edge_%d_%d" % (j, i), reduction="mean")) + self.layers.append(node_ops) + self.in_dim = C_in + self.out_dim = C_out + self.cell_id = cell_id + + def forward(self, inputs): + """ + Parameters + --- + inputs: tensor + the output of the previous layer + """ + nodes = [inputs] + for i in range(1, self.NUM_NODES): + node_feature = sum(self.layers[i][k](nodes[k]) for k in range(i)) + nodes.append(node_feature) + return nodes[-1] diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py new file mode 100644 index 0000000000..c31294a21b --- /dev/null +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py @@ -0,0 +1,173 @@ +import torch +import torch.nn as nn + + +class ReLUConvBN(nn.Module): + """ + Parameters + --- + configs: + # TODO: remove this arg + C_in: int + the number of input channels + C_out: int + the number of output channels + stride: int + stride of the convolution + padding: int + zero-padding added to both sides of the input + dilation: int + spacing between kernel elements + """ + def __init__(self, configs, C_in, C_out, kernel_size, stride, padding, dilation): + super(ReLUConvBN, self).__init__() + self.op = nn.Sequential( + nn.ReLU(inplace=False), + nn.Conv2d(C_in, C_out, kernel_size, stride=stride, + padding=padding, dilation=dilation, bias=False), + nn.BatchNorm2d(C_out, affine=configs.bn_affine, momentum=configs.bn_momentum, + track_running_stats=configs.bn_track_running_stats) + ) + + def forward(self, x): + """ + Parameters + --- + x: torch.Tensor + input tensor + """ + return self.op(x) + + +class Pooling(nn.Module): + """ + Parameters + --- + configs: + # TODO: remove this arg + C_in: int + the number of input channels + C_out: int + the number of output channels + stride: int + stride of the convolution + mode: str + must be ``avg`` or ``max`` + """ + def __init__(self, configs, C_in, C_out, stride, mode): + super(Pooling, self).__init__() + if C_in == C_out: + self.preprocess = None + else: + self.preprocess = ReLUConvBN(configs, C_in, C_out, 1, 1, 0) + if mode == "avg": + self.op = nn.AvgPool2d(3, stride=stride, padding=1, count_include_pad=False) + elif mode == "max": + self.op = nn.MaxPool2d(3, stride=stride, padding=1) + else: + raise ValueError("Invalid mode={:} in Pooling".format(mode)) + + def forward(self, x): + """ + Parameters + --- + x: torch.Tensor + input tensor + """ + if self.preprocess: + x = self.preprocess(x) + return self.op(x) + + +class Zero(nn.Module): + """ + Parameters + --- + configs: + # TODO: remove this arg + C_in: int + the number of input channels + C_out: int + the number of output channels + stride: int + stride of the convolution + """ + def __init__(self, configs, C_in, C_out, stride): + super(Zero, self).__init__() + self.C_in = C_in + self.C_out = C_out + self.stride = stride + self.is_zero = True + + def forward(self, x): + """ + Parameters + --- + x: torch.Tensor + input tensor + """ + if self.C_in == self.C_out: + if self.stride == 1: + return x.mul(0.) + else: + return x[:, :, ::self.stride, ::self.stride].mul(0.) + else: + shape = list(x.shape) + shape[1] = self.C_out + zeros = x.new_zeros(shape, dtype=x.dtype, device=x.device) + return zeros + + +class FactorizedReduce(nn.Module): + def __init__(self, configs, C_in, C_out, stride): + super(FactorizedReduce, self).__init__() + self.stride = stride + self.C_in = C_in + self.C_out = C_out + self.relu = nn.ReLU(inplace=False) + if stride == 2: + C_outs = [C_out // 2, C_out - C_out // 2] + self.convs = nn.ModuleList() + for i in range(2): + self.convs.append(nn.Conv2d(C_in, C_outs[i], 1, stride=stride, padding=0, bias=False)) + self.pad = nn.ConstantPad2d((0, 1, 0, 1), 0) + else: + raise ValueError("Invalid stride : {:}".format(stride)) + self.bn = nn.BatchNorm2d(C_out, affine=configs.bn_affine, momentum=configs.bn_momentum, + track_running_stats=configs.bn_track_running_stats) + + def forward(self, x): + x = self.relu(x) + y = self.pad(x) + out = torch.cat([self.convs[0](x), self.convs[1](y[:, :, 1:, 1:])], dim=1) + out = self.bn(out) + return out + + +class ResNetBasicblock(nn.Module): + + def __init__(self, configs, inplanes, planes, stride): + super(ResNetBasicblock, self).__init__() + assert stride == 1 or stride == 2, "invalid stride {:}".format(stride) + self.conv_a = ReLUConvBN(configs, inplanes, planes, 3, stride, 1, 1) + self.conv_b = ReLUConvBN(configs, planes, planes, 3, 1, 1, 1) + if stride == 2: + self.downsample = nn.Sequential( + nn.AvgPool2d(kernel_size=2, stride=2, padding=0), + nn.Conv2d(inplanes, planes, kernel_size=1, stride=1, padding=0, bias=False)) + elif inplanes != planes: + self.downsample = ReLUConvBN(configs, inplanes, planes, 1, 1, 0, 1) + else: + self.downsample = None + self.in_dim = inplanes + self.out_dim = planes + self.stride = stride + self.num_conv = 2 + + def forward(self, inputs): + basicblock = self.conv_a(inputs) + basicblock = self.conv_b(basicblock) + + if self.downsample is not None: + inputs = self.downsample(inputs) # residual + return inputs + basicblock \ No newline at end of file From 0daacf4cd50ddc1db2b4c00a1e160e1d4a113534 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Tue, 4 Aug 2020 15:32:30 +0800 Subject: [PATCH 05/43] base --- docs/en_US/NAS/SearchSpaceZoo.md | 4 +- .../NASBench201Dataset/__init__.py | 2 + .../NASBench201Dataset/dataset.py | 101 ++++++++++ .../downsampled_imagenet.py | 106 +++++++++++ .../NASBench201Dataset/image_data.py | 75 ++++++++ .../nas/search_space_zoo/nas_bench_201.py | 179 ++++++++++++++++++ examples/nas/search_space_zoo/utils.py | 20 ++ .../nni/nas/pytorch/nas_bench_201/__init__.py | 1 + .../nni/nas/pytorch/nas_bench_201/mutator.py | 14 ++ .../pytorch/search_space_zoo/nas_bench_201.py | 12 +- .../search_space_zoo/nas_bench_201_ops.py | 71 +++---- 11 files changed, 532 insertions(+), 53 deletions(-) create mode 100644 examples/nas/search_space_zoo/NASBench201Dataset/__init__.py create mode 100644 examples/nas/search_space_zoo/NASBench201Dataset/dataset.py create mode 100644 examples/nas/search_space_zoo/NASBench201Dataset/downsampled_imagenet.py create mode 100644 examples/nas/search_space_zoo/NASBench201Dataset/image_data.py create mode 100644 examples/nas/search_space_zoo/nas_bench_201.py create mode 100644 src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py create mode 100644 src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index c1b18bfbc2..d6659c90ed 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -174,7 +174,7 @@ All supported operations for ENAS macro search are listed below. ## NAS Bench 201 -NAS Bench 201 defines a unified search space, which is algorithm agnostic. The predefined skeleton consists of a stack of cells that share the same architecture. Every cell consists of four nodes and a DAG is formed by connecting edges among them, where the node represents the sum of feature maps and the edge stands for an operation transforming a tensor from the source node to the target node. The predefined candidate operations can be found in [reference](#nas-bench-201-reference). +NAS Bench 201 defines a unified search space, which is algorithm agnostic. The predefined skeleton consists of a stack of cells that share the same architecture. Every cell contains four nodes and a DAG is formed by connecting edges among them, where the node represents the sum of feature maps and the edge stands for an operation transforming a tensor from the source node to the target node. The predefined candidate operations can be found in [reference](#nas-bench-201-reference). The search space of NAS Bench 201 is shown below. @@ -194,6 +194,8 @@ The search space of NAS Bench 201 is shown below. All supported operations for NAS Bench 201 are listed below. * MaxPool / AvgPool + + If the number of input channels is not equal to the number of output channels, the input will first pass through a `ReLUConvBN` layer with `kernel_size=1`, `stride=1`, `padding=0`, and `dilation=0`. * MaxPool: Call `torch.nn.MaxPool2d`. This operation applies a 2D max pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3` and `padding=1`. * AvgPool: Call `torch.nn.AvgPool2d`. This operation applies a 2D average pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3` and `padding=1`. diff --git a/examples/nas/search_space_zoo/NASBench201Dataset/__init__.py b/examples/nas/search_space_zoo/NASBench201Dataset/__init__.py new file mode 100644 index 0000000000..9b300e3e7d --- /dev/null +++ b/examples/nas/search_space_zoo/NASBench201Dataset/__init__.py @@ -0,0 +1,2 @@ +from .dataset import Nb201Dataset +from .image_data import get_dataloader diff --git a/examples/nas/search_space_zoo/NASBench201Dataset/dataset.py b/examples/nas/search_space_zoo/NASBench201Dataset/dataset.py new file mode 100644 index 0000000000..0acf44b4af --- /dev/null +++ b/examples/nas/search_space_zoo/NASBench201Dataset/dataset.py @@ -0,0 +1,101 @@ +import copy +import json +import logging +import os +import pickle +from collections import OrderedDict +from copy import deepcopy + +import h5py +import numpy as np +import torch +from torch.utils.data import Dataset + + +class Nb201Dataset(Dataset): + """Modified from NAS-Bench-201 API""" + NUM_OP_CANDIDATES = 5 + MEAN = { + "cifar10-valid": 0.8365059831110725, + "cifar10": 0.8704242346666666, + "cifar100": 0.6125050707032396, + "imagenet-16-120": 0.3376692831133674 + } + STD = { + "cifar10-valid": 0.12823611180535768, + "cifar10": 0.12939587274278727, + "cifar100": 0.12148740127516067, + "imagenet-16-120": 0.09261513033735617 + } + + @staticmethod + def get_available_datasets(): + return ["cifar10-valid", "cifar10", "cifar100", "imagenet-16-120"] + + @staticmethod + def get_available_splits(): + with open("data/nb201/available_splits.json") as f: + return json.load(f) + + def __init__(self, dataset, split, data_dir="data/nb201", + acc_normalize=True, ops_onehot=True, input_dtype=np.float32): + self.dataset = dataset + self.acc_normalize = acc_normalize + self.ops_onehot = ops_onehot + self.input_dtype = input_dtype + with h5py.File(os.path.join(data_dir, "nb201.hdf5"), mode="r") as f: + self.matrix = f["matrix"][()] + self.metrics = f[dataset][()] + with open(os.path.join(data_dir, "splits.pkl"), "rb") as f: + pkl = pickle.load(f) + if split not in pkl: + raise KeyError("'%s' not in splits. Available splits are: %s" % (split, pkl.keys())) + self.samples = pkl[split] + self.seed = 0 + + def normalize(self, num): + return (num - self.MEAN[self.dataset]) / self.STD[self.dataset] + + def denormalize(self, num): + return num * self.STD[self.dataset] + self.MEAN[self.dataset] + + def get_all_metrics(self, split=None, average=False): + if split == "val": + metrics_index = 3 + elif split == "test": + metrics_index = 5 + else: + metrics_index = [3, 5] + data = self.metrics[self.samples, :, :][:, :, metrics_index] + data = data.reshape((data.shape[0], -1)) + if average: + data = np.mean(data, axis=1) + return data + + def __getitem__(self, index): + return self.retrieve_by_arch_id(self.samples[index]) + + def retrieve_by_arch_id(self, arch_id): + matrix = self.matrix[arch_id] + num_nodes = matrix.shape[0] + if self.ops_onehot: + onehot_matrix = np.zeros((num_nodes, num_nodes, self.NUM_OP_CANDIDATES), dtype=self.input_dtype) + for i in range(num_nodes): + for j in range(i + 1, num_nodes): + onehot_matrix[i, j, matrix[i][j]] = 1 + matrix = onehot_matrix + val_acc = self.metrics[arch_id, self.seed, 3] + test_acc = self.metrics[arch_id, self.seed, 5] + if self.acc_normalize: + val_acc, test_acc = self.normalize(val_acc), self.normalize(test_acc) + return { + "matrix": matrix, + "val_acc": val_acc, + "test_acc": test_acc, + "flops": self.metrics[arch_id, self.seed, 6], + "params": self.metrics[arch_id, self.seed, 7], + "latency": self.metrics[arch_id, self.seed, 8] + } + + def __len__(self): + return len(self.samples) diff --git a/examples/nas/search_space_zoo/NASBench201Dataset/downsampled_imagenet.py b/examples/nas/search_space_zoo/NASBench201Dataset/downsampled_imagenet.py new file mode 100644 index 0000000000..5d1eb7fa03 --- /dev/null +++ b/examples/nas/search_space_zoo/NASBench201Dataset/downsampled_imagenet.py @@ -0,0 +1,106 @@ +import hashlib +import os +import pickle +import sys + +import numpy as np +import torch +import torch.utils.data as data +from PIL import Image + + +def calculate_md5(fpath, chunk_size=1024 * 1024): + md5 = hashlib.md5() + with open(fpath, "rb") as f: + for chunk in iter(lambda: f.read(chunk_size), b""): + md5.update(chunk) + return md5.hexdigest() + + +def check_md5(fpath, md5, **kwargs): + return md5 == calculate_md5(fpath, **kwargs) + + +def check_integrity(fpath, md5=None): + if not os.path.isfile(fpath): + return False + if md5 is None: + return True + else: + return check_md5(fpath, md5) + + +class ImageNet16(data.Dataset): + # http://image-net.org/download-images + # A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets + # https://arxiv.org/pdf/1707.08819.pdf + + train_list = [ + ["train_data_batch_1", "27846dcaa50de8e21a7d1a35f30f0e91"], + ["train_data_batch_2", "c7254a054e0e795c69120a5727050e3f"], + ["train_data_batch_3", "4333d3df2e5ffb114b05d2ffc19b1e87"], + ["train_data_batch_4", "1620cdf193304f4a92677b695d70d10f"], + ["train_data_batch_5", "348b3c2fdbb3940c4e9e834affd3b18d"], + ["train_data_batch_6", "6e765307c242a1b3d7d5ef9139b48945"], + ["train_data_batch_7", "564926d8cbf8fc4818ba23d2faac7564"], + ["train_data_batch_8", "f4755871f718ccb653440b9dd0ebac66"], + ["train_data_batch_9", "bb6dd660c38c58552125b1a92f86b5d4"], + ["train_data_batch_10", "8f03f34ac4b42271a294f91bf480f29b"], + ] + valid_list = [ + ["val_data", "3410e3017fdaefba8d5073aaa65e4bd6"], + ] + + def __init__(self, root, train, transform, num_classes=None): + self.root = root + self.transform = transform + self.train = train # training set or valid set + if not self._check_integrity(): + raise RuntimeError("Dataset not found or corrupted.") + + if self.train: + downloaded_list = self.train_list + else: + downloaded_list = self.valid_list + self.data = [] + self.targets = [] + + for i, (file_name, checksum) in enumerate(downloaded_list): + file_path = os.path.join(self.root, file_name) + with open(file_path, "rb") as f: + if sys.version_info[0] == 2: + entry = pickle.load(f) + else: + entry = pickle.load(f, encoding="latin1") + self.data.append(entry["data"]) + self.targets.extend(entry["labels"]) + self.data = np.vstack(self.data).reshape(-1, 3, 16, 16) + self.data = np.ascontiguousarray(self.data.transpose((0, 2, 3, 1))) # convert to HWC + if num_classes is not None: + assert isinstance(num_classes, int) and \ + num_classes > 0 and \ + num_classes < 1000, \ + "invalid num_classes : {:}".format(num_classes) + self.samples = [i for i, l in enumerate(self.targets) if 1 <= l <= num_classes] + else: + self.samples = list(range(len(self.targets))) + + def __getitem__(self, index): + index = self.samples[index] + img, target = self.data[index], self.targets[index] - 1 + img = Image.fromarray(img) + if self.transform is not None: + img = self.transform(img) + return img, target + + def __len__(self): + return len(self.samples) + + def _check_integrity(self): + root = self.root + for fentry in (self.train_list + self.valid_list): + filename, md5 = fentry[0], fentry[1] + fpath = os.path.join(root, filename) + if not check_integrity(fpath, md5): + return False + return True diff --git a/examples/nas/search_space_zoo/NASBench201Dataset/image_data.py b/examples/nas/search_space_zoo/NASBench201Dataset/image_data.py new file mode 100644 index 0000000000..c875ace41b --- /dev/null +++ b/examples/nas/search_space_zoo/NASBench201Dataset/image_data.py @@ -0,0 +1,75 @@ +import json + +from torch.utils.data import DataLoader +from torch.utils.data.sampler import SubsetRandomSampler +from torchvision import datasets, transforms + +from .downsampled_imagenet import ImageNet16 + + +def get_datasets(configs): + if configs.dataset.startswith("cifar100"): + mean = [x / 255 for x in [129.3, 124.1, 112.4]] + std = [x / 255 for x in [68.2, 65.4, 70.4]] + elif configs.dataset.startswith("cifar10"): # cifar10 is a prefix of cifar100 + mean = [x / 255 for x in [125.3, 123.0, 113.9]] + std = [x / 255 for x in [63.0, 62.1, 66.7]] + elif configs.dataset.startswith('imagenet-16'): + mean = [x / 255 for x in [122.68, 116.66, 104.01]] + std = [x / 255 for x in [63.22, 61.26, 65.09]] + else: + raise NotImplementedError + + normalization = [transforms.ToTensor(), transforms.Normalize(mean, std)] + if configs.dataset.startswith("cifar10") or configs.dataset.startswith("cifar100"): + augmentation = [transforms.RandomHorizontalFlip(), transforms.RandomCrop(32, padding=4)] + elif configs.dataset.startswith("imagenet-16"): + augmentation = [transforms.RandomHorizontalFlip(), transforms.RandomCrop(16, padding=2)] + train_transform = transforms.Compose(augmentation + normalization) + test_transform = transforms.Compose(normalization) + + if configs.dataset.startswith("cifar100"): + train_data = datasets.CIFAR100("data/cifar100", train=True, transform=train_transform) + valid_data = datasets.CIFAR100("data/cifar100", train=False, transform=test_transform) + assert len(train_data) == 50000 and len(valid_data) == 10000 + elif configs.dataset.startswith("cifar10"): + train_data = datasets.CIFAR10("data/cifar10", train=True, transform=train_transform) + valid_data = datasets.CIFAR10("data/cifar10", train=False, transform=test_transform) + assert len(train_data) == 50000 and len(valid_data) == 10000 + elif configs.dataset.startswith("imagenet-16"): + num_classes = int(configs.dataset.split("-")[-1]) + train_data = ImageNet16("data/imagenet16", train=True, transform=train_transform, num_classes=num_classes) + valid_data = ImageNet16("data/imagenet16", train=False, transform=test_transform, num_classes=num_classes) + assert len(train_data) == 151700 and len(valid_data) == 6000 + return train_data, valid_data + + +def load_split(config_file_path, split_names): + with open(config_file_path, "r") as f: + data = json.load(f) + result = [] + for name in split_names: + _, arr = data[name] + result.append(list(map(int, arr))) + return result + + +def get_dataloader(configs): + train_data, valid_data = get_datasets(configs) + split_path = "data/nb201/split-{}.txt".format(configs.dataset) + kwargs = {"batch_size": configs.batch_size, "num_workers": configs.num_threads} + if configs.dataset == "cifar10-valid": + train_split, valid_split = load_split(split_path, ["train", "valid"]) + train_loader = DataLoader(train_data, sampler=SubsetRandomSampler(train_split), drop_last=True, **kwargs) + valid_loader = DataLoader(train_data, sampler=SubsetRandomSampler(valid_split), **kwargs) + test_loader = DataLoader(valid_data, **kwargs) + elif configs.dataset == "cifar10": + train_loader = DataLoader(train_data, shuffle=True, drop_last=True, **kwargs) + valid_loader = DataLoader(valid_data, **kwargs) + test_loader = DataLoader(valid_data, **kwargs) + else: + valid_split, test_split = load_split(split_path, ["xvalid", "xtest"]) + train_loader = DataLoader(train_data, shuffle=True, drop_last=True, **kwargs) + valid_loader = DataLoader(valid_data, sampler=SubsetRandomSampler(valid_split), **kwargs) + test_loader = DataLoader(valid_data, sampler=SubsetRandomSampler(test_split), **kwargs) + return train_loader, valid_loader, test_loader diff --git a/examples/nas/search_space_zoo/nas_bench_201.py b/examples/nas/search_space_zoo/nas_bench_201.py new file mode 100644 index 0000000000..d9dc6d3daa --- /dev/null +++ b/examples/nas/search_space_zoo/nas_bench_201.py @@ -0,0 +1,179 @@ +import argparse +import json +import logging +import os + +import numpy as np +import pandas as pd +import torch +import torch.nn as nn +import torch.optim as optim +from nni.nas.pytorch.utils import AverageMeterGroup +from nni.nas.pytorch.nas_bench_201.mutator import NASBench201Mutator +from nni.nas.pytorch.search_space_zoo import NASBench201Cell + +from NASBench201Dataset import Nb201Dataset, get_dataloader +from .utils import nas_bench_201_accuracy + + +class ResNetBasicBlock(nn.Module): + def __init__(self, configs, inplanes, planes, stride): + super(ResNetBasicBlock, self).__init__() + assert stride == 1 or stride == 2, "invalid stride {:}".format(stride) + self.conv_a = ReLUConvBN(configs, inplanes, planes, 3, stride, 1, 1) + self.conv_b = ReLUConvBN(configs, planes, planes, 3, 1, 1, 1) + if stride == 2: + self.downsample = nn.Sequential( + nn.AvgPool2d(kernel_size=2, stride=2, padding=0), + nn.Conv2d(inplanes, planes, kernel_size=1, stride=1, padding=0, bias=False)) + elif inplanes != planes: + self.downsample = ReLUConvBN(configs, inplanes, planes, 1, 1, 0, 1) + else: + self.downsample = None + self.in_dim = inplanes + self.out_dim = planes + self.stride = stride + self.num_conv = 2 + + def forward(self, inputs): + basicblock = self.conv_a(inputs) + basicblock = self.conv_b(basicblock) + + if self.downsample is not None: + inputs = self.downsample(inputs) + return inputs + basicblock + + +class NASBench201Network(nn.Module): + def __init__(self, configs): + super(NASBench201Network, self).__init__() + self.channels = C = configs.stem_out_channels + self.num_modules = N = configs.num_modules_per_stack + self.num_labels = 10 + if configs.dataset.startswith("cifar100"): + self.num_labels = 100 + if configs.dataset == "imagenet-16-120": + self.num_labels = 120 + + self.stem = nn.Sequential( + nn.Conv2d(3, C, kernel_size=3, padding=1, bias=False), + nn.BatchNorm2d(C, momentum=configs.bn_momentum) + ) + + layer_channels = [C] * N + [C * 2] + [C * 2] * N + [C * 4] + [C * 4] * N + layer_reductions = [False] * N + [True] + [False] * N + [True] + [False] * N + + C_prev = C + self.cells = nn.ModuleList() + for i, (C_curr, reduction) in enumerate(zip(layer_channels, layer_reductions)): + if reduction: + cell = ResNetBasicBlock(configs, C_prev, C_curr, 2) + else: + cell = NASBench201Cell(configs, i, C_prev, C_curr, 1) + self.cells.append(cell) + C_prev = C_curr + + self.lastact = nn.Sequential( + nn.BatchNorm2d(C_prev, momentum=configs.bn_momentum), + nn.ReLU(inplace=True) + ) + self.global_pooling = nn.AdaptiveAvgPool2d(1) + self.classifier = nn.Linear(C_prev, self.num_labels) + + def forward(self, inputs): + feature = self.stem(inputs) + for cell in self.cells: + feature = cell(feature) + + out = self.lastact(feature) + out = self.global_pooling(out) + out = out.view(out.size(0), -1) + logits = self.classifier(out) + + return logits + + +def train(model, loader, criterion, optimizer, scheduler, args, epoch): + logger = logging.getLogger("nb201st.train.%d" % args.index) + model.train() + meters = AverageMeterGroup() + + logger.info("Current learning rate: %.6f", optimizer.param_groups[0]["lr"]) + for step, (inputs, targets) in enumerate(loader): + inputs, targets = inputs.cuda(), targets.cuda() + optimizer.zero_grad() + logits = model(inputs) + loss = criterion(logits, targets) + loss.backward() + if args.grad_clip: + nn.utils.clip_grad_norm_(model.parameters(), args.grad_clip) + optimizer.step() + meters.update({"acc": nas_bench_201_accuracy(logits, targets), "loss": loss.item()}) + + if step % args.log_frequency == 0 or step + 1 == len(loader): + logger.info("Epoch [%d/%d] Step [%d/%d] %s", epoch, args.epochs, step + 1, len(loader), meters) + scheduler.step() + return meters.acc.avg, meters.loss.avg + + +def eval(model, split, loader, criterion, args, epoch): + logger = logging.getLogger("nb201st.eval.%d" % args.index) + model.eval() + correct = loss = total = 0. + with torch.no_grad(): + for inputs, targets in loader: + inputs, targets = inputs.cuda(), targets.cuda() + bs = targets.size(0) + logits = model(inputs) + loss += criterion(logits, targets).item() * bs + correct += nas_bench_201_accuracy(logits, targets) * bs + total += bs + logger.info("%s Epoch [%d/%d] Loss = %.6f Acc = %.6f", split.capitalize(), + epoch, args.epochs, loss / total, correct / total) + return correct / total, loss / total + + +if __name__ == '__main__': + args = parse_configs(Nb201Parser, "nb201st") + logger = logging.getLogger("nb201st.main.%d" % args.index) + + train_loader, valid_loader, test_loader = get_dataloader(args) + + nasbench = Nb201Dataset(args.dataset, args.split, input_dtype=np.int8, acc_normalize=False) + arch = nasbench[args.index] + logger.info("Arch: %s", arch) + model = NASBench201Network(args) + mutator = NASBench201Mutator(model) + mutator.reset(arch["matrix"]) + if args.resume_checkpoint: + model.load_state_dict(torch.load(args.resume_checkpoint, map_location="cpu")) + model.cuda() + + criterion = nn.CrossEntropyLoss() + if args.optimizer.startswith("sgd"): + optimizer = optim.SGD(model.parameters(), lr=args.initial_lr, momentum=args.momentum, + nesterov="nesterov" in args.optimizer, weight_decay=args.weight_decay) + else: + raise ValueError + if args.lr_scheduler == "cosine": + scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, args.epochs * len(train_loader), + eta_min=args.ending_lr) + else: + raise ValueError + + if args.keep_checkpoint != "none": + os.makedirs(args.ckpt_dir, exist_ok=True) + + csv_records = [] + for epoch in range(1, args.epochs + 1): + train_acc, train_loss = train(model, train_loader, criterion, optimizer, scheduler, args, epoch) + val_acc, val_loss = eval(model, "val", valid_loader, criterion, args, epoch) + csv_records.append({"epoch": epoch - 1, "accuracy": train_acc, "loss": train_loss, + "val_accuracy": val_acc, "val_loss": val_loss}) + if args.keep_checkpoint == "epoch": + torch.save(model.state_dict(), os.path.join(args.ckpt_dir, "epoch_%06d.pth.tar" % epoch)) + + pd.DataFrame.from_records(csv_records).to_csv(os.path.join(args.output_dir, "training.csv"), index=False) + eval(model, "test", test_loader, criterion, args, epoch) + if args.keep_checkpoint == "end": + torch.save(model.state_dict(), os.path.join(args.ckpt_dir, "final.pth.tar")) diff --git a/examples/nas/search_space_zoo/utils.py b/examples/nas/search_space_zoo/utils.py index f680db479f..dbf3b0a0eb 100644 --- a/examples/nas/search_space_zoo/utils.py +++ b/examples/nas/search_space_zoo/utils.py @@ -28,3 +28,23 @@ def reward_accuracy(output, target, topk=(1,)): batch_size = target.size(0) _, predicted = torch.max(output.data, 1) return (predicted == target).sum().item() / batch_size + + +def nas_bench_201_accuracy(output, target, topk=(1,)): + if topk == 1 or topk == (1,): + _, predict = torch.max(output.data, 1) + correct = predict.eq(target.data).cpu().sum().item() + return correct / outputs.size(0) + + maxk = max(topk) + batch_size = target.size(0) + + _, pred = output.topk(maxk, 1, True, True) + pred = pred.t() + correct = pred.eq(target.view(1, -1).expand_as(pred)) + + res = [] + for k in topk: + correct_k = correct[:k].view(-1).float().sum(0, keepdim=True) + res.append(correct_k.div_(batch_size).item()) + return res diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py new file mode 100644 index 0000000000..be2d14a9dd --- /dev/null +++ b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py @@ -0,0 +1 @@ +from .mutator import NASBench201Mutator \ No newline at end of file diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py new file mode 100644 index 0000000000..ac2d622d04 --- /dev/null +++ b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py @@ -0,0 +1,14 @@ +import torch +from nni.nas.pytorch.mutator import Mutator +from nni.nas.pytorch.mutables import LayerChoice + + +class NASBench201Mutator(Mutator): + def reset(self, matrix): + result = dict() + for mutable in self.mutables: + if isinstance(mutable, LayerChoice): + keywords_split = mutable.key.split("_") + onehot = matrix[int(keywords_split[-2]), int(keywords_split[-1])] + result[mutable.key] = torch.tensor(onehot, dtype=torch.bool) + self._cache = result diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py index 75bbbc7892..1c418e92be 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py @@ -13,8 +13,6 @@ class NASBench201Cell(nn.Module): Parameters --- - configs: dict - # TODO: repalce this with speific parameters cell_id: str the name of this cell C_in: int @@ -24,7 +22,7 @@ class NASBench201Cell(nn.Module): stride: int stride of the convolution """ - def __init__(self, configs, cell_id, C_in, C_out, stride): + def __init__(self, configs, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): super(NASBench201Cell, self).__init__() self.NUM_NODES = 4 @@ -33,10 +31,10 @@ def __init__(self, configs, cell_id, C_in, C_out, stride): OPS = { "none": lambda configs, C_in, C_out, stride: Zero(configs, C_in, C_out, stride), - "avg_pool_3x3": lambda configs, C_in, C_out, stride: Pooling(configs, C_in, C_out, stride, "avg"), - "max_pool_3x3": lambda configs, C_in, C_out, stride: Pooling(configs, C_in, C_out, stride, "max"), - "nor_conv_3x3": lambda configs, C_in, C_out, stride: ReLUConvBN(configs, C_in, C_out, (3, 3), (stride, stride), (1, 1), (1, 1)), - "nor_conv_1x1": lambda configs, C_in, C_out, stride: ReLUConvBN(configs, C_in, C_out, (1, 1), (stride, stride), (0, 0), (1, 1)), + "avg_pool_3x3": lambda configs, C_in, C_out, stride: Pooling(C_in, C_out, stride, "avg"), + "max_pool_3x3": lambda configs, C_in, C_out, stride: Pooling(C_in, C_out, stride, "max"), + "nor_conv_3x3": lambda configs, C_in, C_out, stride: ReLUConvBN(C_in, C_out, (3, 3), (stride, stride), (1, 1), (1, 1)), + "nor_conv_1x1": lambda configs, C_in, C_out, stride: ReLUConvBN(C_in, C_out, (1, 1), (stride, stride), (0, 0), (1, 1)), "skip_connect": lambda configs, C_in, C_out, stride: nn.Identity() if stride == 1 and C_in == C_out else FactorizedReduce(configs, C_in, C_out, stride), } PRIMITIVES = ["none", "skip_connect", "nor_conv_1x1", "nor_conv_3x3", "avg_pool_3x3"] diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py index c31294a21b..68ee2591a0 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py @@ -5,9 +5,7 @@ class ReLUConvBN(nn.Module): """ Parameters - --- - configs: - # TODO: remove this arg + --- C_in: int the number of input channels C_out: int @@ -18,15 +16,22 @@ class ReLUConvBN(nn.Module): zero-padding added to both sides of the input dilation: int spacing between kernel elements + bn_affine: bool + If set to ``True``, ``torch.nn.BatchNorm2d`` will have learnable affine parameters. Default: True + bn_momentun: float + the value used for the running_mean and running_var computation. Default: 0.1 + bn_track_running_stats: bool + When set to ``True``, ``torch.nn.BatchNorm2d`` tracks the running mean and variance. Default: True """ - def __init__(self, configs, C_in, C_out, kernel_size, stride, padding, dilation): + def __init__(self, C_in, C_out, kernel_size, stride, padding, dilation, + bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): super(ReLUConvBN, self).__init__() self.op = nn.Sequential( nn.ReLU(inplace=False), nn.Conv2d(C_in, C_out, kernel_size, stride=stride, padding=padding, dilation=dilation, bias=False), - nn.BatchNorm2d(C_out, affine=configs.bn_affine, momentum=configs.bn_momentum, - track_running_stats=configs.bn_track_running_stats) + nn.BatchNorm2d(C_out, affine=bn_affine, momentum=bn_momentum, + track_running_stats=bn_track_running_stats) ) def forward(self, x): @@ -43,8 +48,6 @@ class Pooling(nn.Module): """ Parameters --- - configs: - # TODO: remove this arg C_in: int the number of input channels C_out: int @@ -53,13 +56,21 @@ class Pooling(nn.Module): stride of the convolution mode: str must be ``avg`` or ``max`` + bn_affine: bool + If set to ``True``, ``torch.nn.BatchNorm2d`` will have learnable affine parameters. Default: True + bn_momentun: float + the value used for the running_mean and running_var computation. Default: 0.1 + bn_track_running_stats: bool + When set to ``True``, ``torch.nn.BatchNorm2d`` tracks the running mean and variance. Default: True """ - def __init__(self, configs, C_in, C_out, stride, mode): + def __init__(self, C_in, C_out, stride, mode, bn_affine=True, + bn_momentum=0.1, bn_track_running_stats=True): super(Pooling, self).__init__() if C_in == C_out: self.preprocess = None else: - self.preprocess = ReLUConvBN(configs, C_in, C_out, 1, 1, 0) + self.preprocess = ReLUConvBN(C_in, C_out, 1, 1, 0, 0, + bn_affine, bn_momentum, bn_track_running_stats) if mode == "avg": self.op = nn.AvgPool2d(3, stride=stride, padding=1, count_include_pad=False) elif mode == "max": @@ -83,8 +94,6 @@ class Zero(nn.Module): """ Parameters --- - configs: - # TODO: remove this arg C_in: int the number of input channels C_out: int @@ -92,7 +101,7 @@ class Zero(nn.Module): stride: int stride of the convolution """ - def __init__(self, configs, C_in, C_out, stride): + def __init__(self, C_in, C_out, stride): super(Zero, self).__init__() self.C_in = C_in self.C_out = C_out @@ -119,7 +128,8 @@ def forward(self, x): class FactorizedReduce(nn.Module): - def __init__(self, configs, C_in, C_out, stride): + def __init__(self, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, + bn_track_running_stats=True): super(FactorizedReduce, self).__init__() self.stride = stride self.C_in = C_in @@ -133,8 +143,8 @@ def __init__(self, configs, C_in, C_out, stride): self.pad = nn.ConstantPad2d((0, 1, 0, 1), 0) else: raise ValueError("Invalid stride : {:}".format(stride)) - self.bn = nn.BatchNorm2d(C_out, affine=configs.bn_affine, momentum=configs.bn_momentum, - track_running_stats=configs.bn_track_running_stats) + self.bn = nn.BatchNorm2d(C_out, affine=bn_affine, momentum=bn_momentum, + track_running_stats=bn_track_running_stats) def forward(self, x): x = self.relu(x) @@ -142,32 +152,3 @@ def forward(self, x): out = torch.cat([self.convs[0](x), self.convs[1](y[:, :, 1:, 1:])], dim=1) out = self.bn(out) return out - - -class ResNetBasicblock(nn.Module): - - def __init__(self, configs, inplanes, planes, stride): - super(ResNetBasicblock, self).__init__() - assert stride == 1 or stride == 2, "invalid stride {:}".format(stride) - self.conv_a = ReLUConvBN(configs, inplanes, planes, 3, stride, 1, 1) - self.conv_b = ReLUConvBN(configs, planes, planes, 3, 1, 1, 1) - if stride == 2: - self.downsample = nn.Sequential( - nn.AvgPool2d(kernel_size=2, stride=2, padding=0), - nn.Conv2d(inplanes, planes, kernel_size=1, stride=1, padding=0, bias=False)) - elif inplanes != planes: - self.downsample = ReLUConvBN(configs, inplanes, planes, 1, 1, 0, 1) - else: - self.downsample = None - self.in_dim = inplanes - self.out_dim = planes - self.stride = stride - self.num_conv = 2 - - def forward(self, inputs): - basicblock = self.conv_a(inputs) - basicblock = self.conv_b(basicblock) - - if self.downsample is not None: - inputs = self.downsample(inputs) # residual - return inputs + basicblock \ No newline at end of file From 8019875517f43eab154bb9b066fff1b5165220f3 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Tue, 4 Aug 2020 15:46:27 +0800 Subject: [PATCH 06/43] add readme --- docs/en_US/NAS/SearchSpaceZoo.md | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index d6659c90ed..3b954fb5f3 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -187,6 +187,15 @@ The search space of NAS Bench 201 is shown below. ### Example code +[example code](https://github.com/microsoft/nni/tree/master/examples/nas/search_space_zoo/nas_bench_201.py) + +```bash +git clone https://github.com/Microsoft/nni.git +cd nni/examples/nas/search_space_zoo +# search the best cell structure +python3 nas_bench_201.py +``` + ### Reference From 112f67b20dd02b33d38d7387e85984ce4685aa64 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Tue, 4 Aug 2020 15:49:23 +0800 Subject: [PATCH 07/43] doc update --- .../nni/nas/pytorch/search_space_zoo/nas_bench_201.py | 8 +++++++- 1 file changed, 7 insertions(+), 1 deletion(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py index 1c418e92be..bd1ac1b61c 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py @@ -21,8 +21,14 @@ class NASBench201Cell(nn.Module): the number of output channels stride: int stride of the convolution + bn_affine: bool + If set to ``True``, ``torch.nn.BatchNorm2d`` will have learnable affine parameters. Default: True + bn_momentun: float + the value used for the running_mean and running_var computation. Default: 0.1 + bn_track_running_stats: bool + When set to ``True``, ``torch.nn.BatchNorm2d`` tracks the running mean and variance. Default: True """ - def __init__(self, configs, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): + def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): super(NASBench201Cell, self).__init__() self.NUM_NODES = 4 From 7d09850faf018c63a831b4fdbf3ca8f0594fd290 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Tue, 4 Aug 2020 17:35:48 +0800 Subject: [PATCH 08/43] search space refine --- docs/en_US/NAS/SearchSpaceZoo.md | 2 +- .../nas/search_space_zoo/nas_bench_201.py | 134 ++++-------------- .../pytorch/search_space_zoo/nas_bench_201.py | 21 ++- 3 files changed, 42 insertions(+), 115 deletions(-) diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index 3b954fb5f3..4a89b537a7 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -228,7 +228,7 @@ All supported operations for NAS Bench 201 are listed below. * Zeroize - Generate zero tensor indicating there is no connection from the source node to the target node. + Generate zero tensors indicating there is no connection from the source node to the target node. ```eval_rst .. autoclass:: nni.nas.pytorch.search_space_zoo.nas_bench_201_ops.Zero diff --git a/examples/nas/search_space_zoo/nas_bench_201.py b/examples/nas/search_space_zoo/nas_bench_201.py index d9dc6d3daa..924649b3bb 100644 --- a/examples/nas/search_space_zoo/nas_bench_201.py +++ b/examples/nas/search_space_zoo/nas_bench_201.py @@ -12,22 +12,36 @@ from nni.nas.pytorch.nas_bench_201.mutator import NASBench201Mutator from nni.nas.pytorch.search_space_zoo import NASBench201Cell -from NASBench201Dataset import Nb201Dataset, get_dataloader -from .utils import nas_bench_201_accuracy + +class ReLUConvBN(nn.Module): + def __init__(self, C_in, C_out, kernel_size, stride, padding, dilation, + bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): + super(ReLUConvBN, self).__init__() + self.op = nn.Sequential( + nn.ReLU(inplace=False), + nn.Conv2d(C_in, C_out, kernel_size, stride=stride, + padding=padding, dilation=dilation, bias=False), + nn.BatchNorm2d(C_out, affine=bn_affine, momentum=bn_momentum, + track_running_stats=bn_track_running_stats) + ) + + def forward(self, x): + return self.op(x) class ResNetBasicBlock(nn.Module): - def __init__(self, configs, inplanes, planes, stride): + def __init__(self, inplanes, planes, stride, bn_affine=True, + bn_momentum=0.1, bn_track_running_stats=True): super(ResNetBasicBlock, self).__init__() assert stride == 1 or stride == 2, "invalid stride {:}".format(stride) - self.conv_a = ReLUConvBN(configs, inplanes, planes, 3, stride, 1, 1) - self.conv_b = ReLUConvBN(configs, planes, planes, 3, 1, 1, 1) + self.conv_a = ReLUConvBN(inplanes, planes, 3, stride, 1, 1, bn_affine, bn_momentum, bn_track_running_stats) + self.conv_b = ReLUConvBN(planes, planes, 3, 1, 1, 1, bn_affine, bn_momentum, bn_track_running_stats) if stride == 2: self.downsample = nn.Sequential( nn.AvgPool2d(kernel_size=2, stride=2, padding=0), nn.Conv2d(inplanes, planes, kernel_size=1, stride=1, padding=0, bias=False)) elif inplanes != planes: - self.downsample = ReLUConvBN(configs, inplanes, planes, 1, 1, 0, 1) + self.downsample = ReLUConvBN(inplanes, planes, 1, 1, 0, 1, bn_affine, bn_momentum, bn_track_running_stats) else: self.downsample = None self.in_dim = inplanes @@ -45,19 +59,19 @@ def forward(self, inputs): class NASBench201Network(nn.Module): - def __init__(self, configs): + def __init__(self, stem_out_channels, num_modules_per_stack, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): super(NASBench201Network, self).__init__() - self.channels = C = configs.stem_out_channels - self.num_modules = N = configs.num_modules_per_stack + self.channels = C = stem_out_channels + self.num_modules = N = num_modules_per_stack self.num_labels = 10 - if configs.dataset.startswith("cifar100"): - self.num_labels = 100 - if configs.dataset == "imagenet-16-120": - self.num_labels = 120 + + self.bn_momentum = bn_momentum + self.bn_affine = bn_affine + self.bn_track_running_stats = bn_track_running_stats self.stem = nn.Sequential( nn.Conv2d(3, C, kernel_size=3, padding=1, bias=False), - nn.BatchNorm2d(C, momentum=configs.bn_momentum) + nn.BatchNorm2d(C, momentum=self.bn_momentum) ) layer_channels = [C] * N + [C * 2] + [C * 2] * N + [C * 4] + [C * 4] * N @@ -67,14 +81,14 @@ def __init__(self, configs): self.cells = nn.ModuleList() for i, (C_curr, reduction) in enumerate(zip(layer_channels, layer_reductions)): if reduction: - cell = ResNetBasicBlock(configs, C_prev, C_curr, 2) + cell = ResNetBasicBlock(C_prev, C_curr, 2, self.affine, self.momentum, self.bn_track_running_stats) else: - cell = NASBench201Cell(configs, i, C_prev, C_curr, 1) + cell = NASBench201Cell(i, C_prev, C_curr, 1, self.bn_affine, self.momentum, self.bn_track_running_stats) self.cells.append(cell) C_prev = C_curr self.lastact = nn.Sequential( - nn.BatchNorm2d(C_prev, momentum=configs.bn_momentum), + nn.BatchNorm2d(C_prev, momentum=self.bn_momentum), nn.ReLU(inplace=True) ) self.global_pooling = nn.AdaptiveAvgPool2d(1) @@ -91,89 +105,3 @@ def forward(self, inputs): logits = self.classifier(out) return logits - - -def train(model, loader, criterion, optimizer, scheduler, args, epoch): - logger = logging.getLogger("nb201st.train.%d" % args.index) - model.train() - meters = AverageMeterGroup() - - logger.info("Current learning rate: %.6f", optimizer.param_groups[0]["lr"]) - for step, (inputs, targets) in enumerate(loader): - inputs, targets = inputs.cuda(), targets.cuda() - optimizer.zero_grad() - logits = model(inputs) - loss = criterion(logits, targets) - loss.backward() - if args.grad_clip: - nn.utils.clip_grad_norm_(model.parameters(), args.grad_clip) - optimizer.step() - meters.update({"acc": nas_bench_201_accuracy(logits, targets), "loss": loss.item()}) - - if step % args.log_frequency == 0 or step + 1 == len(loader): - logger.info("Epoch [%d/%d] Step [%d/%d] %s", epoch, args.epochs, step + 1, len(loader), meters) - scheduler.step() - return meters.acc.avg, meters.loss.avg - - -def eval(model, split, loader, criterion, args, epoch): - logger = logging.getLogger("nb201st.eval.%d" % args.index) - model.eval() - correct = loss = total = 0. - with torch.no_grad(): - for inputs, targets in loader: - inputs, targets = inputs.cuda(), targets.cuda() - bs = targets.size(0) - logits = model(inputs) - loss += criterion(logits, targets).item() * bs - correct += nas_bench_201_accuracy(logits, targets) * bs - total += bs - logger.info("%s Epoch [%d/%d] Loss = %.6f Acc = %.6f", split.capitalize(), - epoch, args.epochs, loss / total, correct / total) - return correct / total, loss / total - - -if __name__ == '__main__': - args = parse_configs(Nb201Parser, "nb201st") - logger = logging.getLogger("nb201st.main.%d" % args.index) - - train_loader, valid_loader, test_loader = get_dataloader(args) - - nasbench = Nb201Dataset(args.dataset, args.split, input_dtype=np.int8, acc_normalize=False) - arch = nasbench[args.index] - logger.info("Arch: %s", arch) - model = NASBench201Network(args) - mutator = NASBench201Mutator(model) - mutator.reset(arch["matrix"]) - if args.resume_checkpoint: - model.load_state_dict(torch.load(args.resume_checkpoint, map_location="cpu")) - model.cuda() - - criterion = nn.CrossEntropyLoss() - if args.optimizer.startswith("sgd"): - optimizer = optim.SGD(model.parameters(), lr=args.initial_lr, momentum=args.momentum, - nesterov="nesterov" in args.optimizer, weight_decay=args.weight_decay) - else: - raise ValueError - if args.lr_scheduler == "cosine": - scheduler = optim.lr_scheduler.CosineAnnealingLR(optimizer, args.epochs * len(train_loader), - eta_min=args.ending_lr) - else: - raise ValueError - - if args.keep_checkpoint != "none": - os.makedirs(args.ckpt_dir, exist_ok=True) - - csv_records = [] - for epoch in range(1, args.epochs + 1): - train_acc, train_loss = train(model, train_loader, criterion, optimizer, scheduler, args, epoch) - val_acc, val_loss = eval(model, "val", valid_loader, criterion, args, epoch) - csv_records.append({"epoch": epoch - 1, "accuracy": train_acc, "loss": train_loss, - "val_accuracy": val_acc, "val_loss": val_loss}) - if args.keep_checkpoint == "epoch": - torch.save(model.state_dict(), os.path.join(args.ckpt_dir, "epoch_%06d.pth.tar" % epoch)) - - pd.DataFrame.from_records(csv_records).to_csv(os.path.join(args.output_dir, "training.csv"), index=False) - eval(model, "test", test_loader, criterion, args, epoch) - if args.keep_checkpoint == "end": - torch.save(model.state_dict(), os.path.join(args.ckpt_dir, "final.pth.tar")) diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py index bd1ac1b61c..79063008ca 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py @@ -1,4 +1,5 @@ import torch.nn as nn +from collections import OrderedDict from nni.nas.pytorch.mutables import LayerChoice from .nas_bench_201_ops import Pooling, ReLUConvBN, Zero, FactorizedReduce @@ -35,21 +36,19 @@ def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1 self.ENABLE_VIS = False self.layers = nn.ModuleList() - OPS = { - "none": lambda configs, C_in, C_out, stride: Zero(configs, C_in, C_out, stride), - "avg_pool_3x3": lambda configs, C_in, C_out, stride: Pooling(C_in, C_out, stride, "avg"), - "max_pool_3x3": lambda configs, C_in, C_out, stride: Pooling(C_in, C_out, stride, "max"), - "nor_conv_3x3": lambda configs, C_in, C_out, stride: ReLUConvBN(C_in, C_out, (3, 3), (stride, stride), (1, 1), (1, 1)), - "nor_conv_1x1": lambda configs, C_in, C_out, stride: ReLUConvBN(C_in, C_out, (1, 1), (stride, stride), (0, 0), (1, 1)), - "skip_connect": lambda configs, C_in, C_out, stride: nn.Identity() if stride == 1 and C_in == C_out else FactorizedReduce(configs, C_in, C_out, stride), - } - PRIMITIVES = ["none", "skip_connect", "nor_conv_1x1", "nor_conv_3x3", "avg_pool_3x3"] + OPS = lambda layer_idx: OrderedDict([ + ("none", Zero(C_in, C_out, stride)), + ("avg_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, "avg", bn_affine, bn_momentum, bn_track_running_stats)), + ("max_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, "max", bn_affine, bn_momentum, bn_track_running_stats)), + ("nor_conv_3x3", ReLUConvBN(C_in, C_out, 3, stride if layer_idx == 0 else 1, 1, 1, bn_affine, bn_momentum, bn_track_running_stats)), + ("nor_conv_1x1", ReLUConvBN(C_in, C_out, 1, stride if layer_idx == 0 else 1, 0, 1, bn_affine, bn_momentum, bn_track_running_stats)), + ("skip_connect", nn.Identity() if stride == 1 and C_in == C_out else FactorizedReduce(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, bn_track_running_stats)) + ]) for i in range(self.NUM_NODES): node_ops = nn.ModuleList() for j in range(0, i): - op_choices = [OPS[op](configs, C_in, C_out, stride if j == 0 else 1) for op in PRIMITIVES] - node_ops.append(LayerChoice(op_choices, key="edge_%d_%d" % (j, i), reduction="mean")) + node_ops.append(LayerChoice(OPS(j), key="edge_%d_%d" % (j, i), reduction="mean")) self.layers.append(node_ops) self.in_dim = C_in self.out_dim = C_out From 62c1639ebc522997f3f924949983e3f32732b014 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Tue, 4 Aug 2020 17:40:25 +0800 Subject: [PATCH 09/43] Delete __init__.py --- examples/nas/search_space_zoo/NASBench201Dataset/__init__.py | 2 -- 1 file changed, 2 deletions(-) delete mode 100644 examples/nas/search_space_zoo/NASBench201Dataset/__init__.py diff --git a/examples/nas/search_space_zoo/NASBench201Dataset/__init__.py b/examples/nas/search_space_zoo/NASBench201Dataset/__init__.py deleted file mode 100644 index 9b300e3e7d..0000000000 --- a/examples/nas/search_space_zoo/NASBench201Dataset/__init__.py +++ /dev/null @@ -1,2 +0,0 @@ -from .dataset import Nb201Dataset -from .image_data import get_dataloader From a742e7b26720b3a45d0ce47e8f5d72f981df4948 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Tue, 4 Aug 2020 17:40:48 +0800 Subject: [PATCH 10/43] Delete dataset.py --- .../NASBench201Dataset/dataset.py | 101 ------------------ 1 file changed, 101 deletions(-) delete mode 100644 examples/nas/search_space_zoo/NASBench201Dataset/dataset.py diff --git a/examples/nas/search_space_zoo/NASBench201Dataset/dataset.py b/examples/nas/search_space_zoo/NASBench201Dataset/dataset.py deleted file mode 100644 index 0acf44b4af..0000000000 --- a/examples/nas/search_space_zoo/NASBench201Dataset/dataset.py +++ /dev/null @@ -1,101 +0,0 @@ -import copy -import json -import logging -import os -import pickle -from collections import OrderedDict -from copy import deepcopy - -import h5py -import numpy as np -import torch -from torch.utils.data import Dataset - - -class Nb201Dataset(Dataset): - """Modified from NAS-Bench-201 API""" - NUM_OP_CANDIDATES = 5 - MEAN = { - "cifar10-valid": 0.8365059831110725, - "cifar10": 0.8704242346666666, - "cifar100": 0.6125050707032396, - "imagenet-16-120": 0.3376692831133674 - } - STD = { - "cifar10-valid": 0.12823611180535768, - "cifar10": 0.12939587274278727, - "cifar100": 0.12148740127516067, - "imagenet-16-120": 0.09261513033735617 - } - - @staticmethod - def get_available_datasets(): - return ["cifar10-valid", "cifar10", "cifar100", "imagenet-16-120"] - - @staticmethod - def get_available_splits(): - with open("data/nb201/available_splits.json") as f: - return json.load(f) - - def __init__(self, dataset, split, data_dir="data/nb201", - acc_normalize=True, ops_onehot=True, input_dtype=np.float32): - self.dataset = dataset - self.acc_normalize = acc_normalize - self.ops_onehot = ops_onehot - self.input_dtype = input_dtype - with h5py.File(os.path.join(data_dir, "nb201.hdf5"), mode="r") as f: - self.matrix = f["matrix"][()] - self.metrics = f[dataset][()] - with open(os.path.join(data_dir, "splits.pkl"), "rb") as f: - pkl = pickle.load(f) - if split not in pkl: - raise KeyError("'%s' not in splits. Available splits are: %s" % (split, pkl.keys())) - self.samples = pkl[split] - self.seed = 0 - - def normalize(self, num): - return (num - self.MEAN[self.dataset]) / self.STD[self.dataset] - - def denormalize(self, num): - return num * self.STD[self.dataset] + self.MEAN[self.dataset] - - def get_all_metrics(self, split=None, average=False): - if split == "val": - metrics_index = 3 - elif split == "test": - metrics_index = 5 - else: - metrics_index = [3, 5] - data = self.metrics[self.samples, :, :][:, :, metrics_index] - data = data.reshape((data.shape[0], -1)) - if average: - data = np.mean(data, axis=1) - return data - - def __getitem__(self, index): - return self.retrieve_by_arch_id(self.samples[index]) - - def retrieve_by_arch_id(self, arch_id): - matrix = self.matrix[arch_id] - num_nodes = matrix.shape[0] - if self.ops_onehot: - onehot_matrix = np.zeros((num_nodes, num_nodes, self.NUM_OP_CANDIDATES), dtype=self.input_dtype) - for i in range(num_nodes): - for j in range(i + 1, num_nodes): - onehot_matrix[i, j, matrix[i][j]] = 1 - matrix = onehot_matrix - val_acc = self.metrics[arch_id, self.seed, 3] - test_acc = self.metrics[arch_id, self.seed, 5] - if self.acc_normalize: - val_acc, test_acc = self.normalize(val_acc), self.normalize(test_acc) - return { - "matrix": matrix, - "val_acc": val_acc, - "test_acc": test_acc, - "flops": self.metrics[arch_id, self.seed, 6], - "params": self.metrics[arch_id, self.seed, 7], - "latency": self.metrics[arch_id, self.seed, 8] - } - - def __len__(self): - return len(self.samples) From f6f7b9b9b6657b0233c9ae6d59bbc4191504fcc5 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Tue, 4 Aug 2020 17:41:07 +0800 Subject: [PATCH 11/43] Delete downsampled_imagenet.py --- .../downsampled_imagenet.py | 106 ------------------ 1 file changed, 106 deletions(-) delete mode 100644 examples/nas/search_space_zoo/NASBench201Dataset/downsampled_imagenet.py diff --git a/examples/nas/search_space_zoo/NASBench201Dataset/downsampled_imagenet.py b/examples/nas/search_space_zoo/NASBench201Dataset/downsampled_imagenet.py deleted file mode 100644 index 5d1eb7fa03..0000000000 --- a/examples/nas/search_space_zoo/NASBench201Dataset/downsampled_imagenet.py +++ /dev/null @@ -1,106 +0,0 @@ -import hashlib -import os -import pickle -import sys - -import numpy as np -import torch -import torch.utils.data as data -from PIL import Image - - -def calculate_md5(fpath, chunk_size=1024 * 1024): - md5 = hashlib.md5() - with open(fpath, "rb") as f: - for chunk in iter(lambda: f.read(chunk_size), b""): - md5.update(chunk) - return md5.hexdigest() - - -def check_md5(fpath, md5, **kwargs): - return md5 == calculate_md5(fpath, **kwargs) - - -def check_integrity(fpath, md5=None): - if not os.path.isfile(fpath): - return False - if md5 is None: - return True - else: - return check_md5(fpath, md5) - - -class ImageNet16(data.Dataset): - # http://image-net.org/download-images - # A Downsampled Variant of ImageNet as an Alternative to the CIFAR datasets - # https://arxiv.org/pdf/1707.08819.pdf - - train_list = [ - ["train_data_batch_1", "27846dcaa50de8e21a7d1a35f30f0e91"], - ["train_data_batch_2", "c7254a054e0e795c69120a5727050e3f"], - ["train_data_batch_3", "4333d3df2e5ffb114b05d2ffc19b1e87"], - ["train_data_batch_4", "1620cdf193304f4a92677b695d70d10f"], - ["train_data_batch_5", "348b3c2fdbb3940c4e9e834affd3b18d"], - ["train_data_batch_6", "6e765307c242a1b3d7d5ef9139b48945"], - ["train_data_batch_7", "564926d8cbf8fc4818ba23d2faac7564"], - ["train_data_batch_8", "f4755871f718ccb653440b9dd0ebac66"], - ["train_data_batch_9", "bb6dd660c38c58552125b1a92f86b5d4"], - ["train_data_batch_10", "8f03f34ac4b42271a294f91bf480f29b"], - ] - valid_list = [ - ["val_data", "3410e3017fdaefba8d5073aaa65e4bd6"], - ] - - def __init__(self, root, train, transform, num_classes=None): - self.root = root - self.transform = transform - self.train = train # training set or valid set - if not self._check_integrity(): - raise RuntimeError("Dataset not found or corrupted.") - - if self.train: - downloaded_list = self.train_list - else: - downloaded_list = self.valid_list - self.data = [] - self.targets = [] - - for i, (file_name, checksum) in enumerate(downloaded_list): - file_path = os.path.join(self.root, file_name) - with open(file_path, "rb") as f: - if sys.version_info[0] == 2: - entry = pickle.load(f) - else: - entry = pickle.load(f, encoding="latin1") - self.data.append(entry["data"]) - self.targets.extend(entry["labels"]) - self.data = np.vstack(self.data).reshape(-1, 3, 16, 16) - self.data = np.ascontiguousarray(self.data.transpose((0, 2, 3, 1))) # convert to HWC - if num_classes is not None: - assert isinstance(num_classes, int) and \ - num_classes > 0 and \ - num_classes < 1000, \ - "invalid num_classes : {:}".format(num_classes) - self.samples = [i for i, l in enumerate(self.targets) if 1 <= l <= num_classes] - else: - self.samples = list(range(len(self.targets))) - - def __getitem__(self, index): - index = self.samples[index] - img, target = self.data[index], self.targets[index] - 1 - img = Image.fromarray(img) - if self.transform is not None: - img = self.transform(img) - return img, target - - def __len__(self): - return len(self.samples) - - def _check_integrity(self): - root = self.root - for fentry in (self.train_list + self.valid_list): - filename, md5 = fentry[0], fentry[1] - fpath = os.path.join(root, filename) - if not check_integrity(fpath, md5): - return False - return True From bdd9a098273ddb1334220ec292a1d29baa577d5e Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Tue, 4 Aug 2020 17:41:21 +0800 Subject: [PATCH 12/43] Delete image_data.py --- .../NASBench201Dataset/image_data.py | 75 ------------------- 1 file changed, 75 deletions(-) delete mode 100644 examples/nas/search_space_zoo/NASBench201Dataset/image_data.py diff --git a/examples/nas/search_space_zoo/NASBench201Dataset/image_data.py b/examples/nas/search_space_zoo/NASBench201Dataset/image_data.py deleted file mode 100644 index c875ace41b..0000000000 --- a/examples/nas/search_space_zoo/NASBench201Dataset/image_data.py +++ /dev/null @@ -1,75 +0,0 @@ -import json - -from torch.utils.data import DataLoader -from torch.utils.data.sampler import SubsetRandomSampler -from torchvision import datasets, transforms - -from .downsampled_imagenet import ImageNet16 - - -def get_datasets(configs): - if configs.dataset.startswith("cifar100"): - mean = [x / 255 for x in [129.3, 124.1, 112.4]] - std = [x / 255 for x in [68.2, 65.4, 70.4]] - elif configs.dataset.startswith("cifar10"): # cifar10 is a prefix of cifar100 - mean = [x / 255 for x in [125.3, 123.0, 113.9]] - std = [x / 255 for x in [63.0, 62.1, 66.7]] - elif configs.dataset.startswith('imagenet-16'): - mean = [x / 255 for x in [122.68, 116.66, 104.01]] - std = [x / 255 for x in [63.22, 61.26, 65.09]] - else: - raise NotImplementedError - - normalization = [transforms.ToTensor(), transforms.Normalize(mean, std)] - if configs.dataset.startswith("cifar10") or configs.dataset.startswith("cifar100"): - augmentation = [transforms.RandomHorizontalFlip(), transforms.RandomCrop(32, padding=4)] - elif configs.dataset.startswith("imagenet-16"): - augmentation = [transforms.RandomHorizontalFlip(), transforms.RandomCrop(16, padding=2)] - train_transform = transforms.Compose(augmentation + normalization) - test_transform = transforms.Compose(normalization) - - if configs.dataset.startswith("cifar100"): - train_data = datasets.CIFAR100("data/cifar100", train=True, transform=train_transform) - valid_data = datasets.CIFAR100("data/cifar100", train=False, transform=test_transform) - assert len(train_data) == 50000 and len(valid_data) == 10000 - elif configs.dataset.startswith("cifar10"): - train_data = datasets.CIFAR10("data/cifar10", train=True, transform=train_transform) - valid_data = datasets.CIFAR10("data/cifar10", train=False, transform=test_transform) - assert len(train_data) == 50000 and len(valid_data) == 10000 - elif configs.dataset.startswith("imagenet-16"): - num_classes = int(configs.dataset.split("-")[-1]) - train_data = ImageNet16("data/imagenet16", train=True, transform=train_transform, num_classes=num_classes) - valid_data = ImageNet16("data/imagenet16", train=False, transform=test_transform, num_classes=num_classes) - assert len(train_data) == 151700 and len(valid_data) == 6000 - return train_data, valid_data - - -def load_split(config_file_path, split_names): - with open(config_file_path, "r") as f: - data = json.load(f) - result = [] - for name in split_names: - _, arr = data[name] - result.append(list(map(int, arr))) - return result - - -def get_dataloader(configs): - train_data, valid_data = get_datasets(configs) - split_path = "data/nb201/split-{}.txt".format(configs.dataset) - kwargs = {"batch_size": configs.batch_size, "num_workers": configs.num_threads} - if configs.dataset == "cifar10-valid": - train_split, valid_split = load_split(split_path, ["train", "valid"]) - train_loader = DataLoader(train_data, sampler=SubsetRandomSampler(train_split), drop_last=True, **kwargs) - valid_loader = DataLoader(train_data, sampler=SubsetRandomSampler(valid_split), **kwargs) - test_loader = DataLoader(valid_data, **kwargs) - elif configs.dataset == "cifar10": - train_loader = DataLoader(train_data, shuffle=True, drop_last=True, **kwargs) - valid_loader = DataLoader(valid_data, **kwargs) - test_loader = DataLoader(valid_data, **kwargs) - else: - valid_split, test_split = load_split(split_path, ["xvalid", "xtest"]) - train_loader = DataLoader(train_data, shuffle=True, drop_last=True, **kwargs) - valid_loader = DataLoader(valid_data, sampler=SubsetRandomSampler(valid_split), **kwargs) - test_loader = DataLoader(valid_data, sampler=SubsetRandomSampler(test_split), **kwargs) - return train_loader, valid_loader, test_loader From 03eb9fe249573e14ae451913038e2fa4c43fac06 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Tue, 4 Aug 2020 17:42:19 +0800 Subject: [PATCH 13/43] remove nb201 related --- examples/nas/search_space_zoo/utils.py | 20 -------------------- 1 file changed, 20 deletions(-) diff --git a/examples/nas/search_space_zoo/utils.py b/examples/nas/search_space_zoo/utils.py index dbf3b0a0eb..f680db479f 100644 --- a/examples/nas/search_space_zoo/utils.py +++ b/examples/nas/search_space_zoo/utils.py @@ -28,23 +28,3 @@ def reward_accuracy(output, target, topk=(1,)): batch_size = target.size(0) _, predicted = torch.max(output.data, 1) return (predicted == target).sum().item() / batch_size - - -def nas_bench_201_accuracy(output, target, topk=(1,)): - if topk == 1 or topk == (1,): - _, predict = torch.max(output.data, 1) - correct = predict.eq(target.data).cpu().sum().item() - return correct / outputs.size(0) - - maxk = max(topk) - batch_size = target.size(0) - - _, pred = output.topk(maxk, 1, True, True) - pred = pred.t() - correct = pred.eq(target.view(1, -1).expand_as(pred)) - - res = [] - for k in topk: - correct_k = correct[:k].view(-1).float().sum(0, keepdim=True) - res.append(correct_k.div_(batch_size).item()) - return res From 1c4a2154cb07c5c7bba69e63494c3518168695dc Mon Sep 17 00:00:00 2001 From: ltg001 Date: Wed, 5 Aug 2020 09:35:14 +0800 Subject: [PATCH 14/43] format code --- .../pytorch/search_space_zoo/nas_bench_201.py | 17 ++++++++++++----- 1 file changed, 12 insertions(+), 5 deletions(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py index 79063008ca..09d7ba9bd0 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py @@ -29,6 +29,7 @@ class NASBench201Cell(nn.Module): bn_track_running_stats: bool When set to ``True``, ``torch.nn.BatchNorm2d`` tracks the running mean and variance. Default: True """ + def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): super(NASBench201Cell, self).__init__() @@ -38,11 +39,17 @@ def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1 OPS = lambda layer_idx: OrderedDict([ ("none", Zero(C_in, C_out, stride)), - ("avg_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, "avg", bn_affine, bn_momentum, bn_track_running_stats)), - ("max_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, "max", bn_affine, bn_momentum, bn_track_running_stats)), - ("nor_conv_3x3", ReLUConvBN(C_in, C_out, 3, stride if layer_idx == 0 else 1, 1, 1, bn_affine, bn_momentum, bn_track_running_stats)), - ("nor_conv_1x1", ReLUConvBN(C_in, C_out, 1, stride if layer_idx == 0 else 1, 0, 1, bn_affine, bn_momentum, bn_track_running_stats)), - ("skip_connect", nn.Identity() if stride == 1 and C_in == C_out else FactorizedReduce(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, bn_track_running_stats)) + ("avg_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, "avg", bn_affine, bn_momentum, + bn_track_running_stats)), + ("max_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, "max", bn_affine, bn_momentum, + bn_track_running_stats)), + ("nor_conv_3x3", ReLUConvBN(C_in, C_out, 3, stride if layer_idx == 0 else 1, 1, 1, bn_affine, bn_momentum, + bn_track_running_stats)), + ("nor_conv_1x1", ReLUConvBN(C_in, C_out, 1, stride if layer_idx == 0 else 1, 0, 1, bn_affine, bn_momentum, + bn_track_running_stats)), + ("skip_connect", nn.Identity() if stride == 1 and C_in == C_out + else FactorizedReduce(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, + bn_track_running_stats)) ]) for i in range(self.NUM_NODES): From dab01a98c836bf219d1650a7d3c7dc0b9eda9c7f Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Thu, 6 Aug 2020 15:34:50 +0800 Subject: [PATCH 15/43] format --- .../pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py index 68ee2591a0..129198566f 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py @@ -5,7 +5,7 @@ class ReLUConvBN(nn.Module): """ Parameters - --- + --- C_in: int the number of input channels C_out: int From 0021749924b09f718a0ef7f706080fd807a3f82f Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Thu, 6 Aug 2020 15:36:21 +0800 Subject: [PATCH 16/43] format --- .../pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py index 09d7ba9bd0..bba32369a6 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py @@ -1,5 +1,5 @@ -import torch.nn as nn from collections import OrderedDict +import torch.nn as nn from nni.nas.pytorch.mutables import LayerChoice from .nas_bench_201_ops import Pooling, ReLUConvBN, Zero, FactorizedReduce @@ -10,7 +10,7 @@ class NASBench201Cell(nn.Module): Builtin cell structure of NAS Bench 201. One cell contains four nodes. The First node serves as an input node accepting the output of the previous cell. And other nodes connect to all previous nodes with an edge that represents an operation chosen from a set to transform the tensor from the source node to the target node. - Every node accepts all its inputs and adds them as its output. + Every node accepts all its inputs and adds them as its output. Parameters --- @@ -48,7 +48,7 @@ def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1 ("nor_conv_1x1", ReLUConvBN(C_in, C_out, 1, stride if layer_idx == 0 else 1, 0, 1, bn_affine, bn_momentum, bn_track_running_stats)), ("skip_connect", nn.Identity() if stride == 1 and C_in == C_out - else FactorizedReduce(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, + else FactorizedReduce(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, bn_track_running_stats)) ]) From ddf897ac21fa1ea20000664911d81ba83ce1c8e1 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Fri, 7 Aug 2020 10:16:04 +0800 Subject: [PATCH 17/43] disable pylint inline --- src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py index ac2d622d04..25771bc121 100644 --- a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py +++ b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py @@ -10,5 +10,5 @@ def reset(self, matrix): if isinstance(mutable, LayerChoice): keywords_split = mutable.key.split("_") onehot = matrix[int(keywords_split[-2]), int(keywords_split[-1])] - result[mutable.key] = torch.tensor(onehot, dtype=torch.bool) + result[mutable.key] = torch.tensor(onehot, dtype=torch.bool) # pylint: disable=E1102 self._cache = result From 1e2dfadb332c145357c129c696380001a7826592 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Fri, 7 Aug 2020 10:17:17 +0800 Subject: [PATCH 18/43] format --- src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py index bba32369a6..44f1730505 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py @@ -49,7 +49,7 @@ def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1 bn_track_running_stats)), ("skip_connect", nn.Identity() if stride == 1 and C_in == C_out else FactorizedReduce(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, - bn_track_running_stats)) + bn_track_running_stats)) ]) for i in range(self.NUM_NODES): From 1ad4372ae55691d2637695ea70ee728588903635 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Tue, 11 Aug 2020 17:20:04 +0800 Subject: [PATCH 19/43] update --- docs/en_US/NAS/Overview.md | 1 + docs/en_US/NAS/SearchSpaceZoo.md | 18 ++++++------------ examples/nas/search_space_zoo/nas_bench_201.py | 3 +-- .../nni/nas/pytorch/nas_bench_201/__init__.py | 2 +- .../nni/nas/pytorch/nas_bench_201/mutator.py | 14 -------------- .../nas_bench_201.py | 6 ++---- .../nas_bench_201_ops.py | 9 +-------- .../nas/pytorch/search_space_zoo/__init__.py | 1 - 8 files changed, 12 insertions(+), 42 deletions(-) delete mode 100644 src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py rename src/sdk/pynni/nni/nas/pytorch/{search_space_zoo => nas_bench_201}/nas_bench_201.py (90%) rename src/sdk/pynni/nni/nas/pytorch/{search_space_zoo => nas_bench_201}/nas_bench_201_ops.py (93%) diff --git a/docs/en_US/NAS/Overview.md b/docs/en_US/NAS/Overview.md index 9a6610bed2..deffe22ed7 100644 --- a/docs/en_US/NAS/Overview.md +++ b/docs/en_US/NAS/Overview.md @@ -64,6 +64,7 @@ Search Space Zoo contains the following NAS cells: * [DartsCell](./SearchSpaceZoo.md#DartsCell) * [ENAS micro](./SearchSpaceZoo.md#ENASMicroLayer) * [ENAS macro](./SearchSpaceZoo.md#ENASMacroLayer) +* [NAS Bench 201](./SearchSpaceZoo.md#nas-bench-201) ## Using NNI API to Write Your Search Space diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index 4a89b537a7..b9d6fd9acf 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -189,12 +189,7 @@ The search space of NAS Bench 201 is shown below. [example code](https://github.com/microsoft/nni/tree/master/examples/nas/search_space_zoo/nas_bench_201.py) -```bash -git clone https://github.com/Microsoft/nni.git -cd nni/examples/nas/search_space_zoo -# search the best cell structure -python3 nas_bench_201.py -``` +*No executable example provided.* @@ -202,14 +197,13 @@ python3 nas_bench_201.py All supported operations for NAS Bench 201 are listed below. -* MaxPool / AvgPool +* AvgPool If the number of input channels is not equal to the number of output channels, the input will first pass through a `ReLUConvBN` layer with `kernel_size=1`, `stride=1`, `padding=0`, and `dilation=0`. - * MaxPool: Call `torch.nn.MaxPool2d`. This operation applies a 2D max pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3` and `padding=1`. - * AvgPool: Call `torch.nn.AvgPool2d`. This operation applies a 2D average pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3` and `padding=1`. + Call `torch.nn.AvgPool2d`. This operation applies a 2D average pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3` and `padding=1`. ```eval_rst - .. autoclass:: nni.nas.pytorch.search_space_zoo.nas_bench_201_ops.Pooling + .. autoclass:: nni.nas.pytorch.nas_bench_201.nas_bench_201_ops.Pooling :members: ``` @@ -218,7 +212,7 @@ All supported operations for NAS Bench 201 are listed below. * Conv3x3: On behalf of a sequence of ReLU, `nn.Cinv2d` and BatchNorm. The Conv operation's parameter is fixed to `kernal_size=3`, `padding=1`, and `dilation=1`. ```eval_rst - .. autoclass:: nni.nas.pytorch.search_space_zoo.nas_bench_201_ops.ReLUConvBN + .. autoclass:: nni.nas.pytorch.nas_bench_201.nas_bench_201_ops.ReLUConvBN :members: ``` @@ -231,7 +225,7 @@ All supported operations for NAS Bench 201 are listed below. Generate zero tensors indicating there is no connection from the source node to the target node. ```eval_rst - .. autoclass:: nni.nas.pytorch.search_space_zoo.nas_bench_201_ops.Zero + .. autoclass:: nni.nas.pytorch.nas_bench_201.nas_bench_201_ops.Zero :members: ``` diff --git a/examples/nas/search_space_zoo/nas_bench_201.py b/examples/nas/search_space_zoo/nas_bench_201.py index 924649b3bb..7ce0cd4e8a 100644 --- a/examples/nas/search_space_zoo/nas_bench_201.py +++ b/examples/nas/search_space_zoo/nas_bench_201.py @@ -9,8 +9,7 @@ import torch.nn as nn import torch.optim as optim from nni.nas.pytorch.utils import AverageMeterGroup -from nni.nas.pytorch.nas_bench_201.mutator import NASBench201Mutator -from nni.nas.pytorch.search_space_zoo import NASBench201Cell +from nni.nas.pytorch.nas_bench_201 import NASBench201Cell class ReLUConvBN(nn.Module): diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py index be2d14a9dd..0a6d327e57 100644 --- a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py +++ b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py @@ -1 +1 @@ -from .mutator import NASBench201Mutator \ No newline at end of file +from .nas_bench_201 import NASBench201Cell diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py deleted file mode 100644 index ac2d622d04..0000000000 --- a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/mutator.py +++ /dev/null @@ -1,14 +0,0 @@ -import torch -from nni.nas.pytorch.mutator import Mutator -from nni.nas.pytorch.mutables import LayerChoice - - -class NASBench201Mutator(Mutator): - def reset(self, matrix): - result = dict() - for mutable in self.mutables: - if isinstance(mutable, LayerChoice): - keywords_split = mutable.key.split("_") - onehot = matrix[int(keywords_split[-2]), int(keywords_split[-1])] - result[mutable.key] = torch.tensor(onehot, dtype=torch.bool) - self._cache = result diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py similarity index 90% rename from src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py rename to src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py index 09d7ba9bd0..cb3bd5fd0b 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201.py +++ b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py @@ -39,9 +39,7 @@ def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1 OPS = lambda layer_idx: OrderedDict([ ("none", Zero(C_in, C_out, stride)), - ("avg_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, "avg", bn_affine, bn_momentum, - bn_track_running_stats)), - ("max_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, "max", bn_affine, bn_momentum, + ("avg_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, bn_track_running_stats)), ("nor_conv_3x3", ReLUConvBN(C_in, C_out, 3, stride if layer_idx == 0 else 1, 1, 1, bn_affine, bn_momentum, bn_track_running_stats)), @@ -55,7 +53,7 @@ def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1 for i in range(self.NUM_NODES): node_ops = nn.ModuleList() for j in range(0, i): - node_ops.append(LayerChoice(OPS(j), key="edge_%d_%d" % (j, i), reduction="mean")) + node_ops.append(LayerChoice(OPS(j), key="%d_%d" % (j, i), reduction="mean")) self.layers.append(node_ops) self.in_dim = C_in self.out_dim = C_out diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py similarity index 93% rename from src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py rename to src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py index 68ee2591a0..a57bec1a9a 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/nas_bench_201_ops.py +++ b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py @@ -54,8 +54,6 @@ class Pooling(nn.Module): the number of output channels stride: int stride of the convolution - mode: str - must be ``avg`` or ``max`` bn_affine: bool If set to ``True``, ``torch.nn.BatchNorm2d`` will have learnable affine parameters. Default: True bn_momentun: float @@ -71,12 +69,7 @@ def __init__(self, C_in, C_out, stride, mode, bn_affine=True, else: self.preprocess = ReLUConvBN(C_in, C_out, 1, 1, 0, 0, bn_affine, bn_momentum, bn_track_running_stats) - if mode == "avg": - self.op = nn.AvgPool2d(3, stride=stride, padding=1, count_include_pad=False) - elif mode == "max": - self.op = nn.MaxPool2d(3, stride=stride, padding=1) - else: - raise ValueError("Invalid mode={:} in Pooling".format(mode)) + self.op = nn.AvgPool2d(3, stride=stride, padding=1, count_include_pad=False) def forward(self, x): """ diff --git a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/__init__.py b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/__init__.py index 6a2de3bbc6..59bb3b78d1 100644 --- a/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/__init__.py +++ b/src/sdk/pynni/nni/nas/pytorch/search_space_zoo/__init__.py @@ -2,4 +2,3 @@ from .enas_cell import ENASMicroLayer from .enas_cell import ENASMacroLayer from .enas_cell import ENASMacroGeneralModel -from .nas_bench_201 import NASBench201Cell From 1c70f2a0bf1ae9fb1b4a839582c0aa9d0048d571 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Tue, 11 Aug 2020 23:38:24 +0800 Subject: [PATCH 20/43] update doc --- docs/en_US/NAS/SearchSpaceZoo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index b9d6fd9acf..1847207f3c 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -189,7 +189,7 @@ The search space of NAS Bench 201 is shown below. [example code](https://github.com/microsoft/nni/tree/master/examples/nas/search_space_zoo/nas_bench_201.py) -*No executable example provided.* +*No executable example provided.* From 8d929553ef7714431205266be11495b3f01afad0 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Wed, 12 Aug 2020 18:11:38 +0800 Subject: [PATCH 21/43] doc fix --- docs/en_US/NAS/SearchSpaceZoo.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index 1847207f3c..b5fa6088a4 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -181,7 +181,7 @@ The search space of NAS Bench 201 is shown below. ![](../../img/NAS_Bench_201.svg) ```eval_rst -.. autoclass:: nni.nas.pytorch.search_space_zoo.NASBench201Cell +.. autoclass:: nni.nas.pytorch.nas_bench_201.NASBench201Cell :members: ``` From 2eae0be8c656e743cb90cec63de630fe22176c04 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Thu, 13 Aug 2020 09:08:31 +0800 Subject: [PATCH 22/43] op attr fix --- .../pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py index ce95c81e0d..aa60f8854f 100644 --- a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py +++ b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py @@ -61,8 +61,7 @@ class Pooling(nn.Module): bn_track_running_stats: bool When set to ``True``, ``torch.nn.BatchNorm2d`` tracks the running mean and variance. Default: True """ - def __init__(self, C_in, C_out, stride, mode, bn_affine=True, - bn_momentum=0.1, bn_track_running_stats=True): + def __init__(self, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): super(Pooling, self).__init__() if C_in == C_out: self.preprocess = None From 2621c122ed0de02725ce457165d15d1cbecde41a Mon Sep 17 00:00:00 2001 From: tabVersion Date: Mon, 17 Aug 2020 16:43:20 +0800 Subject: [PATCH 23/43] basic example --- .../nas/search_space_zoo/nas_bench_201.py | 47 ++++++++++++++++++- 1 file changed, 45 insertions(+), 2 deletions(-) diff --git a/examples/nas/search_space_zoo/nas_bench_201.py b/examples/nas/search_space_zoo/nas_bench_201.py index 7ce0cd4e8a..8471ae6ade 100644 --- a/examples/nas/search_space_zoo/nas_bench_201.py +++ b/examples/nas/search_space_zoo/nas_bench_201.py @@ -10,7 +10,13 @@ import torch.optim as optim from nni.nas.pytorch.utils import AverageMeterGroup from nni.nas.pytorch.nas_bench_201 import NASBench201Cell +from nni.nas.pytorch.callbacks import ArchitectureCheckpoint, LRSchedulerCallback +from nni.nas.pytorch.darts import DartsTrainer +from utils import accuracy +import datasets + +logger = logging.getLogger('nni') class ReLUConvBN(nn.Module): def __init__(self, C_in, C_out, kernel_size, stride, padding, dilation, @@ -80,9 +86,9 @@ def __init__(self, stem_out_channels, num_modules_per_stack, bn_affine=True, bn_ self.cells = nn.ModuleList() for i, (C_curr, reduction) in enumerate(zip(layer_channels, layer_reductions)): if reduction: - cell = ResNetBasicBlock(C_prev, C_curr, 2, self.affine, self.momentum, self.bn_track_running_stats) + cell = ResNetBasicBlock(C_prev, C_curr, 2, self.bn_affine, self.bn_momentum, self.bn_track_running_stats) else: - cell = NASBench201Cell(i, C_prev, C_curr, 1, self.bn_affine, self.momentum, self.bn_track_running_stats) + cell = NASBench201Cell(i, C_prev, C_curr, 1, self.bn_affine, self.bn_momentum, self.bn_track_running_stats) self.cells.append(cell) C_prev = C_curr @@ -104,3 +110,40 @@ def forward(self, inputs): logits = self.classifier(out) return logits + +if __name__ == '__main__': + parser = argparse.ArgumentParser("nb201") + parser.add_argument('--stem_out_channels', default=16, type=int) + parser.add_argument('--unrolled', default=False, action='store_true') + parser.add_argument('--batch-size', default=64, type=int) + parser.add_argument('--epochs', default=50, type=int) + parser.add_argument('--num_modules_per_stack', default=5, type=int) + parser.add_argument('--log-frequency', default=10, type=int) + parser.add_argument('--bn_momentum', default=0.1, type=int) + parser.add_argument('--bn_affine', default=True, type=bool) + parser.add_argument('--bn_track_running_stats', default=True, type=bool) + args = parser.parse_args() + + dataset_train, dataset_valid = datasets.get_dataset("cifar10") + model = NASBench201Network(stem_out_channels=args.stem_out_channels, + num_modules_per_stack=args.num_modules_per_stack, + bn_affine=args.bn_affine, + bn_momentum=args.bn_momentum, + bn_track_running_stats=self.bn_track_running_stats) + criterion = nn.CrossEntropyLoss() + + optim = torch.optim.SGD(model.parameters(), 0.025) + lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, args.epochs, eta_min=0.001) + + trainer = DartsTrainer(model, + loss=criterion, + metrics=lambda output, target: accuracy(output, target, topk=(1,)), + optimizer=optim, + num_epochs=args.epochs, + dataset_train=dataset_train, + dataset_valid=dataset_valid, + batch_size=args.batch_size, + log_frequency=args.log_frequency, + unrolled=args.unrolled, + callbacks=[LRSchedulerCallback(lr_scheduler), ArchitectureCheckpoint("./checkpoints")]) + trainer.train() From 556b4759ea657f66ff50550d171dfb9b467636d2 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Mon, 17 Aug 2020 17:41:35 +0800 Subject: [PATCH 24/43] typo fix --- docs/en_US/NAS/SearchSpaceZoo.md | 7 ++++++- examples/nas/search_space_zoo/nas_bench_201.py | 2 +- 2 files changed, 7 insertions(+), 2 deletions(-) diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index b5fa6088a4..456435f091 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -189,7 +189,12 @@ The search space of NAS Bench 201 is shown below. [example code](https://github.com/microsoft/nni/tree/master/examples/nas/search_space_zoo/nas_bench_201.py) -*No executable example provided.* +```bash +# for structure searching +git clone https://github.com/Microsoft/nni.git +cd nni/examples/nas/search_space_zoo +python3 nas_bench_201.py +``` diff --git a/examples/nas/search_space_zoo/nas_bench_201.py b/examples/nas/search_space_zoo/nas_bench_201.py index 8471ae6ade..9b78e920b0 100644 --- a/examples/nas/search_space_zoo/nas_bench_201.py +++ b/examples/nas/search_space_zoo/nas_bench_201.py @@ -129,7 +129,7 @@ def forward(self, inputs): num_modules_per_stack=args.num_modules_per_stack, bn_affine=args.bn_affine, bn_momentum=args.bn_momentum, - bn_track_running_stats=self.bn_track_running_stats) + bn_track_running_stats=args.bn_track_running_stats) criterion = nn.CrossEntropyLoss() optim = torch.optim.SGD(model.parameters(), 0.025) From aa3dcbdb18d65838fde4b9562a1eb3c532a98fe1 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Tue, 18 Aug 2020 13:47:28 +0800 Subject: [PATCH 25/43] Update nas_bench_201.py --- src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py index 5d647acdbf..0c3ae4d560 100644 --- a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py +++ b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py @@ -24,7 +24,7 @@ class NASBench201Cell(nn.Module): stride of the convolution bn_affine: bool If set to ``True``, ``torch.nn.BatchNorm2d`` will have learnable affine parameters. Default: True - bn_momentun: float + bn_momentum: float the value used for the running_mean and running_var computation. Default: 0.1 bn_track_running_stats: bool When set to ``True``, ``torch.nn.BatchNorm2d`` tracks the running mean and variance. Default: True From 91faaff253c685fe4ab1ec372130f90fb537222b Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Tue, 18 Aug 2020 13:48:58 +0800 Subject: [PATCH 26/43] Update nas_bench_201.py --- src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py | 1 - 1 file changed, 1 deletion(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py index 0c3ae4d560..8b2e2fbff0 100644 --- a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py +++ b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py @@ -34,7 +34,6 @@ def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1 super(NASBench201Cell, self).__init__() self.NUM_NODES = 4 - self.ENABLE_VIS = False self.layers = nn.ModuleList() OPS = lambda layer_idx: OrderedDict([ From 0ee1e4e7d02fbcf632a43bd3b84d96ce19b09c87 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Tue, 18 Aug 2020 13:54:28 +0800 Subject: [PATCH 27/43] rename_op --- src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py index 5d647acdbf..20fc09976c 100644 --- a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py +++ b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py @@ -41,9 +41,9 @@ def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1 ("none", Zero(C_in, C_out, stride)), ("avg_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, bn_track_running_stats)), - ("nor_conv_3x3", ReLUConvBN(C_in, C_out, 3, stride if layer_idx == 0 else 1, 1, 1, bn_affine, bn_momentum, + ("conv_3x3", ReLUConvBN(C_in, C_out, 3, stride if layer_idx == 0 else 1, 1, 1, bn_affine, bn_momentum, bn_track_running_stats)), - ("nor_conv_1x1", ReLUConvBN(C_in, C_out, 1, stride if layer_idx == 0 else 1, 0, 1, bn_affine, bn_momentum, + ("conv_1x1", ReLUConvBN(C_in, C_out, 1, stride if layer_idx == 0 else 1, 0, 1, bn_affine, bn_momentum, bn_track_running_stats)), ("skip_connect", nn.Identity() if stride == 1 and C_in == C_out else FactorizedReduce(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, From 9897920d6a2c942a6bc5b5eef01aac83e202bb49 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Tue, 18 Aug 2020 14:36:06 +0800 Subject: [PATCH 28/43] indent fix --- src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py index fe00c7b021..e71ba0c46a 100644 --- a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py +++ b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py @@ -41,9 +41,9 @@ def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1 ("avg_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, bn_track_running_stats)), ("conv_3x3", ReLUConvBN(C_in, C_out, 3, stride if layer_idx == 0 else 1, 1, 1, bn_affine, bn_momentum, - bn_track_running_stats)), + bn_track_running_stats)), ("conv_1x1", ReLUConvBN(C_in, C_out, 1, stride if layer_idx == 0 else 1, 0, 1, bn_affine, bn_momentum, - bn_track_running_stats)), + bn_track_running_stats)), ("skip_connect", nn.Identity() if stride == 1 and C_in == C_out else FactorizedReduce(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, bn_track_running_stats)) From f24984ed55b20962401610b6d76623820cd41ebd Mon Sep 17 00:00:00 2001 From: tabVersion Date: Wed, 19 Aug 2020 18:22:54 +0800 Subject: [PATCH 29/43] example --- .../nas/search_space_zoo/fixed-architecture.json | 8 ++++++++ examples/nas/search_space_zoo/nas_bench_201.py | 16 +++++++++++++++- 2 files changed, 23 insertions(+), 1 deletion(-) create mode 100644 examples/nas/search_space_zoo/fixed-architecture.json diff --git a/examples/nas/search_space_zoo/fixed-architecture.json b/examples/nas/search_space_zoo/fixed-architecture.json new file mode 100644 index 0000000000..ae72840874 --- /dev/null +++ b/examples/nas/search_space_zoo/fixed-architecture.json @@ -0,0 +1,8 @@ +{ + "0_1": "avg_pool_3x3", + "0_2": "conv_1x1", + "1_2": "skip_connect", + "0_3": "conv_1x1", + "1_3": "skip_connect", + "2_3": "skip_connect" +} diff --git a/examples/nas/search_space_zoo/nas_bench_201.py b/examples/nas/search_space_zoo/nas_bench_201.py index 9b78e920b0..476b08b744 100644 --- a/examples/nas/search_space_zoo/nas_bench_201.py +++ b/examples/nas/search_space_zoo/nas_bench_201.py @@ -2,6 +2,7 @@ import json import logging import os +import pprint import numpy as np import pandas as pd @@ -10,6 +11,8 @@ import torch.optim as optim from nni.nas.pytorch.utils import AverageMeterGroup from nni.nas.pytorch.nas_bench_201 import NASBench201Cell +from nni.nas.pytorch.fixed import apply_fixed_architecture +from nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats from nni.nas.pytorch.callbacks import ArchitectureCheckpoint, LRSchedulerCallback from nni.nas.pytorch.darts import DartsTrainer from utils import accuracy @@ -122,6 +125,8 @@ def forward(self, inputs): parser.add_argument('--bn_momentum', default=0.1, type=int) parser.add_argument('--bn_affine', default=True, type=bool) parser.add_argument('--bn_track_running_stats', default=True, type=bool) + parser.add_argument('--arch', default=None, help='json file which should meet requirements in NAS-Bench-201') + parser.add_argument('--visualization', default=False, action='store_true') args = parser.parse_args() dataset_train, dataset_valid = datasets.get_dataset("cifar10") @@ -130,10 +135,17 @@ def forward(self, inputs): bn_affine=args.bn_affine, bn_momentum=args.bn_momentum, bn_track_running_stats=args.bn_track_running_stats) - criterion = nn.CrossEntropyLoss() optim = torch.optim.SGD(model.parameters(), 0.025) lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, args.epochs, eta_min=0.001) + criterion = nn.CrossEntropyLoss() + + if args.arch is not None: + with open(args.arch, 'r') as f: + arch = json.load(f) + for trial in query_nb201_trial_stats(arch, 200, 'cifar100'): + pprint.pprint(trial) + apply_fixed_architecture(model, args.arch) trainer = DartsTrainer(model, loss=criterion, @@ -146,4 +158,6 @@ def forward(self, inputs): log_frequency=args.log_frequency, unrolled=args.unrolled, callbacks=[LRSchedulerCallback(lr_scheduler), ArchitectureCheckpoint("./checkpoints")]) + if args.visualization: + trainer.enable_visualization() trainer.train() From 8594fcdeba13534653619d5870c199513efd23a5 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Thu, 20 Aug 2020 10:31:03 +0800 Subject: [PATCH 30/43] add example --- .../nas/search_space_zoo/nas_bench_201.py | 28 +++++++++++-------- 1 file changed, 16 insertions(+), 12 deletions(-) diff --git a/examples/nas/search_space_zoo/nas_bench_201.py b/examples/nas/search_space_zoo/nas_bench_201.py index 476b08b744..d032b6b411 100644 --- a/examples/nas/search_space_zoo/nas_bench_201.py +++ b/examples/nas/search_space_zoo/nas_bench_201.py @@ -9,13 +9,14 @@ import torch import torch.nn as nn import torch.optim as optim +from nni.nas.pytorch import enas from nni.nas.pytorch.utils import AverageMeterGroup from nni.nas.pytorch.nas_bench_201 import NASBench201Cell from nni.nas.pytorch.fixed import apply_fixed_architecture from nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats from nni.nas.pytorch.callbacks import ArchitectureCheckpoint, LRSchedulerCallback from nni.nas.pytorch.darts import DartsTrainer -from utils import accuracy +from utils import accuracy, reward_accuracy import datasets @@ -147,17 +148,20 @@ def forward(self, inputs): pprint.pprint(trial) apply_fixed_architecture(model, args.arch) - trainer = DartsTrainer(model, - loss=criterion, - metrics=lambda output, target: accuracy(output, target, topk=(1,)), - optimizer=optim, - num_epochs=args.epochs, - dataset_train=dataset_train, - dataset_valid=dataset_valid, - batch_size=args.batch_size, - log_frequency=args.log_frequency, - unrolled=args.unrolled, - callbacks=[LRSchedulerCallback(lr_scheduler), ArchitectureCheckpoint("./checkpoints")]) + mutator = enas.EnasMutator(model, tanh_constant=1.1, cell_exit_extra_step=True) + trainer = enas.EnasTrainer(model, + loss=criterion, + metrics=lambda output, target: accuracy(output, target, topk=(1,)), + reward_function=reward_accuracy, + optimizer=optim, + callbacks=[LRSchedulerCallback(lr_scheduler), ArchitectureCheckpoint("./checkpoints")], + batch_size=args.batch_size, + num_epochs=args.epochs, + dataset_train=dataset_train, + dataset_valid=dataset_valid, + log_frequency=args.log_frequency, + mutator=mutator) + if args.visualization: trainer.enable_visualization() trainer.train() From 85344880f9fa392943a308281363c4dfd60f5b42 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Wed, 9 Sep 2020 10:14:58 +0800 Subject: [PATCH 31/43] Update SearchSpaceZoo.md --- docs/en_US/NAS/SearchSpaceZoo.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index 456435f091..1e320f477b 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -172,7 +172,7 @@ All supported operations for ENAS macro search are listed below. .. autoclass:: nni.nas.pytorch.search_space_zoo.enas_ops.PoolBranch ``` -## NAS Bench 201 +## NAS-Bench-201 NAS Bench 201 defines a unified search space, which is algorithm agnostic. The predefined skeleton consists of a stack of cells that share the same architecture. Every cell contains four nodes and a DAG is formed by connecting edges among them, where the node represents the sum of feature maps and the edge stands for an operation transforming a tensor from the source node to the target node. The predefined candidate operations can be found in [reference](#nas-bench-201-reference). @@ -213,8 +213,8 @@ All supported operations for NAS Bench 201 are listed below. ``` * Conv - * Conv1x1: On behalf of a sequence of ReLU, `nn.Cinv2d` and BatchNorm. The Conv operation's parameter is fixed to `kernal_size=1`, `padding=0`, and `dilation=1`. - * Conv3x3: On behalf of a sequence of ReLU, `nn.Cinv2d` and BatchNorm. The Conv operation's parameter is fixed to `kernal_size=3`, `padding=1`, and `dilation=1`. + * Conv1x1: Consist of a sequence of ReLU, `nn.Cinv2d` and BatchNorm. The Conv operation's parameter is fixed to `kernal_size=1`, `padding=0`, and `dilation=1`. + * Conv3x3: Consist of a sequence of ReLU, `nn.Cinv2d` and BatchNorm. The Conv operation's parameter is fixed to `kernal_size=3`, `padding=1`, and `dilation=1`. ```eval_rst .. autoclass:: nni.nas.pytorch.nas_bench_201.nas_bench_201_ops.ReLUConvBN From 2955422675678ea4828948a50ebc2d11427c9425 Mon Sep 17 00:00:00 2001 From: ltg001 Date: Wed, 9 Sep 2020 13:37:08 +0800 Subject: [PATCH 32/43] fixed arch should be trained directly --- .../nas/search_space_zoo/nas_bench_201.py | 40 ++++++++++++++++++- 1 file changed, 39 insertions(+), 1 deletion(-) diff --git a/examples/nas/search_space_zoo/nas_bench_201.py b/examples/nas/search_space_zoo/nas_bench_201.py index d032b6b411..06cf0109f5 100644 --- a/examples/nas/search_space_zoo/nas_bench_201.py +++ b/examples/nas/search_space_zoo/nas_bench_201.py @@ -9,6 +9,7 @@ import torch import torch.nn as nn import torch.optim as optim +from torch.utils.data import DataLoader from nni.nas.pytorch import enas from nni.nas.pytorch.utils import AverageMeterGroup from nni.nas.pytorch.nas_bench_201 import NASBench201Cell @@ -115,11 +116,43 @@ def forward(self, inputs): return logits + +def train(args, model, train_dataloader, valid_dataloader, criterion, optimizer, device): + model.train() + for epoch in range(args.epochs): + for batch_idx, (data, target) in enumerate(train_dataloader): + data, target = data.to(device), target.to(device) + optimizer.zero_grad() + output = model(data) + loss = criterion(output, target) + loss.backward() + optimizer.step() + + if batch_idx % args.log_frequency == 0: + logger.info('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format( + epoch, batch_idx * len(data), len(train_dataloader.dataset), + 100. * batch_idx / len(train_dataloader), loss.item())) + model.eval() + correct = 0 + test_loss = 0.0 + for data, target in valid_dataloader: + data, target = data.to(device), target.to(device) + output = model(data) + test_loss += criterion(output, target, reduction='sum').item() + pred = output.argmax(dim=1, keepdim=True) + correct += pred.eq(target.view_as(pred)).sum().item() + test_loss /= len(valid_dataloader.dataset) + accuracy = 100. * correct / len(valid_dataloader.dataset) + logger.info('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(test_loss, correct, + len(valid_dataloader.dataset), accuracy)) + model.train() + + if __name__ == '__main__': parser = argparse.ArgumentParser("nb201") parser.add_argument('--stem_out_channels', default=16, type=int) parser.add_argument('--unrolled', default=False, action='store_true') - parser.add_argument('--batch-size', default=64, type=int) + parser.add_argument('--batch_size', default=64, type=int) parser.add_argument('--epochs', default=50, type=int) parser.add_argument('--num_modules_per_stack', default=5, type=int) parser.add_argument('--log-frequency', default=10, type=int) @@ -147,6 +180,11 @@ def forward(self, inputs): for trial in query_nb201_trial_stats(arch, 200, 'cifar100'): pprint.pprint(trial) apply_fixed_architecture(model, args.arch) + dataloader_train = DataLoader(dataset_train, batch_size=args.batch_size, shuffle=True, num_workers=0) + dataloader_valid = DataLoader(dataset_valid, batch_size=args.batch_size, shuffle=True, num_workers=0) + train(args, model, dataloader_train, dataloader_valid, criterion, optim, + torch.device('cuda' if torch.cuda.is_available() else 'cpu')) + exit(0) mutator = enas.EnasMutator(model, tanh_constant=1.1, cell_exit_extra_step=True) trainer = enas.EnasTrainer(model, From 7be65d764c53c22408f4d9f93ba10a47428362b9 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Wed, 9 Sep 2020 14:21:18 +0800 Subject: [PATCH 33/43] fix --- examples/nas/search_space_zoo/nas_bench_201.py | 1 + 1 file changed, 1 insertion(+) diff --git a/examples/nas/search_space_zoo/nas_bench_201.py b/examples/nas/search_space_zoo/nas_bench_201.py index 06cf0109f5..e6bf497935 100644 --- a/examples/nas/search_space_zoo/nas_bench_201.py +++ b/examples/nas/search_space_zoo/nas_bench_201.py @@ -118,6 +118,7 @@ def forward(self, inputs): def train(args, model, train_dataloader, valid_dataloader, criterion, optimizer, device): + model = model.to(device) model.train() for epoch in range(args.epochs): for batch_idx, (data, target) in enumerate(train_dataloader): From 1662f2f8ac70c22f8f6b3f316a6bf7be32256932 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Wed, 9 Sep 2020 15:58:28 +0800 Subject: [PATCH 34/43] rename --- examples/nas/search_space_zoo/nasbench201.py | 206 +++++++++++++++++++ 1 file changed, 206 insertions(+) create mode 100644 examples/nas/search_space_zoo/nasbench201.py diff --git a/examples/nas/search_space_zoo/nasbench201.py b/examples/nas/search_space_zoo/nasbench201.py new file mode 100644 index 0000000000..e6bf497935 --- /dev/null +++ b/examples/nas/search_space_zoo/nasbench201.py @@ -0,0 +1,206 @@ +import argparse +import json +import logging +import os +import pprint + +import numpy as np +import pandas as pd +import torch +import torch.nn as nn +import torch.optim as optim +from torch.utils.data import DataLoader +from nni.nas.pytorch import enas +from nni.nas.pytorch.utils import AverageMeterGroup +from nni.nas.pytorch.nas_bench_201 import NASBench201Cell +from nni.nas.pytorch.fixed import apply_fixed_architecture +from nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats +from nni.nas.pytorch.callbacks import ArchitectureCheckpoint, LRSchedulerCallback +from nni.nas.pytorch.darts import DartsTrainer +from utils import accuracy, reward_accuracy + +import datasets + +logger = logging.getLogger('nni') + +class ReLUConvBN(nn.Module): + def __init__(self, C_in, C_out, kernel_size, stride, padding, dilation, + bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): + super(ReLUConvBN, self).__init__() + self.op = nn.Sequential( + nn.ReLU(inplace=False), + nn.Conv2d(C_in, C_out, kernel_size, stride=stride, + padding=padding, dilation=dilation, bias=False), + nn.BatchNorm2d(C_out, affine=bn_affine, momentum=bn_momentum, + track_running_stats=bn_track_running_stats) + ) + + def forward(self, x): + return self.op(x) + + +class ResNetBasicBlock(nn.Module): + def __init__(self, inplanes, planes, stride, bn_affine=True, + bn_momentum=0.1, bn_track_running_stats=True): + super(ResNetBasicBlock, self).__init__() + assert stride == 1 or stride == 2, "invalid stride {:}".format(stride) + self.conv_a = ReLUConvBN(inplanes, planes, 3, stride, 1, 1, bn_affine, bn_momentum, bn_track_running_stats) + self.conv_b = ReLUConvBN(planes, planes, 3, 1, 1, 1, bn_affine, bn_momentum, bn_track_running_stats) + if stride == 2: + self.downsample = nn.Sequential( + nn.AvgPool2d(kernel_size=2, stride=2, padding=0), + nn.Conv2d(inplanes, planes, kernel_size=1, stride=1, padding=0, bias=False)) + elif inplanes != planes: + self.downsample = ReLUConvBN(inplanes, planes, 1, 1, 0, 1, bn_affine, bn_momentum, bn_track_running_stats) + else: + self.downsample = None + self.in_dim = inplanes + self.out_dim = planes + self.stride = stride + self.num_conv = 2 + + def forward(self, inputs): + basicblock = self.conv_a(inputs) + basicblock = self.conv_b(basicblock) + + if self.downsample is not None: + inputs = self.downsample(inputs) + return inputs + basicblock + + +class NASBench201Network(nn.Module): + def __init__(self, stem_out_channels, num_modules_per_stack, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): + super(NASBench201Network, self).__init__() + self.channels = C = stem_out_channels + self.num_modules = N = num_modules_per_stack + self.num_labels = 10 + + self.bn_momentum = bn_momentum + self.bn_affine = bn_affine + self.bn_track_running_stats = bn_track_running_stats + + self.stem = nn.Sequential( + nn.Conv2d(3, C, kernel_size=3, padding=1, bias=False), + nn.BatchNorm2d(C, momentum=self.bn_momentum) + ) + + layer_channels = [C] * N + [C * 2] + [C * 2] * N + [C * 4] + [C * 4] * N + layer_reductions = [False] * N + [True] + [False] * N + [True] + [False] * N + + C_prev = C + self.cells = nn.ModuleList() + for i, (C_curr, reduction) in enumerate(zip(layer_channels, layer_reductions)): + if reduction: + cell = ResNetBasicBlock(C_prev, C_curr, 2, self.bn_affine, self.bn_momentum, self.bn_track_running_stats) + else: + cell = NASBench201Cell(i, C_prev, C_curr, 1, self.bn_affine, self.bn_momentum, self.bn_track_running_stats) + self.cells.append(cell) + C_prev = C_curr + + self.lastact = nn.Sequential( + nn.BatchNorm2d(C_prev, momentum=self.bn_momentum), + nn.ReLU(inplace=True) + ) + self.global_pooling = nn.AdaptiveAvgPool2d(1) + self.classifier = nn.Linear(C_prev, self.num_labels) + + def forward(self, inputs): + feature = self.stem(inputs) + for cell in self.cells: + feature = cell(feature) + + out = self.lastact(feature) + out = self.global_pooling(out) + out = out.view(out.size(0), -1) + logits = self.classifier(out) + + return logits + + +def train(args, model, train_dataloader, valid_dataloader, criterion, optimizer, device): + model = model.to(device) + model.train() + for epoch in range(args.epochs): + for batch_idx, (data, target) in enumerate(train_dataloader): + data, target = data.to(device), target.to(device) + optimizer.zero_grad() + output = model(data) + loss = criterion(output, target) + loss.backward() + optimizer.step() + + if batch_idx % args.log_frequency == 0: + logger.info('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format( + epoch, batch_idx * len(data), len(train_dataloader.dataset), + 100. * batch_idx / len(train_dataloader), loss.item())) + model.eval() + correct = 0 + test_loss = 0.0 + for data, target in valid_dataloader: + data, target = data.to(device), target.to(device) + output = model(data) + test_loss += criterion(output, target, reduction='sum').item() + pred = output.argmax(dim=1, keepdim=True) + correct += pred.eq(target.view_as(pred)).sum().item() + test_loss /= len(valid_dataloader.dataset) + accuracy = 100. * correct / len(valid_dataloader.dataset) + logger.info('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(test_loss, correct, + len(valid_dataloader.dataset), accuracy)) + model.train() + + +if __name__ == '__main__': + parser = argparse.ArgumentParser("nb201") + parser.add_argument('--stem_out_channels', default=16, type=int) + parser.add_argument('--unrolled', default=False, action='store_true') + parser.add_argument('--batch_size', default=64, type=int) + parser.add_argument('--epochs', default=50, type=int) + parser.add_argument('--num_modules_per_stack', default=5, type=int) + parser.add_argument('--log-frequency', default=10, type=int) + parser.add_argument('--bn_momentum', default=0.1, type=int) + parser.add_argument('--bn_affine', default=True, type=bool) + parser.add_argument('--bn_track_running_stats', default=True, type=bool) + parser.add_argument('--arch', default=None, help='json file which should meet requirements in NAS-Bench-201') + parser.add_argument('--visualization', default=False, action='store_true') + args = parser.parse_args() + + dataset_train, dataset_valid = datasets.get_dataset("cifar10") + model = NASBench201Network(stem_out_channels=args.stem_out_channels, + num_modules_per_stack=args.num_modules_per_stack, + bn_affine=args.bn_affine, + bn_momentum=args.bn_momentum, + bn_track_running_stats=args.bn_track_running_stats) + + optim = torch.optim.SGD(model.parameters(), 0.025) + lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, args.epochs, eta_min=0.001) + criterion = nn.CrossEntropyLoss() + + if args.arch is not None: + with open(args.arch, 'r') as f: + arch = json.load(f) + for trial in query_nb201_trial_stats(arch, 200, 'cifar100'): + pprint.pprint(trial) + apply_fixed_architecture(model, args.arch) + dataloader_train = DataLoader(dataset_train, batch_size=args.batch_size, shuffle=True, num_workers=0) + dataloader_valid = DataLoader(dataset_valid, batch_size=args.batch_size, shuffle=True, num_workers=0) + train(args, model, dataloader_train, dataloader_valid, criterion, optim, + torch.device('cuda' if torch.cuda.is_available() else 'cpu')) + exit(0) + + mutator = enas.EnasMutator(model, tanh_constant=1.1, cell_exit_extra_step=True) + trainer = enas.EnasTrainer(model, + loss=criterion, + metrics=lambda output, target: accuracy(output, target, topk=(1,)), + reward_function=reward_accuracy, + optimizer=optim, + callbacks=[LRSchedulerCallback(lr_scheduler), ArchitectureCheckpoint("./checkpoints")], + batch_size=args.batch_size, + num_epochs=args.epochs, + dataset_train=dataset_train, + dataset_valid=dataset_valid, + log_frequency=args.log_frequency, + mutator=mutator) + + if args.visualization: + trainer.enable_visualization() + trainer.train() From b4b34b66fd97b4204b1e49383bbf394aff69ad00 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Thu, 10 Sep 2020 10:01:33 +0800 Subject: [PATCH 35/43] doc refine --- docs/en_US/NAS/SearchSpaceZoo.md | 24 +-- examples/nas/search_space_zoo/nasbench201.py | 3 +- .../nni/nas/pytorch/nasbench201/__init__.py | 1 + .../nas/pytorch/nasbench201/nasbench201.py | 72 +++++++++ .../pytorch/nasbench201/nasbench201_ops.py | 146 ++++++++++++++++++ 5 files changed, 233 insertions(+), 13 deletions(-) create mode 100644 src/sdk/pynni/nni/nas/pytorch/nasbench201/__init__.py create mode 100644 src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py create mode 100644 src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201_ops.py diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index 1e320f477b..e0f46d8e78 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -2,13 +2,13 @@ ## DartsCell -DartsCell is extracted from [CNN model](./DARTS.md) designed [here](https://github.com/microsoft/nni/tree/master/examples/nas/darts). A DartsCell is a directed acyclic graph containing an ordered sequence of N nodes and each node stands for a latent representation (e.g. feature map in a convolutional network). Directed edges from Node 1 to Node 2 are associated with some operations that transform Node 1 and the result is stored on Node 2. The [operations](#darts-predefined-operations) between nodes is predefined and unchangeable. One edge represents an operation that chosen from the predefined ones to be applied to the starting node of the edge. One cell contains two input nodes, a single output node, and other `n_node` nodes. The input nodes are defined as the cell outputs in the previous two layers. The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to all the intermediate nodes. To make the search space continuous, the categorical choice of a particular operation is relaxed to a softmax over all possible operations. By adjusting the weight of softmax on every node, the operation with the highest probability is chosen to be part of the final structure. A CNN model can be formed by stacking several cells together, which builds a search space. Note that, in DARTS paper all cells in the model share the same structure. +DartsCell is extracted from [CNN model](./DARTS.md) designed [here](https://github.com/microsoft/nni/tree/master/examples/nas/darts). A DartsCell is a directed acyclic graph containing an ordered sequence of N nodes and each node stands for a latent representation (e.g. feature map in a convolutional network). Directed edges from Node 1 to Node 2 are associated with some operations that transform Node 1 and the result is stored on Node 2. The [Candidate operations](#predefined-operations-darts) between nodes is predefined and unchangeable. One edge represents an operation that chosen from the predefined ones to be applied to the starting node of the edge. One cell contains two input nodes, a single output node, and other `n_node` nodes. The input nodes are defined as the cell outputs in the previous two layers. The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to all the intermediate nodes. To make the search space continuous, the categorical choice of a particular operation is relaxed to a softmax over all possible operations. By adjusting the weight of softmax on every node, the operation with the highest probability is chosen to be part of the final structure. A CNN model can be formed by stacking several cells together, which builds a search space. Note that, in DARTS paper all cells in the model share the same structure. One structure in the Darts search space is shown below. Note that, NNI merges the last one of the four intermediate nodes and the output node. ![](../../img/NAS_Darts_cell.svg) -The predefined operations are shown in [references](#predefined-operations-darts). +The predefined operations are shown in [Candidate operations](#predefined-operations-darts). ```eval_rst .. autoclass:: nni.nas.pytorch.search_space_zoo.DartsCell @@ -28,7 +28,7 @@ python3 darts_example.py -### References +### Candidate operations All supported operations for Darts are listed below. @@ -91,7 +91,7 @@ python3 enas_micro_example.py -### References +### Candidate operations All supported operations for ENAS micro search are listed below. @@ -116,7 +116,7 @@ All supported operations for ENAS micro search are listed below. ## ENASMacroLayer -In Macro search, the controller makes two decisions for each layer: i) the [operation](#macro-operations) to perform on the result of the previous layer, ii) which the previous layer to connect to for SkipConnects. ENAS uses a controller to design the whole model architecture instead of one of its components. The output of operations is going to concat with the tensor of the chosen layer for SkipConnect. NNI provides [predefined operations](#macro-operations) for macro search, which are listed in [references](#macro-operations). +In Macro search, the controller makes two decisions for each layer: i) the [operation](#macro-operations) to perform on the result of the previous layer, ii) which the previous layer to connect to for SkipConnects. ENAS uses a controller to design the whole model architecture instead of one of its components. The output of operations is going to concat with the tensor of the chosen layer for SkipConnect. NNI provides [predefined operations](#macro-operations) for macro search, which are listed in [Candidate operations](#macro-operations). Part of one structure in the ENAS macro search space is shown below. @@ -147,7 +147,7 @@ python3 enas_macro_example.py -### References +### Candidate operations All supported operations for ENAS macro search are listed below. @@ -174,14 +174,14 @@ All supported operations for ENAS macro search are listed below. ## NAS-Bench-201 -NAS Bench 201 defines a unified search space, which is algorithm agnostic. The predefined skeleton consists of a stack of cells that share the same architecture. Every cell contains four nodes and a DAG is formed by connecting edges among them, where the node represents the sum of feature maps and the edge stands for an operation transforming a tensor from the source node to the target node. The predefined candidate operations can be found in [reference](#nas-bench-201-reference). +NAS Bench 201 defines a unified search space, which is algorithm agnostic. The predefined skeleton consists of a stack of cells that share the same architecture. Every cell contains four nodes and a DAG is formed by connecting edges among them, where the node represents the sum of feature maps and the edge stands for an operation transforming a tensor from the source node to the target node. The predefined candidate operations can be found in [Candidate operations](#nas-bench-201-reference). The search space of NAS Bench 201 is shown below. ![](../../img/NAS_Bench_201.svg) ```eval_rst -.. autoclass:: nni.nas.pytorch.nas_bench_201.NASBench201Cell +.. autoclass:: nni.nas.pytorch.nasbench201.NASBench201Cell :members: ``` @@ -198,7 +198,7 @@ python3 nas_bench_201.py -### Reference +### Candidate operations All supported operations for NAS Bench 201 are listed below. @@ -208,7 +208,7 @@ All supported operations for NAS Bench 201 are listed below. Call `torch.nn.AvgPool2d`. This operation applies a 2D average pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3` and `padding=1`. ```eval_rst - .. autoclass:: nni.nas.pytorch.nas_bench_201.nas_bench_201_ops.Pooling + .. autoclass:: nni.nas.pytorch.nasbench201.nasbench201_ops.Pooling :members: ``` @@ -217,7 +217,7 @@ All supported operations for NAS Bench 201 are listed below. * Conv3x3: Consist of a sequence of ReLU, `nn.Cinv2d` and BatchNorm. The Conv operation's parameter is fixed to `kernal_size=3`, `padding=1`, and `dilation=1`. ```eval_rst - .. autoclass:: nni.nas.pytorch.nas_bench_201.nas_bench_201_ops.ReLUConvBN + .. autoclass:: nni.nas.pytorch.nasbench201.nasbench201_ops.ReLUConvBN :members: ``` @@ -230,7 +230,7 @@ All supported operations for NAS Bench 201 are listed below. Generate zero tensors indicating there is no connection from the source node to the target node. ```eval_rst - .. autoclass:: nni.nas.pytorch.nas_bench_201.nas_bench_201_ops.Zero + .. autoclass:: nni.nas.pytorch.nasbench201.nasbench201_ops.Zero :members: ``` diff --git a/examples/nas/search_space_zoo/nasbench201.py b/examples/nas/search_space_zoo/nasbench201.py index e6bf497935..2b57b1d051 100644 --- a/examples/nas/search_space_zoo/nasbench201.py +++ b/examples/nas/search_space_zoo/nasbench201.py @@ -175,7 +175,8 @@ def train(args, model, train_dataloader, valid_dataloader, criterion, optimizer, lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, args.epochs, eta_min=0.001) criterion = nn.CrossEntropyLoss() - if args.arch is not None: + if args.arch is not None: + logger.info('model retraining...') with open(args.arch, 'r') as f: arch = json.load(f) for trial in query_nb201_trial_stats(arch, 200, 'cifar100'): diff --git a/src/sdk/pynni/nni/nas/pytorch/nasbench201/__init__.py b/src/sdk/pynni/nni/nas/pytorch/nasbench201/__init__.py new file mode 100644 index 0000000000..1419b3fa92 --- /dev/null +++ b/src/sdk/pynni/nni/nas/pytorch/nasbench201/__init__.py @@ -0,0 +1 @@ +from .nasbench201 import NASBench201Cell diff --git a/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py b/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py new file mode 100644 index 0000000000..a7d05e1dc9 --- /dev/null +++ b/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py @@ -0,0 +1,72 @@ +from collections import OrderedDict +import torch.nn as nn +from nni.nas.pytorch.mutables import LayerChoice + +from .nasbench201_ops import Pooling, ReLUConvBN, Zero, FactorizedReduce + + +class NASBench201Cell(nn.Module): + """ + Builtin cell structure of NAS Bench 201. One cell contains four nodes. The First node serves as an input node + accepting the output of the previous cell. And other nodes connect to all previous nodes with an edge that + represents an operation chosen from a set to transform the tensor from the source node to the target node. + Every node accepts all its inputs and adds them as its output. + + Parameters + --- + cell_id: str + the name of this cell + C_in: int + the number of input channels of the cell + C_out: int + the number of output channels of the cell + stride: int + stride of all convolution operations in the cell + bn_affine: bool + If set to ``True``, all ``torch.nn.BatchNorm2d`` in this cell will have learnable affine parameters. Default: True + bn_momentum: float + the value used for the running_mean and running_var computation. Default: 0.1 + bn_track_running_stats: bool + When set to ``True``, all ``torch.nn.BatchNorm2d`` in this cell tracks the running mean and variance. Default: True + """ + + def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): + super(NASBench201Cell, self).__init__() + + self.NUM_NODES = 4 + self.layers = nn.ModuleList() + + OPS = lambda layer_idx: OrderedDict([ + ("none", Zero(C_in, C_out, stride)), + ("avg_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, + bn_track_running_stats)), + ("conv_3x3", ReLUConvBN(C_in, C_out, 3, stride if layer_idx == 0 else 1, 1, 1, bn_affine, bn_momentum, + bn_track_running_stats)), + ("conv_1x1", ReLUConvBN(C_in, C_out, 1, stride if layer_idx == 0 else 1, 0, 1, bn_affine, bn_momentum, + bn_track_running_stats)), + ("skip_connect", nn.Identity() if stride == 1 and C_in == C_out + else FactorizedReduce(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, + bn_track_running_stats)) + ]) + + for i in range(self.NUM_NODES): + node_ops = nn.ModuleList() + for j in range(0, i): + node_ops.append(LayerChoice(OPS(j), key="%d_%d" % (j, i), reduction="mean")) + self.layers.append(node_ops) + self.in_dim = C_in + self.out_dim = C_out + self.cell_id = cell_id + + def forward(self, inputs): + """ + Parameters + --- + inputs: tensor + the output of the previous layer + """ + nodes = [inputs] + for i in range(1, self.NUM_NODES): + node_feature = sum(self.layers[i][k](nodes[k]) for k in range(i)) + nodes.append(node_feature) + return nodes[-1] diff --git a/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201_ops.py b/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201_ops.py new file mode 100644 index 0000000000..aa60f8854f --- /dev/null +++ b/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201_ops.py @@ -0,0 +1,146 @@ +import torch +import torch.nn as nn + + +class ReLUConvBN(nn.Module): + """ + Parameters + --- + C_in: int + the number of input channels + C_out: int + the number of output channels + stride: int + stride of the convolution + padding: int + zero-padding added to both sides of the input + dilation: int + spacing between kernel elements + bn_affine: bool + If set to ``True``, ``torch.nn.BatchNorm2d`` will have learnable affine parameters. Default: True + bn_momentun: float + the value used for the running_mean and running_var computation. Default: 0.1 + bn_track_running_stats: bool + When set to ``True``, ``torch.nn.BatchNorm2d`` tracks the running mean and variance. Default: True + """ + def __init__(self, C_in, C_out, kernel_size, stride, padding, dilation, + bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): + super(ReLUConvBN, self).__init__() + self.op = nn.Sequential( + nn.ReLU(inplace=False), + nn.Conv2d(C_in, C_out, kernel_size, stride=stride, + padding=padding, dilation=dilation, bias=False), + nn.BatchNorm2d(C_out, affine=bn_affine, momentum=bn_momentum, + track_running_stats=bn_track_running_stats) + ) + + def forward(self, x): + """ + Parameters + --- + x: torch.Tensor + input tensor + """ + return self.op(x) + + +class Pooling(nn.Module): + """ + Parameters + --- + C_in: int + the number of input channels + C_out: int + the number of output channels + stride: int + stride of the convolution + bn_affine: bool + If set to ``True``, ``torch.nn.BatchNorm2d`` will have learnable affine parameters. Default: True + bn_momentun: float + the value used for the running_mean and running_var computation. Default: 0.1 + bn_track_running_stats: bool + When set to ``True``, ``torch.nn.BatchNorm2d`` tracks the running mean and variance. Default: True + """ + def __init__(self, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): + super(Pooling, self).__init__() + if C_in == C_out: + self.preprocess = None + else: + self.preprocess = ReLUConvBN(C_in, C_out, 1, 1, 0, 0, + bn_affine, bn_momentum, bn_track_running_stats) + self.op = nn.AvgPool2d(3, stride=stride, padding=1, count_include_pad=False) + + def forward(self, x): + """ + Parameters + --- + x: torch.Tensor + input tensor + """ + if self.preprocess: + x = self.preprocess(x) + return self.op(x) + + +class Zero(nn.Module): + """ + Parameters + --- + C_in: int + the number of input channels + C_out: int + the number of output channels + stride: int + stride of the convolution + """ + def __init__(self, C_in, C_out, stride): + super(Zero, self).__init__() + self.C_in = C_in + self.C_out = C_out + self.stride = stride + self.is_zero = True + + def forward(self, x): + """ + Parameters + --- + x: torch.Tensor + input tensor + """ + if self.C_in == self.C_out: + if self.stride == 1: + return x.mul(0.) + else: + return x[:, :, ::self.stride, ::self.stride].mul(0.) + else: + shape = list(x.shape) + shape[1] = self.C_out + zeros = x.new_zeros(shape, dtype=x.dtype, device=x.device) + return zeros + + +class FactorizedReduce(nn.Module): + def __init__(self, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, + bn_track_running_stats=True): + super(FactorizedReduce, self).__init__() + self.stride = stride + self.C_in = C_in + self.C_out = C_out + self.relu = nn.ReLU(inplace=False) + if stride == 2: + C_outs = [C_out // 2, C_out - C_out // 2] + self.convs = nn.ModuleList() + for i in range(2): + self.convs.append(nn.Conv2d(C_in, C_outs[i], 1, stride=stride, padding=0, bias=False)) + self.pad = nn.ConstantPad2d((0, 1, 0, 1), 0) + else: + raise ValueError("Invalid stride : {:}".format(stride)) + self.bn = nn.BatchNorm2d(C_out, affine=bn_affine, momentum=bn_momentum, + track_running_stats=bn_track_running_stats) + + def forward(self, x): + x = self.relu(x) + y = self.pad(x) + out = torch.cat([self.convs[0](x), self.convs[1](y[:, :, 1:, 1:])], dim=1) + out = self.bn(out) + return out From b696eb2b00f572880c7afcdca71300a1574f1625 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Thu, 10 Sep 2020 10:25:40 +0800 Subject: [PATCH 36/43] remove nas_bench_201 --- .../nni/nas/pytorch/nas_bench_201/__init__.py | 1 - .../pytorch/nas_bench_201/nas_bench_201.py | 72 --------- .../nas_bench_201/nas_bench_201_ops.py | 146 ------------------ 3 files changed, 219 deletions(-) delete mode 100644 src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py delete mode 100644 src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py delete mode 100644 src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py deleted file mode 100644 index 0a6d327e57..0000000000 --- a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/__init__.py +++ /dev/null @@ -1 +0,0 @@ -from .nas_bench_201 import NASBench201Cell diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py deleted file mode 100644 index e71ba0c46a..0000000000 --- a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201.py +++ /dev/null @@ -1,72 +0,0 @@ -from collections import OrderedDict -import torch.nn as nn -from nni.nas.pytorch.mutables import LayerChoice - -from .nas_bench_201_ops import Pooling, ReLUConvBN, Zero, FactorizedReduce - - -class NASBench201Cell(nn.Module): - """ - Builtin cell structure of NAS Bench 201. One cell contains four nodes. The First node serves as an input node - accepting the output of the previous cell. And other nodes connect to all previous nodes with an edge that - represents an operation chosen from a set to transform the tensor from the source node to the target node. - Every node accepts all its inputs and adds them as its output. - - Parameters - --- - cell_id: str - the name of this cell - C_in: int - the number of input channels - C_out: int - the number of output channels - stride: int - stride of the convolution - bn_affine: bool - If set to ``True``, ``torch.nn.BatchNorm2d`` will have learnable affine parameters. Default: True - bn_momentum: float - the value used for the running_mean and running_var computation. Default: 0.1 - bn_track_running_stats: bool - When set to ``True``, ``torch.nn.BatchNorm2d`` tracks the running mean and variance. Default: True - """ - - def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): - super(NASBench201Cell, self).__init__() - - self.NUM_NODES = 4 - self.layers = nn.ModuleList() - - OPS = lambda layer_idx: OrderedDict([ - ("none", Zero(C_in, C_out, stride)), - ("avg_pool_3x3", Pooling(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, - bn_track_running_stats)), - ("conv_3x3", ReLUConvBN(C_in, C_out, 3, stride if layer_idx == 0 else 1, 1, 1, bn_affine, bn_momentum, - bn_track_running_stats)), - ("conv_1x1", ReLUConvBN(C_in, C_out, 1, stride if layer_idx == 0 else 1, 0, 1, bn_affine, bn_momentum, - bn_track_running_stats)), - ("skip_connect", nn.Identity() if stride == 1 and C_in == C_out - else FactorizedReduce(C_in, C_out, stride if layer_idx == 0 else 1, bn_affine, bn_momentum, - bn_track_running_stats)) - ]) - - for i in range(self.NUM_NODES): - node_ops = nn.ModuleList() - for j in range(0, i): - node_ops.append(LayerChoice(OPS(j), key="%d_%d" % (j, i), reduction="mean")) - self.layers.append(node_ops) - self.in_dim = C_in - self.out_dim = C_out - self.cell_id = cell_id - - def forward(self, inputs): - """ - Parameters - --- - inputs: tensor - the output of the previous layer - """ - nodes = [inputs] - for i in range(1, self.NUM_NODES): - node_feature = sum(self.layers[i][k](nodes[k]) for k in range(i)) - nodes.append(node_feature) - return nodes[-1] diff --git a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py b/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py deleted file mode 100644 index aa60f8854f..0000000000 --- a/src/sdk/pynni/nni/nas/pytorch/nas_bench_201/nas_bench_201_ops.py +++ /dev/null @@ -1,146 +0,0 @@ -import torch -import torch.nn as nn - - -class ReLUConvBN(nn.Module): - """ - Parameters - --- - C_in: int - the number of input channels - C_out: int - the number of output channels - stride: int - stride of the convolution - padding: int - zero-padding added to both sides of the input - dilation: int - spacing between kernel elements - bn_affine: bool - If set to ``True``, ``torch.nn.BatchNorm2d`` will have learnable affine parameters. Default: True - bn_momentun: float - the value used for the running_mean and running_var computation. Default: 0.1 - bn_track_running_stats: bool - When set to ``True``, ``torch.nn.BatchNorm2d`` tracks the running mean and variance. Default: True - """ - def __init__(self, C_in, C_out, kernel_size, stride, padding, dilation, - bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): - super(ReLUConvBN, self).__init__() - self.op = nn.Sequential( - nn.ReLU(inplace=False), - nn.Conv2d(C_in, C_out, kernel_size, stride=stride, - padding=padding, dilation=dilation, bias=False), - nn.BatchNorm2d(C_out, affine=bn_affine, momentum=bn_momentum, - track_running_stats=bn_track_running_stats) - ) - - def forward(self, x): - """ - Parameters - --- - x: torch.Tensor - input tensor - """ - return self.op(x) - - -class Pooling(nn.Module): - """ - Parameters - --- - C_in: int - the number of input channels - C_out: int - the number of output channels - stride: int - stride of the convolution - bn_affine: bool - If set to ``True``, ``torch.nn.BatchNorm2d`` will have learnable affine parameters. Default: True - bn_momentun: float - the value used for the running_mean and running_var computation. Default: 0.1 - bn_track_running_stats: bool - When set to ``True``, ``torch.nn.BatchNorm2d`` tracks the running mean and variance. Default: True - """ - def __init__(self, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): - super(Pooling, self).__init__() - if C_in == C_out: - self.preprocess = None - else: - self.preprocess = ReLUConvBN(C_in, C_out, 1, 1, 0, 0, - bn_affine, bn_momentum, bn_track_running_stats) - self.op = nn.AvgPool2d(3, stride=stride, padding=1, count_include_pad=False) - - def forward(self, x): - """ - Parameters - --- - x: torch.Tensor - input tensor - """ - if self.preprocess: - x = self.preprocess(x) - return self.op(x) - - -class Zero(nn.Module): - """ - Parameters - --- - C_in: int - the number of input channels - C_out: int - the number of output channels - stride: int - stride of the convolution - """ - def __init__(self, C_in, C_out, stride): - super(Zero, self).__init__() - self.C_in = C_in - self.C_out = C_out - self.stride = stride - self.is_zero = True - - def forward(self, x): - """ - Parameters - --- - x: torch.Tensor - input tensor - """ - if self.C_in == self.C_out: - if self.stride == 1: - return x.mul(0.) - else: - return x[:, :, ::self.stride, ::self.stride].mul(0.) - else: - shape = list(x.shape) - shape[1] = self.C_out - zeros = x.new_zeros(shape, dtype=x.dtype, device=x.device) - return zeros - - -class FactorizedReduce(nn.Module): - def __init__(self, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1, - bn_track_running_stats=True): - super(FactorizedReduce, self).__init__() - self.stride = stride - self.C_in = C_in - self.C_out = C_out - self.relu = nn.ReLU(inplace=False) - if stride == 2: - C_outs = [C_out // 2, C_out - C_out // 2] - self.convs = nn.ModuleList() - for i in range(2): - self.convs.append(nn.Conv2d(C_in, C_outs[i], 1, stride=stride, padding=0, bias=False)) - self.pad = nn.ConstantPad2d((0, 1, 0, 1), 0) - else: - raise ValueError("Invalid stride : {:}".format(stride)) - self.bn = nn.BatchNorm2d(C_out, affine=bn_affine, momentum=bn_momentum, - track_running_stats=bn_track_running_stats) - - def forward(self, x): - x = self.relu(x) - y = self.pad(x) - out = torch.cat([self.convs[0](x), self.convs[1](y[:, :, 1:, 1:])], dim=1) - out = self.bn(out) - return out From 69b91f11f34da9b6209899cf256fc6233016d928 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Thu, 10 Sep 2020 10:38:36 +0800 Subject: [PATCH 37/43] Update nas_bench_201.py --- examples/nas/search_space_zoo/nas_bench_201.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/nas/search_space_zoo/nas_bench_201.py b/examples/nas/search_space_zoo/nas_bench_201.py index e6bf497935..4924269ce2 100644 --- a/examples/nas/search_space_zoo/nas_bench_201.py +++ b/examples/nas/search_space_zoo/nas_bench_201.py @@ -12,7 +12,7 @@ from torch.utils.data import DataLoader from nni.nas.pytorch import enas from nni.nas.pytorch.utils import AverageMeterGroup -from nni.nas.pytorch.nas_bench_201 import NASBench201Cell +from nni.nas.pytorch.nasbench201 import NASBench201Cell from nni.nas.pytorch.fixed import apply_fixed_architecture from nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats from nni.nas.pytorch.callbacks import ArchitectureCheckpoint, LRSchedulerCallback From 580db7399930e99a65a2a273653e5f6e465e86e7 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Thu, 10 Sep 2020 10:44:01 +0800 Subject: [PATCH 38/43] fix --- .../nas/search_space_zoo/nas_bench_201.py | 206 ------------------ examples/nas/search_space_zoo/nasbench201.py | 2 +- 2 files changed, 1 insertion(+), 207 deletions(-) delete mode 100644 examples/nas/search_space_zoo/nas_bench_201.py diff --git a/examples/nas/search_space_zoo/nas_bench_201.py b/examples/nas/search_space_zoo/nas_bench_201.py deleted file mode 100644 index 4924269ce2..0000000000 --- a/examples/nas/search_space_zoo/nas_bench_201.py +++ /dev/null @@ -1,206 +0,0 @@ -import argparse -import json -import logging -import os -import pprint - -import numpy as np -import pandas as pd -import torch -import torch.nn as nn -import torch.optim as optim -from torch.utils.data import DataLoader -from nni.nas.pytorch import enas -from nni.nas.pytorch.utils import AverageMeterGroup -from nni.nas.pytorch.nasbench201 import NASBench201Cell -from nni.nas.pytorch.fixed import apply_fixed_architecture -from nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats -from nni.nas.pytorch.callbacks import ArchitectureCheckpoint, LRSchedulerCallback -from nni.nas.pytorch.darts import DartsTrainer -from utils import accuracy, reward_accuracy - -import datasets - -logger = logging.getLogger('nni') - -class ReLUConvBN(nn.Module): - def __init__(self, C_in, C_out, kernel_size, stride, padding, dilation, - bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): - super(ReLUConvBN, self).__init__() - self.op = nn.Sequential( - nn.ReLU(inplace=False), - nn.Conv2d(C_in, C_out, kernel_size, stride=stride, - padding=padding, dilation=dilation, bias=False), - nn.BatchNorm2d(C_out, affine=bn_affine, momentum=bn_momentum, - track_running_stats=bn_track_running_stats) - ) - - def forward(self, x): - return self.op(x) - - -class ResNetBasicBlock(nn.Module): - def __init__(self, inplanes, planes, stride, bn_affine=True, - bn_momentum=0.1, bn_track_running_stats=True): - super(ResNetBasicBlock, self).__init__() - assert stride == 1 or stride == 2, "invalid stride {:}".format(stride) - self.conv_a = ReLUConvBN(inplanes, planes, 3, stride, 1, 1, bn_affine, bn_momentum, bn_track_running_stats) - self.conv_b = ReLUConvBN(planes, planes, 3, 1, 1, 1, bn_affine, bn_momentum, bn_track_running_stats) - if stride == 2: - self.downsample = nn.Sequential( - nn.AvgPool2d(kernel_size=2, stride=2, padding=0), - nn.Conv2d(inplanes, planes, kernel_size=1, stride=1, padding=0, bias=False)) - elif inplanes != planes: - self.downsample = ReLUConvBN(inplanes, planes, 1, 1, 0, 1, bn_affine, bn_momentum, bn_track_running_stats) - else: - self.downsample = None - self.in_dim = inplanes - self.out_dim = planes - self.stride = stride - self.num_conv = 2 - - def forward(self, inputs): - basicblock = self.conv_a(inputs) - basicblock = self.conv_b(basicblock) - - if self.downsample is not None: - inputs = self.downsample(inputs) - return inputs + basicblock - - -class NASBench201Network(nn.Module): - def __init__(self, stem_out_channels, num_modules_per_stack, bn_affine=True, bn_momentum=0.1, bn_track_running_stats=True): - super(NASBench201Network, self).__init__() - self.channels = C = stem_out_channels - self.num_modules = N = num_modules_per_stack - self.num_labels = 10 - - self.bn_momentum = bn_momentum - self.bn_affine = bn_affine - self.bn_track_running_stats = bn_track_running_stats - - self.stem = nn.Sequential( - nn.Conv2d(3, C, kernel_size=3, padding=1, bias=False), - nn.BatchNorm2d(C, momentum=self.bn_momentum) - ) - - layer_channels = [C] * N + [C * 2] + [C * 2] * N + [C * 4] + [C * 4] * N - layer_reductions = [False] * N + [True] + [False] * N + [True] + [False] * N - - C_prev = C - self.cells = nn.ModuleList() - for i, (C_curr, reduction) in enumerate(zip(layer_channels, layer_reductions)): - if reduction: - cell = ResNetBasicBlock(C_prev, C_curr, 2, self.bn_affine, self.bn_momentum, self.bn_track_running_stats) - else: - cell = NASBench201Cell(i, C_prev, C_curr, 1, self.bn_affine, self.bn_momentum, self.bn_track_running_stats) - self.cells.append(cell) - C_prev = C_curr - - self.lastact = nn.Sequential( - nn.BatchNorm2d(C_prev, momentum=self.bn_momentum), - nn.ReLU(inplace=True) - ) - self.global_pooling = nn.AdaptiveAvgPool2d(1) - self.classifier = nn.Linear(C_prev, self.num_labels) - - def forward(self, inputs): - feature = self.stem(inputs) - for cell in self.cells: - feature = cell(feature) - - out = self.lastact(feature) - out = self.global_pooling(out) - out = out.view(out.size(0), -1) - logits = self.classifier(out) - - return logits - - -def train(args, model, train_dataloader, valid_dataloader, criterion, optimizer, device): - model = model.to(device) - model.train() - for epoch in range(args.epochs): - for batch_idx, (data, target) in enumerate(train_dataloader): - data, target = data.to(device), target.to(device) - optimizer.zero_grad() - output = model(data) - loss = criterion(output, target) - loss.backward() - optimizer.step() - - if batch_idx % args.log_frequency == 0: - logger.info('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format( - epoch, batch_idx * len(data), len(train_dataloader.dataset), - 100. * batch_idx / len(train_dataloader), loss.item())) - model.eval() - correct = 0 - test_loss = 0.0 - for data, target in valid_dataloader: - data, target = data.to(device), target.to(device) - output = model(data) - test_loss += criterion(output, target, reduction='sum').item() - pred = output.argmax(dim=1, keepdim=True) - correct += pred.eq(target.view_as(pred)).sum().item() - test_loss /= len(valid_dataloader.dataset) - accuracy = 100. * correct / len(valid_dataloader.dataset) - logger.info('\nTest set: Average loss: {:.4f}, Accuracy: {}/{} ({:.0f}%)\n'.format(test_loss, correct, - len(valid_dataloader.dataset), accuracy)) - model.train() - - -if __name__ == '__main__': - parser = argparse.ArgumentParser("nb201") - parser.add_argument('--stem_out_channels', default=16, type=int) - parser.add_argument('--unrolled', default=False, action='store_true') - parser.add_argument('--batch_size', default=64, type=int) - parser.add_argument('--epochs', default=50, type=int) - parser.add_argument('--num_modules_per_stack', default=5, type=int) - parser.add_argument('--log-frequency', default=10, type=int) - parser.add_argument('--bn_momentum', default=0.1, type=int) - parser.add_argument('--bn_affine', default=True, type=bool) - parser.add_argument('--bn_track_running_stats', default=True, type=bool) - parser.add_argument('--arch', default=None, help='json file which should meet requirements in NAS-Bench-201') - parser.add_argument('--visualization', default=False, action='store_true') - args = parser.parse_args() - - dataset_train, dataset_valid = datasets.get_dataset("cifar10") - model = NASBench201Network(stem_out_channels=args.stem_out_channels, - num_modules_per_stack=args.num_modules_per_stack, - bn_affine=args.bn_affine, - bn_momentum=args.bn_momentum, - bn_track_running_stats=args.bn_track_running_stats) - - optim = torch.optim.SGD(model.parameters(), 0.025) - lr_scheduler = torch.optim.lr_scheduler.CosineAnnealingLR(optim, args.epochs, eta_min=0.001) - criterion = nn.CrossEntropyLoss() - - if args.arch is not None: - with open(args.arch, 'r') as f: - arch = json.load(f) - for trial in query_nb201_trial_stats(arch, 200, 'cifar100'): - pprint.pprint(trial) - apply_fixed_architecture(model, args.arch) - dataloader_train = DataLoader(dataset_train, batch_size=args.batch_size, shuffle=True, num_workers=0) - dataloader_valid = DataLoader(dataset_valid, batch_size=args.batch_size, shuffle=True, num_workers=0) - train(args, model, dataloader_train, dataloader_valid, criterion, optim, - torch.device('cuda' if torch.cuda.is_available() else 'cpu')) - exit(0) - - mutator = enas.EnasMutator(model, tanh_constant=1.1, cell_exit_extra_step=True) - trainer = enas.EnasTrainer(model, - loss=criterion, - metrics=lambda output, target: accuracy(output, target, topk=(1,)), - reward_function=reward_accuracy, - optimizer=optim, - callbacks=[LRSchedulerCallback(lr_scheduler), ArchitectureCheckpoint("./checkpoints")], - batch_size=args.batch_size, - num_epochs=args.epochs, - dataset_train=dataset_train, - dataset_valid=dataset_valid, - log_frequency=args.log_frequency, - mutator=mutator) - - if args.visualization: - trainer.enable_visualization() - trainer.train() diff --git a/examples/nas/search_space_zoo/nasbench201.py b/examples/nas/search_space_zoo/nasbench201.py index 2b57b1d051..57379c9034 100644 --- a/examples/nas/search_space_zoo/nasbench201.py +++ b/examples/nas/search_space_zoo/nasbench201.py @@ -12,7 +12,7 @@ from torch.utils.data import DataLoader from nni.nas.pytorch import enas from nni.nas.pytorch.utils import AverageMeterGroup -from nni.nas.pytorch.nas_bench_201 import NASBench201Cell +from nni.nas.pytorch.nasbench201 import NASBench201Cell from nni.nas.pytorch.fixed import apply_fixed_architecture from nni.nas.benchmarks.nasbench201 import query_nb201_trial_stats from nni.nas.pytorch.callbacks import ArchitectureCheckpoint, LRSchedulerCallback From 974ec15d0164bd027eb3b2cc86fa148de4f1acf8 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Thu, 10 Sep 2020 13:59:01 +0800 Subject: [PATCH 39/43] change inputs to input --- src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py b/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py index a7d05e1dc9..598cdab555 100644 --- a/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py +++ b/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py @@ -58,14 +58,14 @@ def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1 self.out_dim = C_out self.cell_id = cell_id - def forward(self, inputs): + def forward(self, input): """ Parameters --- - inputs: tensor + input: torch.tensor the output of the previous layer """ - nodes = [inputs] + nodes = [input] for i in range(1, self.NUM_NODES): node_feature = sum(self.layers[i][k](nodes[k]) for k in range(i)) nodes.append(node_feature) From 335924b68f9c4a23226b58bc434c492c762a8683 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Thu, 10 Sep 2020 15:09:06 +0800 Subject: [PATCH 40/43] fix ut --- src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py b/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py index 598cdab555..cd42fa1a2f 100644 --- a/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py +++ b/src/sdk/pynni/nni/nas/pytorch/nasbench201/nasbench201.py @@ -58,7 +58,7 @@ def __init__(self, cell_id, C_in, C_out, stride, bn_affine=True, bn_momentum=0.1 self.out_dim = C_out self.cell_id = cell_id - def forward(self, input): + def forward(self, input): # pylint: disable=W0622 """ Parameters --- From 1c4d08341bb2fe69e2797c1da27c531100980de5 Mon Sep 17 00:00:00 2001 From: tabVersion Date: Thu, 10 Sep 2020 17:39:47 +0800 Subject: [PATCH 41/43] change sig --- examples/nas/search_space_zoo/nasbench201.py | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/examples/nas/search_space_zoo/nasbench201.py b/examples/nas/search_space_zoo/nasbench201.py index 57379c9034..8bb2a48c7c 100644 --- a/examples/nas/search_space_zoo/nasbench201.py +++ b/examples/nas/search_space_zoo/nasbench201.py @@ -139,7 +139,7 @@ def train(args, model, train_dataloader, valid_dataloader, criterion, optimizer, for data, target in valid_dataloader: data, target = data.to(device), target.to(device) output = model(data) - test_loss += criterion(output, target, reduction='sum').item() + test_loss += criterion(output, target).item() pred = output.argmax(dim=1, keepdim=True) correct += pred.eq(target.view_as(pred)).sum().item() test_loss /= len(valid_dataloader.dataset) From d20fca833dd58357afc8661ef35e6b8960e37974 Mon Sep 17 00:00:00 2001 From: Tab Zhang Date: Fri, 11 Sep 2020 20:51:41 +0800 Subject: [PATCH 42/43] change operations to operators --- docs/en_US/NAS/SearchSpaceZoo.md | 26 +++++++++++++------------- 1 file changed, 13 insertions(+), 13 deletions(-) diff --git a/docs/en_US/NAS/SearchSpaceZoo.md b/docs/en_US/NAS/SearchSpaceZoo.md index e0f46d8e78..36a519d0a1 100644 --- a/docs/en_US/NAS/SearchSpaceZoo.md +++ b/docs/en_US/NAS/SearchSpaceZoo.md @@ -2,13 +2,13 @@ ## DartsCell -DartsCell is extracted from [CNN model](./DARTS.md) designed [here](https://github.com/microsoft/nni/tree/master/examples/nas/darts). A DartsCell is a directed acyclic graph containing an ordered sequence of N nodes and each node stands for a latent representation (e.g. feature map in a convolutional network). Directed edges from Node 1 to Node 2 are associated with some operations that transform Node 1 and the result is stored on Node 2. The [Candidate operations](#predefined-operations-darts) between nodes is predefined and unchangeable. One edge represents an operation that chosen from the predefined ones to be applied to the starting node of the edge. One cell contains two input nodes, a single output node, and other `n_node` nodes. The input nodes are defined as the cell outputs in the previous two layers. The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to all the intermediate nodes. To make the search space continuous, the categorical choice of a particular operation is relaxed to a softmax over all possible operations. By adjusting the weight of softmax on every node, the operation with the highest probability is chosen to be part of the final structure. A CNN model can be formed by stacking several cells together, which builds a search space. Note that, in DARTS paper all cells in the model share the same structure. +DartsCell is extracted from [CNN model](./DARTS.md) designed [here](https://github.com/microsoft/nni/tree/master/examples/nas/darts). A DartsCell is a directed acyclic graph containing an ordered sequence of N nodes and each node stands for a latent representation (e.g. feature map in a convolutional network). Directed edges from Node 1 to Node 2 are associated with some operations that transform Node 1 and the result is stored on Node 2. The [Candidate operators](#predefined-operations-darts) between nodes is predefined and unchangeable. One edge represents an operation that chosen from the predefined ones to be applied to the starting node of the edge. One cell contains two input nodes, a single output node, and other `n_node` nodes. The input nodes are defined as the cell outputs in the previous two layers. The output of the cell is obtained by applying a reduction operation (e.g. concatenation) to all the intermediate nodes. To make the search space continuous, the categorical choice of a particular operation is relaxed to a softmax over all possible operations. By adjusting the weight of softmax on every node, the operation with the highest probability is chosen to be part of the final structure. A CNN model can be formed by stacking several cells together, which builds a search space. Note that, in DARTS paper all cells in the model share the same structure. One structure in the Darts search space is shown below. Note that, NNI merges the last one of the four intermediate nodes and the output node. ![](../../img/NAS_Darts_cell.svg) -The predefined operations are shown in [Candidate operations](#predefined-operations-darts). +The predefined operators are shown [here](#predefined-operations-darts). ```eval_rst .. autoclass:: nni.nas.pytorch.search_space_zoo.DartsCell @@ -28,9 +28,9 @@ python3 darts_example.py -### Candidate operations +### Candidate operators -All supported operations for Darts are listed below. +All supported operators for Darts are listed below. * MaxPool / AvgPool * MaxPool: Call `torch.nn.MaxPool2d`. This operation applies a 2D max pooling over all input channels. Its parameters `kernel_size=3` and `padding=1` are fixed. The pooling result will pass through a BatchNorm2d then return as the result. @@ -69,7 +69,7 @@ The ENAS micro search space is shown below. ![](../../img/NAS_ENAS_micro.svg) -The predefined operations can be seen [here](#predefined-operations-enas). +The predefined operators can be seen [here](#predefined-operations-enas). ```eval_rst .. autoclass:: nni.nas.pytorch.search_space_zoo.ENASMicroLayer @@ -91,9 +91,9 @@ python3 enas_micro_example.py -### Candidate operations +### Candidate operators -All supported operations for ENAS micro search are listed below. +All supported operators for ENAS micro search are listed below. * MaxPool / AvgPool * MaxPool: Call `torch.nn.MaxPool2d`. This operation applies a 2D max pooling over all input channels followed by BatchNorm2d. Its parameters are fixed to `kernel_size=3`, `stride=1` and `padding=1`. @@ -116,7 +116,7 @@ All supported operations for ENAS micro search are listed below. ## ENASMacroLayer -In Macro search, the controller makes two decisions for each layer: i) the [operation](#macro-operations) to perform on the result of the previous layer, ii) which the previous layer to connect to for SkipConnects. ENAS uses a controller to design the whole model architecture instead of one of its components. The output of operations is going to concat with the tensor of the chosen layer for SkipConnect. NNI provides [predefined operations](#macro-operations) for macro search, which are listed in [Candidate operations](#macro-operations). +In Macro search, the controller makes two decisions for each layer: i) the [operation](#macro-operations) to perform on the result of the previous layer, ii) which the previous layer to connect to for SkipConnects. ENAS uses a controller to design the whole model architecture instead of one of its components. The output of operations is going to concat with the tensor of the chosen layer for SkipConnect. NNI provides [predefined operators](#macro-operations) for macro search, which are listed in [Candidate operators](#macro-operations). Part of one structure in the ENAS macro search space is shown below. @@ -147,9 +147,9 @@ python3 enas_macro_example.py -### Candidate operations +### Candidate operators -All supported operations for ENAS macro search are listed below. +All supported operators for ENAS macro search are listed below. * ConvBranch @@ -174,7 +174,7 @@ All supported operations for ENAS macro search are listed below. ## NAS-Bench-201 -NAS Bench 201 defines a unified search space, which is algorithm agnostic. The predefined skeleton consists of a stack of cells that share the same architecture. Every cell contains four nodes and a DAG is formed by connecting edges among them, where the node represents the sum of feature maps and the edge stands for an operation transforming a tensor from the source node to the target node. The predefined candidate operations can be found in [Candidate operations](#nas-bench-201-reference). +NAS Bench 201 defines a unified search space, which is algorithm agnostic. The predefined skeleton consists of a stack of cells that share the same architecture. Every cell contains four nodes and a DAG is formed by connecting edges among them, where the node represents the sum of feature maps and the edge stands for an operation transforming a tensor from the source node to the target node. The predefined candidate operators can be found in [Candidate operators](#nas-bench-201-reference). The search space of NAS Bench 201 is shown below. @@ -198,9 +198,9 @@ python3 nas_bench_201.py -### Candidate operations +### Candidate operators -All supported operations for NAS Bench 201 are listed below. +All supported operators for NAS Bench 201 are listed below. * AvgPool From 7cb0fc98bb41ec43b7c8dbfaced4041823cefcdc Mon Sep 17 00:00:00 2001 From: tabVersion Date: Tue, 22 Sep 2020 13:34:59 +0800 Subject: [PATCH 43/43] remove mutator --- examples/nas/search_space_zoo/nasbench201.py | 4 +--- 1 file changed, 1 insertion(+), 3 deletions(-) diff --git a/examples/nas/search_space_zoo/nasbench201.py b/examples/nas/search_space_zoo/nasbench201.py index 8bb2a48c7c..1b0e03603c 100644 --- a/examples/nas/search_space_zoo/nasbench201.py +++ b/examples/nas/search_space_zoo/nasbench201.py @@ -188,7 +188,6 @@ def train(args, model, train_dataloader, valid_dataloader, criterion, optimizer, torch.device('cuda' if torch.cuda.is_available() else 'cpu')) exit(0) - mutator = enas.EnasMutator(model, tanh_constant=1.1, cell_exit_extra_step=True) trainer = enas.EnasTrainer(model, loss=criterion, metrics=lambda output, target: accuracy(output, target, topk=(1,)), @@ -199,8 +198,7 @@ def train(args, model, train_dataloader, valid_dataloader, criterion, optimizer, num_epochs=args.epochs, dataset_train=dataset_train, dataset_valid=dataset_valid, - log_frequency=args.log_frequency, - mutator=mutator) + log_frequency=args.log_frequency) if args.visualization: trainer.enable_visualization()