Candidate Issues for stabilization sprint

[scarlett2018]

We plan to have a minor sprint for stabilization, the purpose of the sprint is to work on tech debt, bugs, CICD, etc. No new features are expected. By introducing this minor sprint, we give everyone a pause for the next big release, enjoy the minor sprint =)!

https://github.com/microsoft/nni/labels/v2.9.1

Release manager: @Louis-J
Release start date: TBD
Feature freeze date (at most 30 days): 10.14
Code freeze date: 10.17
First test package (1 week since feature freeze): 10.18
Release date (3 weeks since feature freeze): TBD 11.14

Training Service

• p0 - Does not support non-English path ~12/10 Add BOM to support non-English characters in PowerShell scripts #5164 @Lijiaoa , bugbash: complete
• p2 - nni cannot use tensorboard #5064

HPO

• p0 - WebSocket connection may broken during long run ~12/10 Let tuner auto reconnect to NNI manager #5166 @Lijiaoa , bugbash: complete
• p1 - Make anneal optional ~13/10 Make anneal tuner optional #5165 @Bonytu , bugbash: complete
• p0 - lag after long run
• p2 - Assessor.trial_end() is not invoked if no metrics sent
• p2 - Make REST timeout configurable

NAS

• p0 - Dedup for evolution search Dedup for evolution search #5092 @ultmaster reviewing @liuzhe-lz , bugbash: complete
• p0 - Error: Dispatcher stream error, tuner may have crashed. When using PolicyBasedRL to do nas and set concurrency>1 #5117 Fix the racing issue in RL strategy submit model #5135 @ultmaster reviewing @J-shang , bugbash: complete
• p0 - [nas] fix issue introduced by the trial recovery feature [nas] fix issue introduced by the trial recovery feature #5109 ~30/9 @QuanluZhang reviewing @ultmaster, bugbash: complete
• Unclear what extras to install: import nni.retiarii.execution.api fails due to missing pytorch_lightning #5119 @ultmaster ~10/10 reviewing @QuanluZhang , bugbash: complete

Model Compression

• p1 - verify issue #3514, fix if there are any problems @J-shang @Louis-J , bugbash: complete

ModelSpeedup

• p0 - speedup rand_like_with_shape is easy to overflow when dtype=torch.int8. @J-shang ~30/9 waiting for fix(speedup): re-write aten schema parser to support pytorch versions < 1.9.0 #5138 pr: [BugFix] fix compression bugs #5140 Pruning detectron2's Faster R-CNN Pruning detectron2's Faster R-CNN #4875 The pruning process gets stuck and does not run. #5136 @J-shang ~28/9 pr: [BugFix] fix compression bugs #5140 reviewing @QuanluZhang , bugbash: complete
• p2 - Autoinfer can not deal with some op, sparsity cannot propagate through these ops, like torch.nn.functional.interpolate @J-shang move to 3.0
• p0 - torch._C.parse_schema in pytorch < 1.9.0 do not produce enough information. Writing a schema parser needed AttributeError in "Speedup Model with Mask" Tutorial Codes #5131 @Louis-J ~25/9 reviewing @J-shang , bugbash: complete
• p0 - a bug in speedup propagate: executing random_init the view tensors causes error. ERROR: aten::norm is not Supported! #5018 @Louis-J reviewing
  https://pytorch.org/docs/stable/tensor_view.html#tensor-views @J-shang , bugbash: complete
• p1 - ~~a bug in speedup replace_module: EmptyLayerError() or UnBalancedGroupError() during pruning depthwise separable convolution #4991. @Louis-J ~10/10~~
• p2 - refactor the inference process in speedup: in 'speedup support list/dict inputspeedup doesn't support list/dict input', fix(speedup): refactor the execution logic of functions in speedup #5107, solving Support for aten::randn #4998 torch.full can't use implicit scalars #5097 speedup_model() causes a error: “NotImplementedError: There were no tensor arguments to this function (e.g., you passed an empty list of Tensors)” #5130 @Louis-J
• p2 - ~~speedup doesn't support registered buffer. @Louis-J
• p2 - speedup get correct masks in cpu but wrong masks in gpu. Supported for torch.sum #4945 @Louis-J

WebUI

• p0 - HPO experiment hyper-parameters page error. · Issue HPO experiment hyper-parameters page error. #5127 ~28/9 reviewing @liuzhe-lz , bugbash: complete
• p0 - Broken link on WebUI #3549 ~30/9 @Louis-J , bugbash: complete

[ultmaster]

NAS

• Unclear what extras to install: import nni.retiarii.execution.api fails due to missing pytorch_lightning #5119
• Error: Dispatcher stream error, tuner may have crashed. When using PolicyBasedRL to do nas and set concurrency>1 #5117
• Dedup for evolution search #5092

[J-shang]

Compression

Speedup

- #5104

• torch.full can't use implicit scalars #5097
• ERROR: aten:: is not Supported! #5090
• nni.compression.pytorch.speedup.error_code.UnBalancedGroupError: The number remained filters in each group is different #4864
• Pruning detectron2's Faster R-CNN #4875
  - speedup rand_like_with_shape is easy to overflow when dtype=torch.int8.
• speedup doesn't support registered buffer.
• Autoinfer can not deal with some op, sparsity cannot propagate through these ops, like torch.nn.functional.interpolate

Example

- #5037
- #5019
- #4818

Training Service

• nni cannot use tensorboard #5064

[matluster]

Candidate items for v3.0

I can't edit the Word document. :(

• API (breaking) change and code refactor of nas.nn and nas.experiment. Benefits include:

  • Incidentally rename retiarii to nas.
  • Code-base will be much simpler, more readable and maintainable.
  • Writing search space:
    • Support mutable types other than choice, e.g., uniform, normal.
    • Support constraints between parameters, e.g., a + b < 2.
    • Support annotation of search space, e.g., some choices constitute a DAG.
    • Static mutable types (e.g, choice), are promoted as a first-class citizen. Mutator will be a special kind of mutable.
    • Prettify model space interface. @model_wrapper and @basic_unit will be removed, in favor of ModelSpace and BasicUnit.
    • import nni.nas.nn.pytorch as nn is no longer required.
  • Strategy-related:
    • Symbolic execution engine as a utility, in preparation of hardware-aware NAS.
    • Unified search space format simplifies the programming of strategies.
  • Execution:
    • Python execution engine is promoted as a first-class citizen.
    • Support single-process mode. Training service is no longer a must for execution engine.
  • Experiment:
    • Rewrite interface of exp.__init__ and exp.run, to align with nni.Experiment.
    • Systematically support the "checkpoint" of a experiment.
  • Debuggability:
    • Export summary of search space for every submodule.
    • Ability to freeze any submodule to any specific architecture.
    • Validation of sampled architecture.
    • Ability to run sampled models in one single process, sequentially.

• Refactor hardware-aware NAS. Benefits:

  • Converting to Graph IR will be no longer required.
  • Latency queries will be greatly accelerated.

• [Maybe 3.1] Composition of strategy. Some challenges:

  • Chain two individual strategies in one experiment.
  • "Finetune":
    • Resume a previous experiment with another (different) strategy.
    • Use the (partial) results from a previous (different) strategy.
  • Use two strategies in parallel.
  • Strategy middleware:
    • Filter the submission of one strategy with another.
    • Bring about early-stop, successive halving, etc.

• [Need help] Evaluation of existing strategies and models.

  • On ImageNet.
  • On non-image-classification tasks.

[nnzs9248]

p1 - ~~a bug in speedup replace_module: #4991. @Louis-J ~10/10~~

any progress in this？