NNI Backlog - welcome to comment and suggest

[scarlett2018]

This ticket is to track issues that are not yet planned but considered as future candidate items. To review or follow active undergoing efforts, please refer to the following plans:

v2.3-monthly-release-candiate

NAS 2.0-Retiarii-support-roadmap

As NNI's user, you are highly encouraged to comment and suggest the priority, preference, suggestions, and needs for the backlog tickets. Please feel free to add your thoughts on this backlog item or the issue you are interested in.

conventions

Mark	Description
🏃‍♂️	work in progress
👪	work under review
✋	blocked task
🔴	missing issue/pull request reference
🔵	more investigation and survey required
⚫	under discussion within the team / not started / external contribution
⬛	a large work item, larger than one iteration

BACKLOG (no rank)

To review the full listed issues: Backlog Milestone

NAS

NAS2.0-Retiarii-support-roadmap

• Support control flow in graph/code generation
• P2 - Migrate all the one-shot trainer from NAS v1 to Retiarii framework (i.e., NAS v2) (deferred from v2.1)
• P2 - Support one system-level optimization: CSE optimization to deduplicate computation (deferred from v2.1)
• P3 - Test classic nas tuners on NNI NAS benchmark and report evaluation results (deferred from v2.0) @ultmaster
• More example for one-shot NAS practice of DNN/MLP models Can we have example of utilizing one-shot NAS for DNN models? #3593
• Transformations available on 'ValueChoice' Transformations available on 'ValueChoice'  #3571
• 🔴 DARTS+
• 🔴 Single-Path-NAS
• 🔴 CycleNAS
• 🔵🔴 AutoDeepLab (TBD)
• ⚫ ProxylessNas implementation support optimizing inference latency Does ProxylessNas implementation really support optimizing inference latency? #3113
• 🏃‍♂️ NAS优化相关 WebUI, NasUI Enhancement Backlog #2683
  • WebUI improvement and bugs backlog
  • Yuge's current NAS algo adding to NNI as example

HAO - Joint Hyperparameter and Architecture Optimization

• more to add

Hyper-parameter tuning

• P2 - Improve hyperband, PBT with shared storage (dependency on shared storage implementation) @J-shang
• P3 - Formal Advisor API @liuzhe-lz
• P2 - Multi-objective optimization algorithm (deferred from v2.0) @ultmaster NNI多目标搜索算法 #2525
• set default values for hyper parameters Can we set default values of hyper parameters for HPO? #3697
• ⚫ Tuners should have precondition check Tuners should have precondition check #1667
• 🔵 model ensembling by custom tuner is there plan to integrate ensemble step into the nni? #2414
• ⚫ SimpleTuner [Vote] Need a SimpleTuner interface like this proposal? #2151 (@ultmaster ) (need discussion, book meeting)
• ⚫ PPO Tuner for HPO Is PPO tuner in HPO working for hyperparameter optimization? #3370

Model Compression

• ⚫🔴 Data-independent algorithms
• ⚫🔴 Data-dependent algorithms
• ⚫🔴 Network Trimming: A Data-Driven Neuron Pruning Approach towards Efficient Deep Architectures
• ⚫🔴 Pruning Convolutional Neural Networks for Resource Efficient Inference
• ⚫ Enhance the group dependency analysis tools enhance the group dependency analysis tools #2740
• ⚫ Model speedup support request -> refactor in Model Speedup Refactor #3462
  • ⚫ ModelSpeedup support for Upsample, aten::chunk and aten::exp ModelSpeedup support for Upsample, aten::chunk and aten::exp #3329
  • ⚫ ModelSpeedup support for LeakyReLU and aten::split Add support for LeakyReLU and Split in ModelSpeedup? #3393
  • ⚫ ModelSpeedup support fine-grained pruning: 3D model compression [New Feature Request] support 3D model compression #3363
• 3D model compression [New Feature Request] support 3D model compression #3363
• ⚫🔴 BinaryConnect: Training Deep Neural Networks with binary weights during propagations
• ⚫🔴 Binarized Neural Networks: Training Neural Networks with Weights and Activations Constrained to +1 or−1
• ⚫🔴 TRAINED TERNARY QUANTIZATION
• ⚫🔴 Ternary weight networks
• ⚫🔴 XNOR-Net:ImageNet Classification Using Binary Convolutional Neural Networks
• ⚫ Post-training quantization support -> TensorRT supported in v2.2
• ⚫ 精进 Model compression 实现 for real workload like face detection

training service

• P2 - Hybrid kubeflow support @SparkSnail
• P2 - Documentation improvement @QuanluZhang
• P2 - Add more error message for remote training service (more error message for remote training service #3259) @QuanluZhang (fixed upstream, to verify the fix)
• P3 - Unify IPC mechanism for local and others @liuzhe-lz @ultmaster
• ⚫ Several issues in AML training service Several issues in AML training service #3174
• ⚫ Enable optional Pod Spec for FrameworkController platform Enable optional Pod Spec for FrameworkController platform #3350
• ⚫ Remote training service: the error is inaccurate remote training service: the error is inaccurate #2804
• 🏃‍♂️ Reusable environment enhancement Reusable environment enhancement #2671
• 🏃‍♂️ nni PAI training service user experience issues nni PAI training service user experience issues #2567

nnictl & nni.experiment & nnimanager & rest server

• P2 - internal tuner APIs for improving tuner debuggability, allow debugger tool @liuzhe-lz
• P2 - nnictl auto-completion @liuzhe-lz
• P2 - add/del tag of experiments (pending support from rest server)
• P3 - launch NNI manager in remote container #modelbuilderneeds @liuzhe-lz

webui

• P2 - add/del tag of experiments
• P2 - Adaptive UI (height) for high resolution, current window for high resolution need improvement @yangou1988 @Lijiaoa
• P3 - Fix redundant margin at the bottom of the page
• 🏃‍♂️ All experiments managements page improvement HPO experiments management web dashboard #3100
• 🏃‍♂️ NAS visualization WebUI, NasUI Enhancement Backlog #2683
• ⚫ Show the current ram load/cpu gpu/load on WebUI Add Htop to the Webui #3441
• ⬛ Hyperparameter graph: val_loss as metric Hyperparameter graph: val_loss as metric #1483 (shall we consider this for WebUI? @Lijiaoa
• 🔴 Update search space on WebUI manually
• 🔴 Model Graph
• 🔴 Hyperparameter Importance Analysis

API and Command related

• 🔴 Customized trial API for tuner
• 🔴 Advisor API
• 🔴 Support IPV6
• 🔴 Support configurate the parameters of each trial manually
• ⚫ Add more log info about GPU usage add more log info about GPU usage #1723
• ⚫ Too many setup.py's Too many setup.py's #1626
• ⚫ Get rid of netifaces dependency Suggest to get rid of netifaces dependency #2895
• ⚫ Add support for Jupyter notebooks [Enhancement] Add support for Jupyter notebooks #3338 Is there a way to use NNI inside Jupyter Notebook? #2932
• ⚫ Save the models and weights in PTH format I work in an AI company, and we hope that NNI can not only save JSON representing the optimal model, but also the optimal model containing weights. #2620
• ⚫ Provide one click to collect all troubleshooting information Provide one click to collect all troubleshooting information #2539
• ⚫ Saved NAS model should not have NNI dependency Saved NAS model should not have NNI dependency  #2770
• ⚫ Support unicode in metric data better. Support unicode in metric data better. #2575
• ⚫ Add a new API for tuner "get_best_parameter" Add a new API for tuner "get_best_parameter" #1530
• 🔵 Refactor gpu collector logic Refactor gpu collector logic #1442
• 🔵 Add a new API for tuner "get_best_parameter" Add a new API for tuner "get_best_parameter" #1530
• 🔵 name in annotation shouldn't depends on following line. name in annotation shouldn't depends on following line. #687
• 🔵 Split hyperparams tuner into a separate lib and combine with UniOpt Split hyperparams tuner into a separate lib and combine with UniOpt #1774
• 🔵 There is some risk when using nnictl update concurrency There is some risk when using nnictl update concurrency #1725

Others

• P3 - Bump dependency version (deferred from v2.0) @liuzhe-lz
• P4 - AutoML for data augmentation (image and NLP) (research)
• P3 - Improve dev-install on Windows @liuzhe-lz
• P3 - Support code coverage check, add this check to IT (deferred from v2.0) @ultmaster
• P2 - Support nnictl tensorboard in Windows @J-shang
• P2 - Mask confidential config fields @liuzhe-lz
• ⚫ Provide one click to collect all troubleshooting information Provide one click to collect all troubleshooting information #2539
• ⚫ Framework for Model Selection Framework for Model Selection #3376
• ⚫ What's the input and the output of nni? What's the input and the output of nni? #1371
• 🏃‍♂️ development doc @liuzhe-lz (Improve development doc #2307) (in wiki, ongoing)
• 🏃‍♂️ Internal scenario based usage: CV and NLP
• 🔴 Advancing job performance: working pool to reuse job Advancing job performance: working pool to reuse job #2391 (@squirrelsc )

documentation

• P2 - Improve crowdin workflow @ultmaster

Design Discussion Needed

• Design discussion for next steps of manage all experiments, including "actions" for Manage experiments, etc. HPO experiments management web dashboard #3100
• Design discussion for Duration issue [webui] about trial duration question #3012
• fail trials counted in total trial nums
• Improve a more typing-friendly experiment id pattern (Proposal: a more typing-friendly experiment id pattern #3102)
• Avoid reloading data for each trail (Did every trail need to load data? #3294) @QuanluZhang
• Experiment tag

🐛 Open Bugs

• ⬛ open bugs: bug Something isn't working

[squirrelsc]

#2390 is already done. I'm working on #2391