Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Support perf result reference check for benchmark v2 #3096

Closed
wants to merge 243 commits into from
Closed

Support perf result reference check for benchmark v2 #3096

wants to merge 243 commits into from

Conversation

goodsong81
Copy link
Contributor

Summary

  • Add reference data & check method
  • Support otx-ref which enables benchmarking other versions of otx on this repo
    image
  • Fix a few issues

How to test

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added e2e tests for validation.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

harimkang and others added 30 commits November 23, 2023 13:24
@harimkang
Initialize v2 branch
c73eb19
@harimkang
Initialize roots settings
c6c2a17
@harimkang
Fix labeler GH workflow
291b716
@vinnamkim
Initial commit (#2670)
e538298 
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@sungmanc
Initialize classification for the v2 (#2678)
aac8c2a 
* Init classification

* Wrap _get_img_data

* Edit detection part

* change the ignore folders

* Reflect reviews

* Fix wrong addition

* Reflect reviews

* Edit the ignore scope

* Decouple OTXModel from MMX frameworks

* Fix pre-commit

* Remove MMXCompatibleModel and make separate function

* Rename the function

* Rename the functions

* Fix precommit
@harimkang
Enable local package installation and CLI draft implementation (#2679)
8a602e9 
* Edit pyproject.toml for packaging and installation & Add CLI class

* Add test_install test cases & fix description

* Fix install.py docstring style

* Align docsting types and replace tmp_dir_path to tmp_path
@vinnamkim
Add IterationTimer callback (#2682)
299d6c8 
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@harimkang
Enable Ruff-UP007 (PEP604) (#2684)
1be07c0 
Enable Ruff-UP007
@vinnamkim
Fix coverage error on CI (#2687)
4013bd5 
* Fix coverage error

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@vinnamkim
Make subset_name configurable (#2685)
9690df7 
* Make subset_name configurable

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@harimkang
Fix tox configuration in CI (#2689)
ee68ecd
@vinnamkim
Add memory caching (#2686)
f292868 
* Make subset_name configurable

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Add tests

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix templates to enable test

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Add memory caching

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Revise to make it prettier

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix guide.md

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix SIGINT handler

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

---------

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@jaegukhyun
Migrate Geti object detection models (#2690)
ced28c3 
* Migrate Geti object detection models

* Resolve wrong labeling

* Additional hyper params tuning

* Resolve cli issues

* Remove duplicated codes

* Add packages for classification

* Change message to msg

* Change log method
@jaegukhyun
Update recipe data subsets (hot-fix) (#2694)
368527b 
Update recipe data subsets
@vinnamkim
Fix warning from batch_size auto-inference failure (#2692)
b53095b 
* Fix warning from batch_size auto-inference failure

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Update unit test

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

---------

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@harimkang
Add Simple Integration test (CLI - 'otx train') (#2695)
ef7fdfb 
* Add Simple Integration test - CLI otx train

* Fix installation case without cuda/version.json

* Add classification overrides value
@vinnamkim
Turn on setting seed and deterministic flags (#2700)
6234433 
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@jaegukhyun
Change docstring location and learning hyperparams for object detecti…
7b3b23e 
…on (#2699)

Change docstring location and learning hyperparams
@kprokofi @vinnamkim
Add semantic segmentation models to new design (#2693)
62e99f1 
* added mmsegmentation as dependency

* fixed common_semantic_dataset

* fixed integration tests for segmentation

* fixed hard coded model loading. Changed metric

* add aditional dependency for segmentation

---------

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Co-authored-by: Kim, Vinnam <vinnam.kim@intel.com>
@harimkang
Fix integration test config settings (#2701)
c7e59e3 
* Add integration setting cfg into debug folder & Fix integration test to use os.system

* Replace os.system to main function

* Revert debug in TrainConfig

* mmsegmentation registry issue resolve
@harimkang
Organize and Enable callbacks per task (#2703)
c1708db 
* Align callbacks

* Add segmentation callbacks
@vinnamkim
Mem cache with resize (#2702)
abcf46f 
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@harimkang
Add 'otx test' command (#2707)
92ecbfc 
* Add 'otx test' & initial commits

* Add setup_guide to otx_test & add GlobalHydra().clear() into train, test

* Replace GlobalHydra clear to with initialize
@vinnamkim
Add torchvision v2 transform (#2709)
0ae7561 
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@sungmanc @vinnamkim
Bring the SOTA(DINOv2) model to the classification task (#2708)
9a1dc19 
* Initial commit for dinov2 implementation

* Add conftest

* Sync with the upstream

* Update test and add freeze, normalize

* Update tests

* Reflect reviews

* Remove num_classes at data preprocessor

* Edit the unit-test

* Remove useless data_preprocessor.num_classes

* mock the torch.hub.load, change the normalize

* Reflect reviews

---------

Co-authored-by: Vinnam Kim <vinnam.kim@intel.com>
@jaegukhyun
Add unit tests for otx/algo/detection and otx/core/data/(base, detect…
b57cee4 
…ion) (#2711)

* Add unit tests for algo

* Add unit tests for otx/core/data/

* Reflect latest PR

* Reflect reviews

* Set max_refetch
@eugene123tw @vinnamkim
@jaegukhyun @kprokofi @harimkang
Add Instance Segmentation models to new design (#2698)
479868b 
* add new inst-seg to new design

---------

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Co-authored-by: Kim, Vinnam <vinnam.kim@intel.com>
Co-authored-by: jaegukhyun <jaeguk.hyun@intel.com>
Co-authored-by: Prokofiev Kirill <kirill.prokofiev@intel.com>
Co-authored-by: Harim Kang <harim.kang@intel.com>
@vinnamkim
Add custom model example (#2712)
ee33e0e 
* Update data config

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Add resizing to mem caching

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Add example notebook

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix rebase conflicts

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Update docstring for custom resolvers

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix conflicts

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Add yaml config

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Fix test

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Rerun notebook

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Add example notebook for adding custom model

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Add more descriptions

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

* Remove manually not beautiful stdout from example

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>

---------

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@sungmanc
Change the cls head to make them same with the otx1.5 (#2716)
13d6bf3 
* Change the cls head to make them same with the otx1.5

* Fix mis-input channel for mvn3

* Add light models to guarantee the backward compatibility

* Fix mis-channel of mvn3
@vinnamkim
Add regression test workflow (#2718)
539cbe7 
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
eugene123tw and others added 24 commits March 11, 2024 09:32
@eugene123tw
Add RTMDet Inst Tiny (#3039)
196dc83 
* add rtmdet inst

* update recipe

* update rtmdet recipe

* reformat

* add pad_value
@sungchul2
Add optimize for visual prompting to 2.0 (#3040)
506a22f 
* Fix return's shape when all predicted masks are zero

* Swap order of prompts in segment_anything.py

* Enable to export segment anything modules respectively

* Update prompt order and refactoring

* Fix `return_single_mask` to True

* Add OVIR inference logic

* Updates for integration test

* precommit

* Update prompt getter

* (WIP) Update zero-shot

* (WIP) Internalize preprocess into model

* Update reference infos to buffer

* Update reference_info path

* Enable standalone infer logic

* Update location of `-load_latest_reference_info` for OVModel

* Update recipes

* Enable `infer` ov inference

* Enable standalone `infer` logic on OVModel

* precommit

* Update unittest

* Update for integration test

* precommit

* Fix

* Fix

* Fix

* Fix intg tests

* Enable to update `export_args` to `deploy_cfg`

* Update with walrus

* Update to use dict labels

* Change openvino model names

* Update compatibility with zero-shot

* Refactoring for unnecessary assigned variables

* Avoid repeatedly executing `torch.cat`

* precommit

* Fix unit test

* Update variable name

* Add `example_inputs` in anomaly

* Fix unit test

* Fix

* Update `model_names` for visual prompting

* precommit

* Not to include other params in `example_inputs`

* Disable condition for visual prompting

* Update to `example_inputs`

* Remove unused kwargs

* Update

* Remove unused parts

* Update exported models' names

* Remove `example_inputs`

* Add `OTXVisualPromptingModelExporter`

* Update overlapped region refinement

* Update `export_params`

* Enable optimize

* Add exportable code, but updating `demo.py` is required

* Update model_type

* Fix integration test
@sungmanc
Add hlabel parent resolver and integrate HLabelData + HLabelInfo --> …
8737903 
…HLabelInfo (#3046)

* Initial commit

* Add function: add ancestor

* Integrate HLabelData and HLabelInfo

* Add unit-test for _add_ancestor func

* Fix invalid names

* Fix precommit

* Fix hlabel{

* Add type checking

* enlarge the threshold

* Fix intg test
@goodsong81
Refine perf benchmark v2 (#3052)
457826a 
* Evauate exported models

* Evaluate optimized models

* Fix metric names

* Fix --model.model_name to --checkpoint for 'otx optimize' input

* Refactor date string tag

* Log metric to MLFlow

* Merge accuracy & efficiency tests

* Fix pre-commit

* Add per-task summary to csv output

* Fix default num_repeat to 5 for all data sizes

* Fix pre-commit

* Add train/e2e_time & train/epoch criteria

* Fix pre-commit

* Fix pre-commit
@goodsong81
Dedicate perf-benchmark v2 CI host (#3063)
25fc3e1
@sungmanc
Make the same config with OTX1.x : Edit the default factor for lr sch…
ec68bac 
…eduler and update warmup_step logic (#3060)

* Edit the default factor for lr scheduler and update warmup_step logic

* Edit the warmup logic, update the lr shceduler patience

* Fix unit tests

* Fix the logic

* Fix the logic

* Fix precommit
@eugene123tw
Adapt mask prediction to rotated box prediction (#3029)
5000bd9 
* update recipes

* add inst seg to module

* revert recipe changes

* udpate

* update pred_step

* update predict_step

* add predict in api test
@yunchu
Add dependabot.yaml (#3053)
3256f5c 
* add dependabot config
* fix bug on tox.ini
@yunchu
Update for resolving security issues (#3055)
0c4293d 
* update for resolving security issues

* add dependencies for installing pypa/build

* fix pip install command in dockerfile
@yunchu
Fix trivy-scan testenv (#3072)
5969033 
bump mlflow version to 2.11.1
@jaegukhyun
Add Movinet to action classification (#3059)
5d1358d 
* Add MoViNet

* Add regression test

* Add unit tests

* Modify unit test
@jaegukhyun
Add classification backbones unit test (#3065)
fdecc34
@goodsong81
Add test models for perf benchmark v2 (#3083)
50a4d17
@harimkang
Modify the Anomaly Task so that the integration Test works (#3007)
a616ce9 
* Add Anomaly intg test

* Fix anoamly integration test

* Fix unit-test

* Revert lower torchmetric things

* Fix export tests

* Fix anomalib version to hot-fix commit

* Fix anomalib issues

* Fix tox.ini

* Update otx_efficientnet_v2.yaml
@goodsong81
Add perf reference from v1.6
7b8c158
@goodsong81
Fix xml name edge case for visual prompting
371a160
@yunchu
Add docs workflow for latest & testing (#3089)
bf6709a 
add docs workflow for latest & testing
@harimkang
Bring docs from OTX 1.6 to 2.0 (#3087)
6b55251 
* Initial commit for documentation

* Fix typo

* Fix api tutorials

* Fix anomaly task part

* Fix typo train_config to train_subset

* Re-Rocate of api tutorial & write config file explain

* Fix wrong path in api_tutorial

* Change index to grid

* Fix tox build-docs

* Change tabs to tab-set
@sungmanc
Detect num_classes information for the H-label classification (#3064)
7b7de3c 
* Initial commit

* Add the logic for auto hlabel information

* Fix precommit
@goodsong81
Support perf reference checking
a8f04f2
@goodsong81
Support 'otx-ref' test independent from current branch
74c2d05
@goodsong81
Ignore MLFlow logging failure
7c94b80
@goodsong81
Merge remote-tracking branch 'upstream/v2' into test/perf-check-ref-v2
c59686b
@goodsong81
Add __CURRENT_BRANCH_COMMIT__ flag to select default otx-ref
5358a00
@github-actions github-actions bot added DEPENDENCY Any changes in any dependencies (new dep or its version) should be produced via Change Request on PM TEST Any changes in tests BUILD labels Mar 14, 2024
@goodsong81 goodsong81 changed the base branch from v2 to develop March 15, 2024 00:45
@goodsong81 goodsong81 closed this Mar 15, 2024
@goodsong81 goodsong81 deleted the test/perf-check-ref-v2 branch March 15, 2024 05:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@samet-akcay samet-akcay Awaiting requested review from samet-akcay samet-akcay will be requested when the pull request is marked ready for review samet-akcay is a code owner

@eugene123tw eugene123tw Awaiting requested review from eugene123tw eugene123tw will be requested when the pull request is marked ready for review eugene123tw is a code owner

@kprokofi kprokofi Awaiting requested review from kprokofi kprokofi will be requested when the pull request is marked ready for review kprokofi is a code owner

@sovrasov sovrasov Awaiting requested review from sovrasov sovrasov will be requested when the pull request is marked ready for review sovrasov is a code owner

@negvet negvet Awaiting requested review from negvet negvet will be requested when the pull request is marked ready for review negvet is a code owner

@Daankrol Daankrol Awaiting requested review from Daankrol Daankrol will be requested when the pull request is marked ready for review Daankrol is a code owner

@djdameln djdameln Awaiting requested review from djdameln djdameln will be requested when the pull request is marked ready for review djdameln is a code owner

@ashwinvaidya17 ashwinvaidya17 Awaiting requested review from ashwinvaidya17 ashwinvaidya17 will be requested when the pull request is marked ready for review ashwinvaidya17 is a code owner

@rajeshgangireddy rajeshgangireddy Awaiting requested review from rajeshgangireddy rajeshgangireddy will be requested when the pull request is marked ready for review rajeshgangireddy is a code owner

@atwinand atwinand Awaiting requested review from atwinand atwinand will be requested when the pull request is marked ready for review atwinand is a code owner

Assignees
No one assigned
Labels
BUILD DEPENDENCY Any changes in any dependencies (new dep or its version) should be produced via Change Request on PM TEST Any changes in tests
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.