Optimize counting train & inference speed and memory consumption #2172

sungmanc · 2023-05-22T06:43:14Z

Summary

This PR includes some draft PRs,
Tiling Optimization for Zoom Factor

Optimize adaptive tiling #2164

Disable including full image and enable multi-batch tile classifier

[Counting Optimize] support multi-batch in tile classifier #2153

And

Change the default image resolution for tiling
Add deployment_tile.py to change the deploy_cfg for changed tiling parameters.
Add postprocessing named use_ellipse_shapes to use the Ellipse in the inference phase. It could be applied to inference and IR inference. So, it includes some changes in exportable_code.
Add tile_sampling to tiling detection task.

How to test

Checklist

I have added unit tests to cover my changes.
I have added integration tests to cover my changes.
I have added e2e tests for validation.
I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).
I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
I have linked related issues.

License

I submit my code changes under the same Apache License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

jaegukhyun

Thanks for your effort! I left some comments

otx/algorithms/common/configs/training_base.py

otx/algorithms/detection/adapters/mmdet/task.py

otx/algorithms/detection/configs/detection/configuration.yaml

otx/algorithms/detection/configs/instance_segmentation/configuration.yaml

otx/algorithms/detection/utils/data.py

otx/api/usecases/exportable_code/prediction_to_annotation_converter.py

goodsong81

Thank you for the gathering effort. A few comments of mine:

otx/algorithms/common/configs/training_base.py

otx/algorithms/detection/configs/instance_segmentation/efficientnetb2b_maskrcnn/deployment.py

otx/algorithms/detection/configs/instance_segmentation/resnet50_maskrcnn/deployment.py

otx/core/data/adapter/detection_dataset_adapter.py

...algorithms/detection/configs/instance_segmentation/efficientnetb2b_maskrcnn/tile_pipeline.py

otx/api/usecases/exportable_code/prediction_to_annotation_converter.py

exp.sh

otx/algorithms/detection/adapters/mmdet/task.py

otx/algorithms/detection/adapters/mmdet/utils/config_utils.py

tox.ini

otx/algorithms/detection/utils/data.py

jaegukhyun

I left some comments, and we need to add some unit tests for each optimization method

exp.sh

otx/algorithms/detection/adapters/mmdet/task.py

otx/algorithms/detection/configs/instance_segmentation/configuration.yaml

otx/algorithms/detection/configs/detection/configuration.yaml

…ence threshold filter for openvino eval

…iling

goodsong81

LGTM. Let's merge after test. Refinement could be followed in next PRs.

* Add sampling tiling dataset method * Add unit test * update * update * update configs * update * Update * Fix configuration for deployment * Add experimental script * Fix smallthing in exp.sh * Change default labels type from list to np.array * Revert resolution, num_workers * Fix typo * Skip mask if its confidence is under threshold * Add prediction with user defined confidence threshold * Update exp.sh * support multi-batch in tile classifier * polygon sampling * exclude full image in training * Add * Refine adaptive tile params - Use size rather than area - 32 pixels as min detectable size - Default object_tile_ratio = 32 / 1024 = 0.03 * Tune params * Fix tile on the edge * Fix IR overlap ratio according to IR scale factor Signed-off-by: Songki Choi <songki.choi@intel.com> * Set default object_tile_ratio = 0.06 (32/512) * Add tile_deployment * Keep aspect ratio for tiling * Fix tile patching * Fix adaptive tile logic for robustness * Delete useless file * Cleansing * Update exp.sh * Fix precommit * Refine adaptive tile params - Use size rather than area - 32 pixels as min detectable size - Default object_tile_ratio = 32 / 1024 = 0.03 * Tune params * Set default object_tile_ratio = 0.06 (32/512) * Keep aspect ratio for tiling * Fix tile patching * Fix adaptive tile logic for robustness * Remove useless line * Apply comments * Remove some comments * make black happy * Update default value of object tile ratio * Fix polygon append * Fix polygon append * Fix merge error, revert temp tox change * Fix precommit, remove exp.sh * Remove 20 points sampling * Fix mypy issue * Modify way to change subset type of _infer_model dataset & Add confidence threshold filter for openvino eval * Remove breakpoint * Update config param description, expose ellipse option * Fix unit test * Fix deploy patch bug (w,h) -> (h,w) * Fix iseg intg test * Revert tiling-ins-seg intg test * Fix for det tiling intg test --------- Signed-off-by: Songki Choi <songki.choi@intel.com> Co-authored-by: jaegukhyun <jaeguk.hyun@intel.com> Co-authored-by: Eugene Liu <eugene.liu@intel.com> Co-authored-by: sungmanc <sungmanc@intel.com> Co-authored-by: Songki Choi <songki.choi@intel.com>

* Optimize counting train & inference speed and memory consumption (#2172) * Do not skip full img tile classifier + Fix Sequencial Export Issue (#2174) * Add warning message to tiling parameter (#2193) * Change the way to pad for instance-segmentation * Hotfix: use 0 confidence threshold when computing best threshold based on F1 * Fix possible bw issue in exportable code (#2212) * Fix ellipse load (#2214) --------- Signed-off-by: Songki Choi <songki.choi@intel.com> Co-authored-by: Sungman Cho <sungman.cho@intel.com> Co-authored-by: jaegukhyun <jaeguk.hyun@intel.com> Co-authored-by: Eugene Liu <eugene.liu@intel.com> Co-authored-by: sungmanc <sungmanc@intel.com> Co-authored-by: Vladislav Sovrasov <sovrasov.vlad@gmail.com>

* Fix e2e (#2179) * Fix iseg e2e (#2173) * Fix det e2e (#2176) * Update noisy label det docs for detection task (#2183) * Fix iseg e2e (#2173) * Fix det e2e (#2176) * Add supcon seg for regression (#2177) * Fix iseg e2e (#2173) * Fix det e2e (#2176) * Update regression test config (#2186) * Fix for anomaly regression test (#2188) * Fix iseg regression (#2190) * Set path to save pseudo masks into workspace (#2185) * Temporary fix override_configs bug (#2187) * Fix iseg e2e (#2173) * Fix det e2e (#2176) * Cleaned up skip/xfail TCs on the classification e2e tests (#2196) * Added csv output to export dependencies (#2178) * Fix typo and enlarge the threshold for KPI tests (#2198) * bugfix: auto adapt batch size doesn't work with cls incr case (#2199) * Fix for ssd (#2197) * Turn off persistent worker if num_worker is zero in classification task (#2208) * Update test cases of common_semantic_segmentation_dataset (#2210) * Optimize counting train & inference speed and memory consumption (#2172) * Do not skip full img tile classifier + Fix Sequencial Export Issue (#2174) * Add warning message to tiling parameter (#2193) * Change the way to pad for instance-segmentation * Hotfix: use 0 confidence threshold when computing best threshold based on F1 * Fix possible bw issue in exportable code (#2212) * Fix ellipse load (#2214) * Fix GT vs prediction order mismatch due to different sorting result (#2216) * Return raw anomaly maps as metadata (#2217) * Fix merge error, revert to pymongo w/o version spec --------- Signed-off-by: Songki Choi <songki.choi@intel.com> Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com> Co-authored-by: Jaeguk Hyun <jaeguk.hyun@intel.com> Co-authored-by: Vinnam Kim <vinnam.kim@intel.com> Co-authored-by: Sungman Cho <sungman.cho@intel.com> Co-authored-by: Eunwoo Shin <eunwoo.shin@intel.com> Co-authored-by: Kim, Sungchul <sungchul.kim@intel.com> Co-authored-by: Yunchu Lee <yunchu.lee@intel.com> Co-authored-by: Jihwan Eom <jihwan.eom@intel.com> Co-authored-by: Eugene Liu <eugene.liu@intel.com> Co-authored-by: sungmanc <sungmanc@intel.com> Co-authored-by: Vladislav Sovrasov <sovrasov.vlad@gmail.com> Co-authored-by: Dick Ameln <dick.ameln@intel.com>

github-actions bot added ALGO Any changes in OTX Algo Tasks implementation API Any changes in OTX API BUILD CLI Any changes in OTE CLI labels May 22, 2023

sungmanc requested review from goodsong81, eugene123tw and jaegukhyun May 22, 2023 07:03

github-actions bot removed the CLI Any changes in OTE CLI label May 22, 2023

jaegukhyun reviewed May 22, 2023

View reviewed changes

goodsong81 reviewed May 22, 2023

View reviewed changes

eugene123tw reviewed May 22, 2023

View reviewed changes

...algorithms/detection/configs/instance_segmentation/efficientnetb2b_maskrcnn/tile_pipeline.py Show resolved Hide resolved

jaegukhyun previously approved these changes May 23, 2023

View reviewed changes

goodsong81 reviewed May 23, 2023

View reviewed changes

otx/api/usecases/exportable_code/prediction_to_annotation_converter.py Outdated Show resolved Hide resolved

jaegukhyun reviewed May 23, 2023

View reviewed changes

otx/api/usecases/exportable_code/prediction_to_annotation_converter.py Outdated Show resolved Hide resolved

jaegukhyun and others added 16 commits May 23, 2023 13:33

Add sampling tiling dataset method

68f264d

Add unit test

aec5fdc

update

5e66c9c

update

d990edf

update configs

dac1748

update

9959139

Update

16ae9c3

Fix configuration for deployment

c59425a

Add experimental script

a69e93b

Fix smallthing in exp.sh

0f8cd20

Change default labels type from list to np.array

e6a99bc

Revert resolution, num_workers

175d524

Fix typo

24bbcdc

Skip mask if its confidence is under threshold

eac92bb

Add prediction with user defined confidence threshold

9724030

Update exp.sh

fd0253c

sungmanc commented May 23, 2023

View reviewed changes

exp.sh Outdated Show resolved Hide resolved

otx/algorithms/detection/adapters/mmdet/task.py Outdated Show resolved Hide resolved

otx/algorithms/detection/adapters/mmdet/utils/config_utils.py Outdated Show resolved Hide resolved

tox.ini Outdated Show resolved Hide resolved

goodsong81 reviewed May 23, 2023

View reviewed changes

otx/algorithms/detection/utils/data.py Show resolved Hide resolved

jaegukhyun reviewed May 23, 2023

View reviewed changes

exp.sh Outdated Show resolved Hide resolved

otx/algorithms/detection/adapters/mmdet/task.py Outdated Show resolved Hide resolved

otx/algorithms/detection/adapters/mmdet/task.py Show resolved Hide resolved

Fix precommit, remove exp.sh

e116537

goodsong81 reviewed May 23, 2023

View reviewed changes

otx/algorithms/detection/configs/instance_segmentation/configuration.yaml Show resolved Hide resolved

goodsong81 reviewed May 23, 2023

View reviewed changes

otx/algorithms/detection/configs/instance_segmentation/configuration.yaml Outdated Show resolved Hide resolved

goodsong81 added 2 commits May 23, 2023 16:54

Remove 20 points sampling

c8e4df6

Fix mypy issue

4ad3b7f

eugene123tw reviewed May 23, 2023

View reviewed changes

otx/algorithms/detection/configs/detection/configuration.yaml Show resolved Hide resolved

Modify way to change subset type of _infer_model dataset & Add confid…

28ec2ae

…ence threshold filter for openvino eval

github-actions bot added the CLI Any changes in OTE CLI label May 23, 2023

jaegukhyun and others added 4 commits May 23, 2023 17:53

Remove breakpoint

b155fbf

Update config param description, expose ellipse option

6e38804

Merge remote-tracking branch 'sungmanc/optimal-tiling' into optimal-t…

22377d6

…iling

Fix unit test

befc201

goodsong81 previously approved these changes May 23, 2023

View reviewed changes

Fix deploy patch bug (w,h) -> (h,w)

a4bbbd5

goodsong81 dismissed their stale review via a4bbbd5 May 23, 2023 14:10

goodsong81 changed the title ~~Optimize counting [all gathered]~~ Optimize counting train & inference speed and memory consumption May 23, 2023

jaegukhyun added 2 commits May 24, 2023 01:47

Fix iseg intg test

516da40

Revert tiling-ins-seg intg test

8433197

jaegukhyun force-pushed the optimal-tiling branch from ea106e9 to 8433197 Compare May 24, 2023 01:30

Fix for det tiling intg test

9859994

goodsong81 marked this pull request as ready for review May 24, 2023 03:57

goodsong81 requested a review from a team as a code owner May 24, 2023 03:57

jaegukhyun approved these changes May 24, 2023

View reviewed changes

goodsong81 merged commit 156d1e8 into openvinotoolkit:releases/1.2.1 May 24, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimize counting train & inference speed and memory consumption #2172

Optimize counting train & inference speed and memory consumption #2172

sungmanc commented May 22, 2023

jaegukhyun left a comment

goodsong81 left a comment

jaegukhyun left a comment

goodsong81 left a comment

Optimize counting train & inference speed and memory consumption #2172

Optimize counting train & inference speed and memory consumption #2172

Conversation

sungmanc commented May 22, 2023

Summary

How to test

Checklist

License

jaegukhyun left a comment

Choose a reason for hiding this comment

goodsong81 left a comment

Choose a reason for hiding this comment

jaegukhyun left a comment

Choose a reason for hiding this comment

goodsong81 left a comment

Choose a reason for hiding this comment