Allow custom model inference and upload #1696

ghost · 2020-06-10T20:13:51Z

This patch allows users to upload any compatible object detection or segmentation model and run automatic annotation. This will close #1028

Motivation and context

Right now, you can only upload custom models that are supported by OpenVINO. Tensorflow Object Detection API is widely used and it would be helpful if we allow support for custom model inference. The same goes for MaskRCNN.

For object detection, you can upload .pb file for model and .csv file for classes. For auto segmentation, you can upload .h5 for model and .csv for classes. The format of csv is slighly different for both of them which we can put it in the docs.

How has this been tested?

This has been tested on local machine.

Checklist

[x ] I submit my changes into the develop branch
I have added description of my changes into CHANGELOG file
I have updated the documentation accordingly
I have added tests to cover my changes
I have linked related issues (read github docs)
I have increased versions of npm packages if it is necessary (cvat-canvas,
cvat-core, cvat-data and cvat-ui)

I will update CHANGELOG and add documentation once this has been approved. I expect some changes based on your suggestion. Please feel free to suggest any changes that can make this better.

License

I submit my code changes under the same MIT License that covers the project.
Feel free to contact the maintainers if that's a concern.
I have updated the license header for each file (see an example below)

# Copyright (C) 2020 Intel Corporation
#
# SPDX-License-Identifier: MIT

requesting for a review @nmanovic @azhavoro

coveralls · 2020-06-10T20:31:25Z

Pull Request Test Coverage Report for Build 5827

0 of 185 (0.0%) changed or added relevant lines in 4 files are covered.
223 unchanged lines in 12 files lost coverage.
Overall coverage decreased (-0.6%) to 65.23%

Changes Missing Coverage	Changed/Added Lines	%
cvat/apps/auto_annotation/model_manager.py	25	0.0%
cvat/apps/auto_annotation/views.py	39	0.0%
cvat/apps/tf_annotation/views.py	47	0.0%
cvat/apps/auto_segmentation/views.py	74	0.0%

Files with Coverage Reduction	New Missed Lines	%
cvat/apps/tf_annotation/views.py	1	0%
cvat/apps/auto_annotation/views.py	2	0%
datumaro/datumaro/plugins/coco_format/extractor.py	4	96.6%
cvat/apps/engine/media_extractors.py	5	76.18%
datumaro/datumaro/util/init.py	6	65.93%
datumaro/datumaro/util/tf_util.py	10	57.14%
cvat/apps/dataset_manager/bindings.py	12	93.29%
cvat/apps/engine/task.py	15	77.57%
cvat/apps/engine/utils.py	20	21.52%
datumaro/datumaro/plugins/voc_format/converter.py	29	89.26%

Totals
Change from base Build 5621:	-0.6%
Covered Lines:	10943
Relevant Lines:	16357

💛 - Coveralls

nmanovic · 2020-06-15T19:40:24Z

@azhavoro , could you please review, test, and comment?

cvat/apps/tf_annotation/views.py

azhavoro · 2020-06-16T09:43:59Z

cvat/apps/auto_segmentation/views.py

-        for key, labels in db_labels.items():
-            if labels in auto_segmentation_labels.keys():
-                labels_mapping[auto_segmentation_labels[labels]] = key
+        if int(mid) == 989898:


Please use django fixtures or post migrate signals to add proper records to the database. It will allow to not use a magic numbers.

Thanks. I need to check that. I will get back to you. Just to summarize what we do here, we assigned a specific unique number to the default models because we had int in the request.

azhavoro · 2020-06-16T09:44:07Z

cvat/apps/auto_segmentation/views.py

-    frames = frame_provider.get_frames(frame_provider.Quality.ORIGINAL)
-    for image_num, (image_bytes, _) in enumerate(frames):
+    image_num = 0
+    image_list = list(frame_provider.get_frames(frame_provider.Quality.ORIGINAL))


Could there be memory issue here? Why do you need cast to list here?

We didn't face any issues while testing, but yes I agree with you. We need a length of the frame_provider to figure out distribution among available GPUs. The Generator class does not have length function. Let me see if we can have an alternative way. Do you have any suggestions?

frame_provider has __len__ definition https://github.com/opencv/cvat/blob/develop/cvat/apps/engine/frame_provider.py#L83

thanks, I will update this.

cvat/apps/auto_segmentation/views.py

azhavoro · 2020-06-16T09:44:53Z

cvat/apps/auto_annotation/model_manager.py

@@ -144,16 +148,26 @@ def save_file_as_tmp(data):
        dl_model_id = create_empty(owner=owner)

    run_tests = bool(model_file or weights_file or labelmap_file or interpretation_file)
+    if is_custom in ["tensorflow","maskrcnn"]:


I see that these string constants are used many times, could you add enum for them?

Sure, do we have any specific file where we should create enum class? or do you have any suggestion where this should be created?

I think this file will be ok.

cvat/apps/auto_annotation/model_manager.py

azhavoro · 2020-06-16T09:45:35Z

components/auto_segmentation/install.sh

@@ -3,11 +3,11 @@

 set -e

-MASK_RCNN_URL=https://github.com/matterport/Mask_RCNN
+MASK_RCNN_URL=https://github.com/onepanelio/Mask_RCNN


The same as above.

We made several changes in Mask_RCNN repo to make it work with multiple gpus. So, if we change this then it won't work with multiple GPUs.

.vscode/settings.json

.vscode/launch.json

nmanovic · 2020-06-16T21:27:34Z

Here is an overview of what got changed by this pull request:

Complexity increasing per file
==============================
- cvat/apps/auto_segmentation/views.py  8
- cvat/apps/tf_annotation/views.py  6
- cvat/apps/auto_annotation/model_manager.py  1

See the complete overview on Codacy

nmanovic · 2020-06-17T18:11:51Z

@san999p , let's meet next week and discuss the PR. I want to present to you an alternative solution using nuclio serverless platform (https://github.com/nuclio/nuclio): https://github.com/opencv/cvat/tree/nm/serverless.

ghost · 2020-06-17T19:14:56Z

sure @nmanovic let me know what time works for you. Maybe send a calendly link?

nmanovic · 2020-06-30T06:53:59Z

@san999p , I will schedule a meeting this week. I will contact Rush and Donald. You can look at the PR to understand future changes: #1767

SoftnauticsLtf · 2020-07-13T12:19:51Z

@san999p @nmanovic
I think there is better way to support Tensorflow models. There should be generic flow to execute any tensorflow models rather than defining MASK_RCNN as separate framework.

We have simple implementation where user can:

Create model using Tensorflow frozen .pb file, inference script and label config file
Auto annotation will work in same flow.

This gives user freedom to use their own models for automation and also they can use any models from TF model zoo.

Let me know your views on this. Will submit PR soon, may be it will give more clear idea.

nmanovic · 2020-07-13T12:44:25Z

@kishan-sib ,

Let's look at your implementation. The main problem that every DL model needs an inference script which translates output layer to actual annotations which CVAT can understand. The script can depends on different python modules, it can be implemented using different programming languages, potentially need to support several TF frameworks and custom extensions. Too many problems which need to solve.

It is the primary reason why I'm looking at "serverless approach" to resolve all issues because each DL model will be a separate container. You can look at Mask RCNN implementation as serverless function for nuclio framework here: https://github.com/opencv/cvat/tree/nm/serverless/serverless/tensorflow/matterport/mask_rcnn/nuclio

savan77 · 2020-07-17T19:18:59Z

I think #1767 closes this PR.

nmanovic · 2020-07-18T04:55:04Z

@savan77 , thanks for the input. I will close the PR.

Savan Visalpara and others added 6 commits June 9, 2020 15:12

forntend changes for custom model inference and upload

b6f2601

add backend code for custom model upload

4402ab7

update backedn for custom model inference

9ac83c8

solve issues faced during testing locally

de7a3ba

fixed an oversight with num_classes calculation

f6ce918

fixed typo with framework

cddd354

ghost requested review from azhavoro, bsekachev and nmanovic as code owners June 10, 2020 20:13

azhavoro requested changes Jun 16, 2020

View reviewed changes

san999p added 6 commits June 16, 2020 11:09

update python interpreter

59f6875

update python path

904c8a7

change num_gpus check condition

7e18883

change is_custom to framework

2d147ea

changed is_custom to framework

02fc9f3

convert tabs to spaces

b568577

nmanovic closed this Jul 18, 2020

rushtehrani deleted the custom-model-upload-inference branch January 9, 2021 05:10

snyk-bot mentioned this pull request Apr 14, 2021

[Snyk] Upgrade react-redux from 7.2.2 to 7.2.3 #3089

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow custom model inference and upload #1696

Allow custom model inference and upload #1696

ghost commented Jun 10, 2020 •

edited by ghost

Loading

coveralls commented Jun 10, 2020 •

edited

Loading

nmanovic commented Jun 15, 2020

azhavoro Jun 16, 2020

ghost Jun 16, 2020

azhavoro Jun 16, 2020

ghost Jun 16, 2020

azhavoro Jun 18, 2020

ghost Jun 24, 2020

azhavoro Jun 16, 2020

ghost Jun 16, 2020

azhavoro Jun 18, 2020

ghost Jun 24, 2020

azhavoro Jun 16, 2020

ghost Jun 16, 2020

nmanovic commented Jun 16, 2020

nmanovic commented Jun 17, 2020

ghost commented Jun 17, 2020

nmanovic commented Jun 30, 2020

SoftnauticsLtf commented Jul 13, 2020

nmanovic commented Jul 13, 2020 •

edited

Loading

savan77 commented Jul 17, 2020

nmanovic commented Jul 18, 2020

Allow custom model inference and upload #1696

Allow custom model inference and upload #1696

Conversation

ghost commented Jun 10, 2020 • edited by ghost Loading

Motivation and context

How has this been tested?

Checklist

License

coveralls commented Jun 10, 2020 • edited Loading

Pull Request Test Coverage Report for Build 5827

💛 - Coveralls

nmanovic commented Jun 15, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

nmanovic commented Jun 16, 2020

nmanovic commented Jun 17, 2020

ghost commented Jun 17, 2020

nmanovic commented Jun 30, 2020

SoftnauticsLtf commented Jul 13, 2020

nmanovic commented Jul 13, 2020 • edited Loading

savan77 commented Jul 17, 2020

nmanovic commented Jul 18, 2020

ghost commented Jun 10, 2020 •

edited by ghost

Loading

coveralls commented Jun 10, 2020 •

edited

Loading

nmanovic commented Jul 13, 2020 •

edited

Loading