Add otx micro benchmark

[sovrasov]

Summary

otx benchmark command aims to provide micro benchmarking capabilities for OTX users.
Given a model (OV or torch) otx benchmark could quickly estimate realistic deployment performance and explore the effect of varying batch size. The evaluation works on synthetic data, which is provided by OTXModel.get_dummy_input(batch_size) method. Also, in case of torch models theoretical complexity and amount of parameters are available.

Issues:

• get_dummy_input() should be implemented for each model type. Current limitation is hard-coded per model input resolutions. The same problem is valid for export.
• torch 2.1 messes up with Tensor types in dispatch mode, a workaround was added. There is a chance torch 2.3 resolves the issue.
• Models coverage.
• Unclear console output model in Engine.

How to test

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have ran e2e tests and there is no issues.
• I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• I have linked related issues.

License

• I submit my code changes under the same Apache License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

[eunwoosh]

Thanks for your work. I left some comments. Please take a look.

[kprokofi]

Thank you for this valuable feature.

I have few comments/proposals. I checked this functionality with segmentation models.

1. Currently, it is required to provide data_root for all OTX CLI commands. Since this functionality doesn't require any datasets, can we add some default datamodule (if there is no other provided) to run otx benchmark without --data_root option?

2. It will be a good option if we integrate otx.export (OV) to otx.benchmark. For example, I want to test torch models and OV models. For now, I should train them first (or remove checkpoint loading logic in OTX and use default weights), then export and after that use otx benchmark. It ends up to 2-3 commands and searching for checkpoint paths. Could we add like "convert_to_openvino" or "test_openvino" option, call export and call benchmark again with OV model?

[sovrasov]

Thank you for this valuable feature.

I have few comments/proposals. I checked this functionality with segmentation models.

1. Currently, it is required to provide data_root for all OTX CLI commands. Since this functionality doesn't require any datasets, can we add some default datamodule (if there is no other provided) to run otx benchmark without --data_root option?
2. It will be a good option if we integrate otx.export (OV) to otx.benchmark. For example, I want to test torch models and OV models. For now, I should train them first (or remove checkpoint loading logic in OTX and use default weights), then export and after that use otx benchmark. It ends up to 2-3 commands and searching for checkpoint paths. Could we add like "convert_to_openvino" or "test_openvino" option, call export and call benchmark again with OV model?

1. That's the limitation of current OTX implementation. Without having a dataset one should perform full model auto-configuration, including setting num classes for instance. In theory, export command also doesn't require any data, but we just don't have a mechanism to skip the creation of an actual dataloader. I'll think about an API-based benchmark, so we could streamline the workflow at least there.
2. The best we can do now is to benchmark a torch model without --checkpoint specified (~no training for torch model is required). otx * tools are supposed to be orthogonal, so none of it implicitly uses another. This introduces a clear understanding of dependencies required for each otx * tool, but it could take more time to achieve the result. If we relax otx export requirements (omit --checkpoint), that could mitigate the need of pretraining weights, but still 2 calls are required.

[sungchul2]

Thanks for your hard work! It seems the feature is what we really need.
I left some minor comments.

[ashwinvaidya17]

This is a nice addition. Looks good from Anomaly perspective.

[kprokofi]

Thank you for this feature, overall looks good