Add Simple Integration test (CLI - 'otx train')

[harimkang]

Summary

• Add CLI Integration Test using recipes
  • 1 epochs training with all recipes
  • Currently, only classification and detection work
• Add Classification test assets dataset: tests/assets/classification_dataset
• Add trainer.device=auto setting
• Add tox integration-test
• Add integration test into CI
• Fix install issue of case without 'version.json' in '/cuda/'

How to test

• TOX test
  tox -vv -e integration-test

• Local Test
  python -m pytest tests/integration/cli/test_cli.py

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added e2e tests for validation.
• I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• I have linked related issues.

License

• I submit my code changes under the same Apache License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

[sungmanc]

LGTM

[kprokofi]

In general, LGTM. But it might be worth extending the tests by 2 epochs instead of 1 to make sure that any processes between epochs also work.
What do you think?

[harimkang]

In general, LGTM. But it might be worth extending the tests by 2 epochs instead of 1 to make sure that any processes between epochs also work. What do you think?

I think it's a good idea, but for now I'll merge this PR for another task model recipe integration test and create a new PR after we discuss it together.

[vinnamkim]

I do not want to spawn a separate process for the integration test. This is because it makes a developer harder to debug if a problem is found during the integration test. Please replace it with executing the CLI Python function in the main process.

[vinnamkim]

Rather than manually overriding at the variable level, consider to override this https://github.com/openvinotoolkit/training_extensions/blob/v2/src/otx/config/debug/default.yaml

[harimkang]

Thanks for the comment. I will create a PR to reflect your comments. I have a question regarding this comment. I want to create a config file called debug/intg_test.yaml and want to apply it to test.
Can you tell me what args should be added to override the config in this debug folder in the cli?

[vinnamkim]

I guess that you need to add debug optional field to TrainConfig as same as recipe.

Then, it might be working by adding argument +debug=intg_test. This is exactly same with what we do with +recipe=???.

[harimkang]

Thank you. I'll give it a try.

[vinnamkim]

Why is this needed?

[harimkang]

I wanted to make sure that CI would work whether the test was using cpu or gpu, so I thought using the auto provided by lightning for the test would be appropriate. What do you think? This too can be modified as per your suggestion.

[vinnamkim]

Is that possible by giving trainer=cpu or trainer=gpu explicitly outside? If you set it as auto, then it seems not traceable which device we used for the tests.

[harimkang]

For test, yes i understand, I'm going to modify it in intg-test to gpu for now. Do you want to remove this file?
I guess auto might be a good option for the user as well.

[vinnamkim]

Then, let it leave as is. However, the decision for which device will be used should be explicit in the tests. This is what I only concern.

[harimkang]

@vinnamkim I create next PR #2701