Align Engine.from_config with CLI behavior

[harimkang]

Summary

• Refactoring Engine.from_config() & CLI
• Align Engine.from_config and CLI to have the same behavior as much as possible

The two examples below do the same thing.
API: from_config

from otx.engine import Engine

recipe_path = "src/otx/recipe/classification/multi_class_cls/otx_efficientnet_b0.yaml"
data_root = "/home/harimkan/workspace/repo/datasets/otx_v2_dataset/multiclass_classification/multiclass_CUB_medium"


engine = Engine.from_config(
    config_path=recipe_path, data_root=data_root, work_dir="otx-workspace-api", seed=1234
)

# Metric is now engine.train, engine.test argument, So we need to declare this separately.
from functools import partial
from torchmetrics.classification.accuracy import Accuracy

metric = partial(Accuracy, task="multiclass")
engine.train(
    deterministic=True,
    max_epochs=90,
    precision=16,
    metric=metric,
)

engine.test(metric=metric)

CLI command

otx train \
--config src/otx/recipe/classification/multi_class_cls/otx_efficientnet_b0.yaml \
--data_root /home/harimkan/workspace/repo/datasets/otx_v2_dataset/multiclass_classification/multiclass_CUB_medium \
--seed 1234 \
--deterministic True \
--work_dir otx-workspace-cli

Same Test results

How to test

Checklist

• I have added unit tests to cover my changes.​
• I have added integration tests to cover my changes.​
• I have added e2e tests for validation.
• I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
• I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
• I have linked related issues.

License

• I submit my code changes under the same Apache License that covers the project.
  Feel free to contact the maintainers if that's a concern.
• I have updated the license header for each file (see an example below).

# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

[sungmanc]

Thanks for the PR, please check my comments

[vinnamkim]

I don't know why you need this because it will show a different behavior from my expectation if an user executes the code not exactly as you provided such as

engine = Engine.from_config(
    config_path=recipe_path, data_root=data_root, work_dir="otx-workspace-api", seed=1234
)

***********************************************************
Execute some codes generating random variables,
so that change random number generator's inner states
***********************************************************

engine.train(
    deterministic=True,
    max_epochs=90,
    precision=16,
    metric=metric,
)

engine.test(metric=metric)

If you really need this, I think that it only happens if deterministic=True, I would recommend you to implement

class Engine:
    def train(self, deterministic=True):
          if deterministic:
              reset_random_number_generator()
              reinit_model_weights()
              reinit_data_sampler()
          train()

[harimkang]

I don't know why you need this because it will show a different behavior from my expectation if an user executes the code not exactly as you provided such as

If you really need this, I think that it only happens if deterministic=True, I would recommend you to implement

class Engine:
    def train(self, deterministic=True):
          if deterministic:
              reset_random_number_generator()
              reinit_model_weights()
              reinit_data_sampler()
          train()

Yes you are right, we need to take care of this on the seed, deterministic side to get the same behavior for now (which probably doesn't look natural to the user).
I'll probably think about your suggestion in a future PR and take a look at it, thanks for the suggestion!