Skip to content

Commit

Permalink
fix conflicts
Browse files Browse the repository at this point in the history
  • Loading branch information
Evgeny Tsykunov committed Jan 23, 2024
2 parents 0902aa1 + b75185c commit e0402bd
Show file tree
Hide file tree
Showing 204 changed files with 4,906 additions and 4,896 deletions.
198 changes: 198 additions & 0 deletions for_developers/cli_guide.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,198 @@
# How to use OTX CLI

## Installation

Please see [setup_guide.md](setup_guide.md).

## otx help

```console
otx --help
```

```powershell
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────╮
│ Usage: otx [-h] [-v] {install,train,test,predict,export} ... │
│ │
│ │
│ OpenVINO Training-Extension command line tool │
│ │
│ │
│ Options: │
│ -h, --help Show this help message and exit. │
│ -v, --version Display OTX version number. │
│ │
│ Subcommands: │
│ For more details of each subcommand, add it as an argument followed by --help. │
│ │
│ │
│ Available subcommands: │
│ install Install OTX requirements. │
│ train Trains the model using the provided LightningModule and OTXDataModule. │
│ test Run the testing phase of the engine. │
│ predict Run predictions using the specified model and data. │
│ export Export the trained model to OpenVINO Intermediate Representation (IR) o │
│ ONNX formats. │
│ │
╰─────────────────────────────────────────────────────────────────────────────────────────────────╯
```

The subcommand can get help output in the following way.
For basic subcommand help, the Verbosity Level is 0. In this case, the CLI provides a Quick-Guide in markdown.

```console
# otx {subcommand} --help
otx train --help
```

```powershell
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ OpenVINO™ Training Extensions CLI Guide ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Github Repository:
https://github.com/openvinotoolkit/training_extensions.
A better guide is provided by the documentation.
╭─ Quick-Start ─────────────────────────────────────────────────────────╮
│ │
│ 1 you can train with data_root only. then OTX will provide default │
│ model. │
│ │
│ │
│ otx train --data_root <DATASET_PATH> │
│ │
│ │
│ 2 you can pick a model or datamodule as Config file or Class. │
│ │
│ │
│ otx train │
│ --data_root <DATASET_PATH> │
│ --model <CONFIG | CLASS_PATH_OR_NAME> --data <CONFIG | │
│ CLASS_PATH_OR_NAME> │
│ │
│ │
│ 3 Of course, you can override the various values with commands. │
│ │
│ │
│ otx train │
│ --data_root <DATASET_PATH> │
│ --max_epochs <EPOCHS, int> --checkpoint <CKPT_PATH, str> │
│ │
│ │
│ 4 If you have a complete configuration file, run it like this. │
│ │
│ │
│ otx train --data_root <DATASET_PATH> --config <CONFIG_PATH, str> │
│ │
│ │
│ To get more overridable argument information, run the command below. │
│ │
│ │
│ # Verbosity Level 1 │
│ otx train [optional_arguments] -h -v │
│ # Verbosity Level 2 │
│ otx train [optional_arguments] -h -vv │
│ │
╰───────────────────────────────────────────────────────────────────────╯
```

For Verbosity Level 1, it shows Quick-Guide & the essential arguments.

```console
otx train --help -v
```

```powershell
┏━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┓
┃ OpenVINO™ Training Extensions CLI Guide ┃
┗━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━┛
Github Repository:
https://github.com/openvinotoolkit/training_extensions.
A better guide is provided by the documentation.
╭─ Quick-Start ─────────────────────────────────────────────────────────╮
│ ... │
╰───────────────────────────────────────────────────────────────────────╯
╭─ Arguments ─────────────────────────────────────────────────────────────────────────────────────╮
│ Usage: otx [options] train [-h] [-c CONFIG] [--print_config [=flags]] │
│ [--data_root DATA_ROOT] [--task TASK] │
│ [--engine CONFIG] │
│ [--engine.work_dir WORK_DIR] │
│ [--engine.checkpoint CHECKPOINT] │
│ [--engine.device {auto,gpu,cpu,tpu,ipu,hpu,mps}] │
│ [--model.help CLASS_PATH_OR_NAME] │
│ [--model CONFIG | CLASS_PATH_OR_NAME | .INIT_ARG_NAME VALUE] │
│ [--data CONFIG] │
│ [--optimizer CONFIG | CLASS_PATH_OR_NAME | .INIT_ARG_NAME VALUE] │
│ [--scheduler CONFIG | CLASS_PATH_OR_NAME | .INIT_ARG_NAME VALUE] │
│ │
...
```

For Verbosity Level 2, it shows all available arguments.

```console
otx train --help -vv
```

## otx {subcommand} --print_config

Preview all configuration values that will be executed through that command line.

```console
otx train --config <config-file-path> --print_config
```

```yaml
data_root: tests/assets/car_tree_bug
callback_monitor: val/map_50
engine:
task: DETECTION
work_dir: ./otx-workspace
device: auto
model:
class_path: otx.algo.detection.atss.ATSS
init_args:
num_classes: 1000
variant: mobilenetv2
optimizer: ...
scheduler: ...
data:
task: DETECTION
config:
data_format: coco_instances
train_subset: ...
val_subset: ...
test_subset: ...
mem_cache_size: 1GB
mem_cache_img_max_size: null
image_color_channel: RGB
include_polygons: false
max_epochs: 2
deterministic: false
precision: 16
callbacks: ...
logger: ...
```
Users can also pre-generate a config file with an example like the one below.
```console
otx train --config <config-file-path> --print_config > config.yaml
```

## otx {subcommand}

Use Configuration file

```console
otx train --config <config-file-path> --data_root <dataset-root>
```

Override Parameters

```console
otx train ... --model.num_classes <num-classes> --max_epochs <max-epochs>
```
6 changes: 2 additions & 4 deletions for_developers/dir_structure.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
root/
algo/ # Custom algo (e.g., hierarchical_cls_head)
cli/ # CLI entrypoints
config/ # Default YAML config files
engine/ # OTX Engine with Entry Point
core/
config/ # Structured data type object for configurations
data/ # Data related things
Expand All @@ -17,9 +17,6 @@ root/
transform_libs/ # To support transform libraries (e.g., MMCV)
factory.py # Factory to instantiate data related objects
module.py # OTXDataModule
engine/ # PyTorchLightning engine
train.py
...
model/ # Model related things
entity/ # OTXModel
base.py
Expand All @@ -32,6 +29,7 @@ root/
types/ # Enum definitions (e.g. OTXTaskType)
utils/ # Utility functions
recipe/ # Recipe YAML config for each model we support
_base_/ # Default YAML config files
detection/ # (e.g., rtmdet_tiny)
...
tools/ # Python runnable scripts for some TBD use cases
Expand Down
36 changes: 11 additions & 25 deletions for_developers/setup_guide.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,15 +12,11 @@ conda activate otx-v2
# Install PyTorch and TorchVision
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

# Install core dependency
pip install lightning datumaro omegaconf hydra-core

# Install mmcv (mmdet)
pip install -U openmim
mim install mmengine "mmcv>=2.0.0" mmdet

# Install this package
# Install otx with core requirements
pip install -e .

# otx install (install mmX)
otx install -v
```

### With PIP & 'otx install'
Expand All @@ -37,12 +33,8 @@ pip install -e .
otx --help

# Install torch & lightning base on user environments
otx install
# or 'otx install -v' (Verbose mode)

# Install other mmlab library or optional-dependencies
otx install --option dev
# or 'otx install --option mmpretrain'
otx install -v
# or 'otx install' (Not verbose mode)
```

Please see [requirements-lock.txt](requirements-lock.txt). This is what I got after the above installation steps by `pip freeze`.
Expand All @@ -52,31 +44,25 @@ Please see [requirements-lock.txt](requirements-lock.txt). This is what I got af
- Launch detection task ATSS-R50-FPN template

```console
otx train +recipe=detection/atss_r50_fpn base.data_dir=tests/assets/car_tree_bug model.otx_model.config.bbox_head.num_classes=3 trainer.max_epochs=50 trainer.check_val_every_n_epoch=10 trainer=gpu base.work_dir=outputs/test_work_dir base.output_dir=outputs/test_output_dir
otx train --config src/otx/recipe/detection/atss_r50_fpn.yaml --data_root tests/assets/car_tree_bug --model.num_classes=3 --max_epochs=50 --check_val_every_n_epoch=10 --engine.device gpu --engine.work_dir ./otx-workspace
```

- Change subset names, e.g., "train" -> "train_16" (for training)

```console
otx train ... data.train_subset.subset_name=<arbitrary-name> data.val_subset.subset_name=<arbitrary-name> data.test_subset.subset_name=<arbitrary-name>
```

- Do test with the best validation model checkpoint

```console
otx train ... test=true
otx train ... --data.config.train_subset.subset_name <arbitrary-name> --data.config.val_subset.subset_name <arbitrary-name> --data.config.test_subset.subset_name <arbitrary-name>
```

- Do train with the existing model checkpoint for resume

```console
otx train ... checkpoint=<checkpoint-path>
otx train ... --checkpoint <checkpoint-path>
```

- Do experiment with deterministic operations and the fixed seed

```console
otx train ... trainer.deterministic=True seed=<arbitrary-seed>
otx train ... --deterministic True --seed <arbitrary-seed>
```

- Do test with the existing model checkpoint
Expand All @@ -85,4 +71,4 @@ Please see [requirements-lock.txt](requirements-lock.txt). This is what I got af
otx test ... checkpoint=<checkpoint-path>
```

`trainer.deterministic=True` might affect to the model performance. Please see [this link](https://lightning.ai/docs/pytorch/stable/common/trainer.html#deterministic). Therefore, it is not recommended to turn on this option for the model performance comparison.
`--deterministic True` might affect to the model performance. Please see [this link](https://lightning.ai/docs/pytorch/stable/common/trainer.html#deterministic). Therefore, it is not recommended to turn on this option for the model performance comparison.
20 changes: 6 additions & 14 deletions pyproject.toml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -28,15 +28,16 @@ classifiers = [
"Programming Language :: Python :: 3.10",
]
dependencies = [
"datumaro",
"datumaro==1.6.0rc1",
"omegaconf",
"hydra-core",
"rich",
"jsonargparse==4.27.1",
"psutil", # Mem cache needs system checks
"ftfy",
"regex",
"importlib_resources"
"importlib_resources",
"docstring_parser", # CLI help-formatter
"rich_argparse", # CLI help-formatter
]

[project.optional-dependencies]
Expand Down Expand Up @@ -70,8 +71,8 @@ base = [
"lightning==2.1.2",
"pytorchcv",
"timm",
"openvino==2023.0.0",
"openvino-dev==2023.0.0",
"openvino==2023.2.0",
"openvino-dev==2023.2.0",
"openvino-model-api==0.1.7",
"onnx==1.13.0"
]
Expand Down Expand Up @@ -245,21 +246,12 @@ exclude = [
"tests/assets",

# it will be cleaned up later
"src/otx/core/engine/utils/*",
"src/otx/algo/classification/backbones/*",
"for_developers/helpers.py",

# Ruff complains it but don't know how to fix since it literally showed no useful logs.
# https://github.com/openvinotoolkit/training_extensions/actions/runs/7176557723/job/19541622452?pr=2718#step:5:170
"tests/regression/*.py",

# Temporarily disable it. This will be fixed in the next PR.
# [TODO] harimkang
"src/otx/core/utils/instantiators.py",
"src/otx/core/utils/logging_utils.py",
"src/otx/core/utils/pylogger.py",
"src/otx/core/utils/rich_utils.py",
"src/otx/core/utils/utils.py",
]

# Same as Black.
Expand Down
10 changes: 9 additions & 1 deletion src/otx/algo/classification/metrics/hlabel_accuracy.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@

from __future__ import annotations

from typing import Callable, Sequence

import torch
from torch import nn
from torchmetrics import Metric
from torchmetrics.classification import Accuracy, MultilabelAccuracy

Expand Down Expand Up @@ -64,6 +67,12 @@ def _metric_to_device(self, metric: Metric, device: str) -> None:
metric.to(device)
self.flag = True

def _apply(self, fn: Callable, exclude_state: Sequence[str] = "") -> nn.Module:
self.multiclass_head_accuracy = [acc._apply(fn, exclude_state) for acc in self.multiclass_head_accuracy] # noqa: SLF001
if self.num_multilabel_classes > 0:
self.multilabel_accuracy = self.multilabel_accuracy._apply(fn, exclude_state) # noqa: SLF001
return self

def update(self, preds: torch.Tensor, target: torch.Tensor) -> None:
"""Update state with predictions and targets."""
# Split preds into multiclass and multilabel parts
Expand All @@ -74,7 +83,6 @@ def update(self, preds: torch.Tensor, target: torch.Tensor) -> None:

is_all_multiclass_ignored = not multiclass_mask.any()
if not is_all_multiclass_ignored:
self._metric_to_device(self.multiclass_head_accuracy[head_idx], preds.device)
self.multiclass_head_accuracy[head_idx].update(
preds_multiclass[multiclass_mask],
target_multiclass[multiclass_mask],
Expand Down
2 changes: 1 addition & 1 deletion src/otx/algo/detection/atss.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@
class ATSS(MMDetCompatibleModel):
"""ATSS Model."""

def __init__(self, num_classes: int, variant: Literal["mobilenetv2", "f50_fpn", "resnext101"]) -> None:
def __init__(self, num_classes: int, variant: Literal["mobilenetv2", "r50_fpn", "resnext101"]) -> None:
model_name = f"atss_{variant}"
config = read_mmconfig(model_name=model_name)
super().__init__(num_classes=num_classes, config=config)
Loading

0 comments on commit e0402bd

Please sign in to comment.