Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add gradient clipping and sync hyper parameters wit OTX1.x #2786

Merged
merged 1 commit into from
Jan 15, 2024
Merged

Add gradient clipping and sync hyper parameters wit OTX1.x #2786

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions src/otx/config/trainer/default.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -17,3 +17,5 @@ check_val_every_n_epoch: 1
# set True to to ensure deterministic results
# makes training slower but gives more reproducibility than just setting seeds
deterministic: False

gradient_clip_val: null
6 changes: 5 additions & 1 deletion src/otx/core/config/trainer.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,8 +3,11 @@
#
"""Config data type objects for trainer."""

from __future__ import annotations

from dataclasses import dataclass
from pathlib import Path
from pathlib import Path # noqa: TCH003
from typing import Optional

from omegaconf import DictConfig

Expand All @@ -21,5 +24,6 @@ class TrainerConfig(DictConfig):
devices: int
check_val_every_n_epoch: int
deterministic: bool
gradient_clip_val: Optional[float]

_target_: str = "lightning.pytorch.trainer.Trainer"
9 changes: 6 additions & 3 deletions src/otx/recipe/detection/atss_mobilenetv2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# @package _global_
defaults:
- override /trainer: default
- override /base: detection
- override /callbacks: detection
- override /data: mmdet
Expand Down Expand Up @@ -39,7 +40,7 @@ data:
- img_shape
- pad_shape
val_subset:
batch_size: 1
batch_size: 8
transforms:
- type: LoadImageFromFile
- type: Resize
Expand All @@ -58,7 +59,7 @@ data:
- img_shape
- pad_shape
test_subset:
batch_size: 1
batch_size: 8
transforms:
- type: LoadImageFromFile
- type: Resize
Expand Down Expand Up @@ -177,5 +178,7 @@ model:
_target_: torch.optim.SGD
_partial_: true
lr: 0.004
weight_decay: 0.001
weight_decay: 0.0001
momentum: 0.9
trainer:
gradient_clip_val: 35.0
11 changes: 7 additions & 4 deletions src/otx/recipe/detection/atss_resnext101.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# @package _global_
defaults:
- override /trainer: default
- override /base: detection
- override /callbacks: detection
- override /data: mmdet
Expand Down Expand Up @@ -39,7 +40,7 @@ data:
- img_shape
- pad_shape
val_subset:
batch_size: 1
batch_size: 4
transforms:
- type: LoadImageFromFile
- type: Resize
Expand All @@ -57,7 +58,7 @@ data:
- img_shape
- pad_shape
test_subset:
batch_size: 1
batch_size: 4
transforms:
- type: LoadImageFromFile
- type: Resize
Expand Down Expand Up @@ -183,6 +184,8 @@ model:
optimizer:
_target_: torch.optim.SGD
_partial_: true
lr: 0.001
weight_decay: 0.001
lr: 0.004
weight_decay: 0.0001
momentum: 0.9
trainer:
gradient_clip_val: 35.0
9 changes: 6 additions & 3 deletions src/otx/recipe/detection/ssd_mobilenetv2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# @package _global_
defaults:
- override /trainer: default
- override /base: detection
- override /callbacks: detection
- override /data: mmdet
Expand Down Expand Up @@ -45,7 +46,7 @@ data:
- img_shape
- pad_shape
val_subset:
batch_size: 1
batch_size: 8
transforms:
- type: LoadImageFromFile
- type: Resize
Expand All @@ -64,7 +65,7 @@ data:
- img_shape
- pad_shape
test_subset:
batch_size: 1
batch_size: 8
transforms:
- type: LoadImageFromFile
- type: Resize
Expand Down Expand Up @@ -186,5 +187,7 @@ model:
_target_: torch.optim.SGD
_partial_: true
lr: 0.01
weight_decay: 0.001
weight_decay: 0.0001
momentum: 0.9
trainer:
gradient_clip_val: 35.0
27 changes: 18 additions & 9 deletions src/otx/recipe/detection/yolox_l.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# @package _global_
defaults:
- override /trainer: default
- override /base: detection
- override /callbacks: detection
- override /data: mmdet
Expand All @@ -20,8 +21,8 @@ data:
- 640
- type: RandomAffine
scaling_ratio_range:
- 0.5
- 1.5
- 0.1
- 2.0
border:
- -320
- -320
Expand All @@ -44,7 +45,7 @@ data:
scale:
- 640
- 640
keep_ratio: false
keep_ratio: true
- type: RandomFlip
prob: 0.5
- type: Pad
Expand All @@ -63,14 +64,17 @@ data:
- img_shape
- pad_shape
val_subset:
batch_size: 1
batch_size: 8
transforms:
- type: LoadImageFromFile
- type: Resize
scale:
- 640
- 640
keep_ratio: false
keep_ratio: True
- type: Pad
pad_to_square: true
pad_val: 114
- type: LoadAnnotations
with_bbox: true
- type: PackDetInputs
Expand All @@ -82,14 +86,17 @@ data:
- img_shape
- pad_shape
test_subset:
batch_size: 1
batch_size: 8
transforms:
- type: LoadImageFromFile
- type: Resize
scale:
- 640
- 640
keep_ratio: false
keep_ratio: True
- type: Pad
pad_to_square: true
pad_val: 114
- type: LoadAnnotations
with_bbox: true
- type: PackDetInputs
Expand Down Expand Up @@ -151,6 +158,8 @@ model:
optimizer:
_target_: torch.optim.SGD
_partial_: true
lr: 0.0005
weight_decay: 0.001
lr: 0.001
weight_decay: 0.0001
momentum: 0.9
trainer:
gradient_clip_val: 35.0
27 changes: 18 additions & 9 deletions src/otx/recipe/detection/yolox_s.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# @package _global_
defaults:
- override /trainer: default
- override /base: detection
- override /callbacks: detection
- override /data: mmdet
Expand All @@ -20,8 +21,8 @@ data:
- 640
- type: RandomAffine
scaling_ratio_range:
- 0.5
- 1.5
- 0.1
- 2.0
border:
- -320
- -320
Expand All @@ -44,7 +45,7 @@ data:
scale:
- 640
- 640
keep_ratio: false
keep_ratio: True
- type: RandomFlip
prob: 0.5
- type: Pad
Expand All @@ -63,14 +64,17 @@ data:
- img_shape
- pad_shape
val_subset:
batch_size: 1
batch_size: 8
transforms:
- type: LoadImageFromFile
- type: Resize
scale:
- 640
- 640
keep_ratio: false
keep_ratio: True
- type: Pad
pad_to_square: true
pad_val: 114
- type: LoadAnnotations
with_bbox: true
- type: PackDetInputs
Expand All @@ -82,14 +86,17 @@ data:
- img_shape
- pad_shape
test_subset:
batch_size: 1
batch_size: 8
transforms:
- type: LoadImageFromFile
- type: Resize
scale:
- 640
- 640
keep_ratio: false
keep_ratio: True
- type: Pad
pad_to_square: true
pad_val: 114
- type: LoadAnnotations
with_bbox: true
- type: PackDetInputs
Expand Down Expand Up @@ -151,6 +158,8 @@ model:
optimizer:
_target_: torch.optim.SGD
_partial_: true
lr: 0.0005
weight_decay: 0.001
lr: 0.001
weight_decay: 0.0001
momentum: 0.9
trainer:
gradient_clip_val: 35.0
21 changes: 15 additions & 6 deletions src/otx/recipe/detection/yolox_tiny.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,6 @@
# @package _global_
defaults:
- override /trainer: default
- override /base: detection
- override /callbacks: detection
- override /data: mmdet
Expand Down Expand Up @@ -38,7 +39,7 @@ data:
scale:
- 640
- 640
keep_ratio: false
keep_ratio: True
- type: RandomFlip
prob: 0.5
- type: Pad
Expand All @@ -57,14 +58,17 @@ data:
- img_shape
- pad_shape
val_subset:
batch_size: 1
batch_size: 8
transforms:
- type: LoadImageFromFile
- type: Resize
scale:
- 416
- 416
keep_ratio: false
keep_ratio: True
- type: Pad
pad_to_square: true
pad_val: 114
- type: LoadAnnotations
with_bbox: true
- type: PackDetInputs
Expand All @@ -76,14 +80,17 @@ data:
- img_shape
- pad_shape
test_subset:
batch_size: 1
batch_size: 8
transforms:
- type: LoadImageFromFile
- type: Resize
scale:
- 416
- 416
keep_ratio: false
keep_ratio: True
- type: Pad
pad_to_square: true
pad_val: 114
- type: LoadAnnotations
with_bbox: true
- type: PackDetInputs
Expand Down Expand Up @@ -146,5 +153,7 @@ model:
_target_: torch.optim.SGD
_partial_: true
lr: 0.0002
weight_decay: 0.001
weight_decay: 0.0001
momentum: 0.9
trainer:
gradient_clip_val: 35.0
Loading