Skip to content
This repository has been archived by the owner on Mar 19, 2024. It is now read-only.

Add registry for the model heads #75

Closed
wants to merge 18 commits into from
Closed
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
18 commits
Select commit Hold shift + click to select a range
af844b3
Experiment with Blur data augmentation in swav training
prigoyal Aug 10, 2020
33628bc
Add cosine warm restart scheduler for full wave and half wave
prigoyal Aug 10, 2020
bb3ace8
Close stream and file loggers at the end of training
prigoyal Aug 10, 2020
5899157
Rewrite attrdict to have getattr and setattr for seamless attribute a…
prigoyal Aug 10, 2020
bbd8bcd
Rename engine to engines to make it clearer that there are multiple
prigoyal Aug 10, 2020
29af593
Rename distributed_train to run_distributed_engines for clarity
prigoyal Aug 10, 2020
8eac7db
Tensor input to loss functions to they follow the pytorch definitions
prigoyal Aug 10, 2020
f3bcae3
Make trainer as the main entry point and refactor task, trainer, step…
prigoyal Aug 10, 2020
7522060
Add a registry for the train steps in vissl
prigoyal Aug 10, 2020
3ac425b
Update model zoo and add more crops settings for swav trainings
prigoyal Aug 10, 2020
9bd7efe
Add a registry for data collators in vissl
prigoyal Aug 10, 2020
93b5c88
Rename PARAMS_FILE to WEIGHTS_INIT and PATH to PARAMS_FILE
prigoyal Aug 10, 2020
ba52eab
Rename ssl_hooks to hooks simply
prigoyal Aug 10, 2020
49b37bc
Rename ssl_losses to losses simply
prigoyal Aug 10, 2020
0bc77ba
Rename ssl_optimizers to optimizers for simplicity
prigoyal Aug 10, 2020
42dad13
Hard fail if specified model weights_init file does not exist
prigoyal Aug 10, 2020
9b005d4
Rename accuracy list meter parameters and also print meter names in c…
prigoyal Aug 10, 2020
8c7271e
Add registry for the model heads (#75)
prigoyal Aug 10, 2020
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add cosine warm restart scheduler for full wave and half wave
Differential Revision: D22856092

fbshipit-source-id: af1287ac6b87a97ac26773056bbf164f1ca7705b
  • Loading branch information
prigoyal authored and facebook-github-bot committed Aug 10, 2020
commit 33628bce3f9d1010979917824345910d564466b4
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# @package _global_
config:
OPTIMIZER:
num_epochs: 100
param_schedulers:
lr:
auto_lr_scaling:
auto_scale: false
base_value: 0.3
base_lr_batch_size: 256
name: composite
schedulers:
- name: linear
start_value: 0.6
end_value: 4.8
- name: cosine_warm_restart
start_value: 4.8
end_value: 0.0048
wave_type: full
restart_interval_length: 0.334 # half wave after the warmup is done
update_interval: step
interval_scaling: [rescaled, rescaled]
lengths: [0.1, 0.9] # 100ep
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# @package _global_
config:
OPTIMIZER:
num_epochs: 100
param_schedulers:
lr:
auto_lr_scaling:
auto_scale: true
base_value: 0.3
base_lr_batch_size: 256
name: composite
schedulers:
- name: linear
start_value: 0.6
end_value: 4.8
- name: cosine_warm_restart
start_value: 4.8
end_value: 0.0048
restart_interval_length: 0.334 # half wave after the warmup is done
wave_type: full
is_adaptive: True
update_interval: step
interval_scaling: [rescaled, rescaled]
lengths: [0.1, 0.9] # 100ep
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
# @package _global_
config:
OPTIMIZER:
num_epochs: 100
param_schedulers:
lr:
auto_lr_scaling:
auto_scale: true
base_value: 0.3
base_lr_batch_size: 256
name: composite
schedulers:
- name: linear
start_value: 0.6
end_value: 4.8
- name: cosine_warm_restart
start_value: 4.8
end_value: 0.0048
wave_type: half
restart_interval_length: 0.5 # half wave after the warmup is done
update_interval: step
interval_scaling: [rescaled, rescaled]
lengths: [0.1, 0.9] # 100ep
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# @package _global_
config:
OPTIMIZER:
num_epochs: 100
param_schedulers:
lr:
auto_lr_scaling:
auto_scale: true
base_value: 0.3
base_lr_batch_size: 256
name: composite
schedulers:
- name: linear
start_value: 0.6
end_value: 4.8
- name: cosine_warm_restart
start_value: 4.8
end_value: 0.0048
restart_interval_length: 0.5 # half wave after the warmup is done
wave_type: half
is_adaptive: True
update_interval: step
interval_scaling: [rescaled, rescaled]
lengths: [0.1, 0.9] # 100ep
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# @package _global_
config:
OPTIMIZER:
num_epochs: 100
param_schedulers:
lr:
auto_lr_scaling:
auto_scale: false
base_value: 0.3
base_lr_batch_size: 256
name: composite
schedulers:
- name: linear
start_value: 0.6
end_value: 4.8
- name: cosine
start_value: 4.8
end_value: 0.0048
- name: linear
start_value: 0.0048
end_value: 2.114
- name: cosine
start_value: 2.114
end_value: 0.0048
update_interval: step
interval_scaling: [rescaled, rescaled, rescaled, rescaled]
lengths: [0.1, 0.45, 0.1, 0.35] # 100ep
20 changes: 10 additions & 10 deletions configs/config/test/integration_test/quick_simclr.yaml
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
# @package _global_
config:
VERBOSE: False
LOG_FREQUENCY: 2
LOG_FREQUENCY: 1
TEST_ONLY: False
TEST_MODEL: False
SEED_VALUE: 0
Expand Down Expand Up @@ -80,28 +80,28 @@ config:
param_schedulers:
lr:
auto_lr_scaling:
auto_scale: true
auto_scale: false
base_value: 0.3
base_lr_batch_size: 256
# name: cosine
# start_value: 0.15 # LR for batch size 256
# end_value: 0.0000
# update_interval: step
name: composite
schedulers:
- name: linear
start_value: 0.6
end_value: 4.8
- name: cosine
- name: cosine_warm_restart
start_value: 4.8
end_value: 0.0000
end_value: 0.0048
# wave_type: half
# restart_interval_length: 0.5
wave_type: full
is_adaptive: True
restart_interval_length: 0.334
interval_scaling: [rescaled, rescaled]
update_interval: step
interval_scaling: [rescaled, fixed]
lengths: [0.1, 0.9] # 100ep
DISTRIBUTED:
BACKEND: nccl
NUM_NODES: 1
# NUM_PROC_PER_NODE: 2
NUM_PROC_PER_NODE: 1
INIT_METHOD: tcp
RUN_ID: auto
Expand Down
88 changes: 88 additions & 0 deletions vissl/ssl_optimizers/param_scheduler/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,88 @@
## How to setup different Learning rate schedule

Below we provide some examples of how to setup various types of Learning rate schedules. Note that these are merely some examples and you should set your desired parameter values.

1. Cosine

```yaml
name: cosine
start_value: 0.15 # LR for batch size 256
end_value: 0.0000
```

2. Multi-Step

```yaml
name: multistep
values: [0.01, 0.001]
milestones: [1]
update_interval: epoch
```

3. Linear Warmup + Cosine

```yaml
name: composite
schedulers:
- name: linear
start_value: 0.6
end_value: 4.8
- name: cosine
start_value: 4.8
end_value: 0.0048
interval_scaling: [rescaled, fixed]
update_interval: step
lengths: [0.1, 0.9] # 100ep
```

4. Cosine with restarts

```yaml
name: cosine_warm_restart
start_value: 0.15 # LR for batch size 256
end_value: 0.00015
# restart_interval_length: 0.334
# wave_type: full
restart_interval_length: 0.5
wave_type: half
```

5. Linear warmup + cosine with restarts

```yaml
name: composite
schedulers:
- name: linear
start_value: 0.6
end_value: 4.8
- name: cosine_warm_restart
start_value: 4.8
end_value: 0.0048
# wave_type: half
# restart_interval_length: 0.5
wave_type: full
restart_interval_length: 0.334
interval_scaling: [rescaled, rescaled]
update_interval: step
lengths: [0.1, 0.9] # 100ep
```

6. Multiple linear warmups and cosine
```yaml
schedulers:
- name: linear
start_value: 0.6
end_value: 4.8
- name: cosine
start_value: 4.8
end_value: 0.0048
- name: linear
start_value: 0.0048
end_value: 2.114
- name: cosine
start_value: 2.114
end_value: 0.0048
update_interval: step
interval_scaling: [rescaled, rescaled, rescaled, rescaled]
lengths: [0.0256, 0.48722, 0.0256, 0.46166] # 1ep IG-500M
```
11 changes: 11 additions & 0 deletions vissl/ssl_optimizers/param_scheduler/__init__.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
# Copyright (c) Facebook, Inc. and its affiliates. All Rights Reserved

from pathlib import Path

from classy_vision.generic.registry_utils import import_all_modules


FILE_ROOT = Path(__file__).parent

# automatically import any Python files in the param_scheduler/ directory
import_all_modules(FILE_ROOT, "vissl.ssl_optimizers.param_scheduler")
Loading