Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enable to override data configurations #3748

Merged

Conversation

sungchul2
Copy link
Contributor

@sungchul2 sungchul2 commented Jul 17, 2024

Summary

This PR includes:

  • Enable to override data configs in recipes
    • It enables to override data configs:
      • including single config (e.g. batch_size or num_workers )
      • nested namespace (e.g. sampler)
      • list of dicts (e.g. transforms)) like callbacks or logger
      ...
      overrides:
        data:
          ...
          train_subset:
            batch_size: 4 # update batch_size to 4
            transforms:
              # set transform's parameter(s) to be updated
              - class_path: old_transform1
                init_args:
                  scale: # update scale to (320, 320)
                    - 320
                    - 320
              # if no old_transform2, it will be used as is
              - class_path: old_transform3
                init_args:
                  prob: 0.7 # update prob to 0.7
              ...
            sampler:
              class_path: new_sampler # update samper from old one to new_sampler
    • If there is no predefined config in base config, it adds this
    • If you want to reset specific configs (especially transforms), use reset with list or string like below:
      ...
      overrides:
        reset:
          - data.train_subset.transforms
          - data.val_subset.transforms
          - data.test_subset.transforms
        ...
        data:
          ...
          train_subset:
            ...
            transforms:
              # set all new transforms for reinitialization
              - class_path: new_transform1
              - class_path: new_transform2
              - class_path: new_transform3
              ...
          # update val_subset.transforms and test_subset.transforms as well
      ...
      overrides:
        reset: data.train_subset.transforms # only update train_subset.transforms
        ...
        data:
          ...
          train_subset:
            ...
            transforms:
              - class_path: new_transform1
              - class_path: new_transform2
              - class_path: new_transform3
              ...
          # val_subset.transforms and test_subset.transforms will not be reset, but can be updated
  • Update recipes

How to test

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have ran e2e tests and there is no issues.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

@github-actions github-actions bot added TEST Any changes in tests OTX 2.0 labels Jul 17, 2024
@sungchul2 sungchul2 force-pushed the enable-data-config-override branch from bd2854b to 671f615 Compare July 19, 2024 08:45
@github-actions github-actions bot added the DOC Improvements or additions to documentation label Jul 22, 2024
Copy link
Contributor

@harimkang harimkang left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM,
As next PR for this, could you please add a description and example of this new feature to the documentation below?
docs/source/guide/tutorials/advanced/configuration.rst

@sungchul2 sungchul2 enabled auto-merge July 23, 2024 04:07
@sungchul2 sungchul2 added this pull request to the merge queue Jul 23, 2024
Merged via the queue into openvinotoolkit:develop with commit 632f5c6 Jul 23, 2024
18 checks passed
@sungchul2 sungchul2 deleted the enable-data-config-override branch July 23, 2024 04:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
DOC Improvements or additions to documentation TEST Any changes in tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants