Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Make Perf test available to load pervious Perf test to skip training stage #3556

Merged
merged 8 commits into from
May 30, 2024

Conversation

eunwoosh
Copy link
Contributor

@eunwoosh eunwoosh commented May 29, 2024

Summary

This PR adds new features that Perf test can skip training stage by loading previous Perf test.
If previous perf test result directory is given as an argument,
Perf test can check training w/ same tags (["data_group", "data", "model", "task", "seed"]) was already done, and skip training stage if possible.
This feature can provide two use cases:

  • check inference performance on various environment w/o training
  • resume performance test when it's aborted after training is done

Additionally, this PR includes

  • updating perf test code to measure e2e_time of test stage
  • when making symbolic link for ./latest, sym link refers relative path not absolute path

How to test

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have ran e2e tests and there is no issues.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2024 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

@github-actions github-actions bot added TEST Any changes in tests OTX 2.0 labels May 29, 2024
Copy link
Contributor

@goodsong81 goodsong81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice feature & refactoring!
I have a few minor comments. Feel free to accept or not (except for the typos ;p)

tests/perf/benchmark.py Show resolved Hide resolved
tests/conftest.py Outdated Show resolved Hide resolved
tests/perf/benchmark.py Outdated Show resolved Hide resolved
tests/perf/conftest.py Outdated Show resolved Hide resolved
tests/perf/test_visual_prompting.py Outdated Show resolved Hide resolved
goodsong81
goodsong81 previously approved these changes May 29, 2024
Copy link
Contributor

@goodsong81 goodsong81 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@sovrasov sovrasov merged commit ec0f906 into openvinotoolkit:develop May 30, 2024
15 checks passed
eunwoosh added a commit to eunwoosh/training_extensions that referenced this pull request Jun 5, 2024
…stage (openvinotoolkit#3556)

* symlink to relative path

* skip training if prev perf result exists

* implement missing part

* align with pre-commit

* udpate test code

* fix typo

* change arg name to resume-from

* revert checkpoint symlink
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
TEST Any changes in tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants