Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change target ci-runner to gh-hosted runner for unittests #2847

Merged
merged 8 commits into from
Jan 30, 2024

Conversation

yunchu
Copy link
Contributor

@yunchu yunchu commented Jan 29, 2024

Summary

To distribute CI workloads across all available resources, unit test for the v2 would be targeted to run on the GH-hosted runner.
For the integration tests, it required larger GPU memory (>15G) so it would keep using the dedicated ci-runner (3090) and it will be re-targeted to the runner on AWS whenever there are the GPU instances that have suitable memory size.

How to test

Checklist

  • I have added unit tests to cover my changes.​
  • I have added integration tests to cover my changes.​
  • I have added e2e tests for validation.
  • I have added the description of my changes into CHANGELOG in my target branch (e.g., CHANGELOG in develop).​
  • I have updated the documentation in my target branch accordingly (e.g., documentation in develop).
  • I have linked related issues.

License

  • I submit my code changes under the same Apache License that covers the project.
    Feel free to contact the maintainers if that's a concern.
  • I have updated the license header for each file (see an example below).
# Copyright (C) 2023 Intel Corporation
# SPDX-License-Identifier: Apache-2.0

@github-actions github-actions bot added the BUILD label Jan 29, 2024
@yunchu yunchu force-pushed the infra/v2/tests-on-aws branch from 0a3ccc7 to 470216f Compare January 29, 2024 11:15
@yunchu yunchu force-pushed the infra/v2/tests-on-aws branch from 7c09060 to 7b1418d Compare January 30, 2024 01:43
harimkang
harimkang previously approved these changes Jan 30, 2024
sungmanc
sungmanc previously approved these changes Jan 30, 2024
Copy link
Contributor

@vinnamkim vinnamkim left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

AFAIK, there is no unit test which needs a GPU device and I think there will not be as well in the future. So, do we need to run our unit test on the T4 node?

@yunchu
Copy link
Contributor Author

yunchu commented Jan 30, 2024

AFAIK, there is no unit test which needs a GPU device and I think there will not be as well in the future. So, do we need to run our unit test on the T4 node?

If you can confirm that all unit tests are not required GPUs, it would be targeted to the Github hosted runner.

@yunchu yunchu dismissed stale reviews from sungmanc and harimkang via 3fda7de January 30, 2024 05:41
@vinnamkim
Copy link
Contributor

vinnamkim commented Jan 30, 2024

AFAIK, there is no unit test which needs a GPU device and I think there will not be as well in the future. So, do we need to run our unit test on the T4 node?

If you can confirm that all unit tests are not required GPUs, it would be targeted to the Github hosted runner.

======================================================================== short test summary info ========================================================================
FAILED tests/unit/algo/segmentation/heads/test_class_incremental_mixin.py::TestClassIncrementalMixin::test_ignore_label - RuntimeError: Attempting to deserialize object on a CUDA device but torch.cuda.is_available() is False. If you are running on a CPU-only machine, please use torch.l...
FAILED tests/unit/algo/visual_prompting/encoders/test_sam_prompt_encoder.py::TestSAMPromptEncoder::test_get_device[cuda] - RuntimeError: No CUDA GPUs are available
======================================================== 2 failed, 370 passed, 1 skipped, 53 warnings in 38.90s =========================================================
(otx-v2) vinnamki@vinnamki:~/otx/training_extensions$ 

I found two cases. I'll open a PR for them. I'll stack commits on this PR.

Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@github-actions github-actions bot added TEST Any changes in tests OTX 2.0 labels Jan 30, 2024
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
Signed-off-by: Kim, Vinnam <vinnam.kim@intel.com>
@yunchu yunchu changed the title Change target ci-runner to T4 node on AWS Change target ci-runner to gh-hosted runner for unittests Jan 30, 2024
.github/workflows/pre_merge.yaml Show resolved Hide resolved
@yunchu yunchu merged commit e65a1ce into v2 Jan 30, 2024
8 checks passed
@yunchu yunchu deleted the infra/v2/tests-on-aws branch January 30, 2024 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
BUILD TEST Any changes in tests
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants