[Data] [Release Test] Use worker node instead of head node for read_images_comparison_microbenchmark_single_node
#47228
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
August 20, 2024 14:34
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
scottjlee
commented
Aug 20, 2024
| MOSAIC_DIR="/tmp/mosaicml-data" | ||
| TFRECORDS_DIR="/tmp/tf-data" | ||
| PARQUET_DIR="/tmp/parquet-data" | ||
| DIR="/mnt/cluster_storage/imagenet-1gb" |
There was a problem hiding this comment.
change to cluster_storage since the data needs to be accessed by worker node.
raulchen
approved these changes
Aug 20, 2024
aslonnie
added a commit
that referenced
this pull request
Aug 21, 2024
…mages_comparison_microbenchmark_single_node cherrypick #47228 Signed-off-by: Lonnie Liu <lonnie@anyscale.com>
aslonnie
added a commit
that referenced
this pull request
Aug 21, 2024
for read_images_comparison_microbenchmark_single_node cherrypick #47228 Signed-off-by: Lonnie Liu <lonnie@anyscale.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
Recently, there was a change to the OOM killing threshold, being reduced from 95% to 75% for head nodes. We suspect this to be the root cause of
read_images_comparison_microbenchmark_single_noderelease test failures, which uses a single head node. To mitigate the issue, this PR changes the compute config of the test, to instead use a 0-CPU head node, and a single worker node.Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.