[core][release] [unstable] chaos_dataset_shuffle_push_based_random_shuffle_1tb failed with OutOfMemoryError

[rickyyx]

What happened + What you expected to happen

chaos_dataset_shuffle_push_based_random_shuffle_1tb

ray.exceptions.OutOfMemoryError: Task was killed due to the node running low on memory.

Memory on the node (IP: 172.31.69.247, ID: 5bf8d24f68a4b2f966b34cb94a41d1a312f406dc9081063ea7dd661b) where the task was running was 54.86GB / 57.60GB (0.952354), which exceeds the memory usage threshold of 0.95. Ray killed this worker (ID: 0c7d1ce94eefa4e408894a6d04511faca3d30bad46c775ba7961b38b) because it was the most recently scheduled task; to see more information about memory usage on this node, use `ray logs raylet.out -ip 172.31.69.247`. To see the logs of the worker, use `ray logs worker-0c7d1ce94eefa4e408894a6d04511faca3d30bad46c775ba7961b38b*out -ip 172.31.69.247`.

Consider provisioning more memory on this node or reducing task parallelism by requesting more CPUs per task. To adjust the eviction threshold, set the environment variable `RAY_memory_usage_threshold_fraction` when starting Ray. To disable worker eviction, set the environment variable `RAY_memory_monitor_interval_ms` to zero.

Versions / Dependencies

Last Success: 45d7cd2
First Failure: fb7472f

fb7472f Remove RAY_RAYLET_NODE_ID (#28715)
93f911e Add API latency and call counts metrics to dashboard APIs (#28279)
66aae4c [Release Test] Make sure to delete all EBS volumes (#28707)
697df80 [Serve] [Docs] Remove incorrect output (#28708)
d8c9aa7 [docs] configurable ecosystem gallery (#28662)
42874e1 [RLlib] Atari gym environments now require ale-py. (#28703)
b7f0346 [AIR] Maintain dtype info in LightGBMPredictor (#28673)
f6ae7ee [tune] Test background syncer serialization (#28699)
87f22e1 [ci] Requirements contains duplicate of 'starlette' (#28698)
2e7040e [Tune] [PBT] Maintain consistent Trial/TrialRunner state when pausing and resuming trial (#28511)
6530635 [ci] Fix mac pipeline (use python 2 in CI scripts) (#28695)
ee2a8da [ci] Move to new hierarchical docker structure + pipeline (#28641)
a3c97b4 [Doc] Revamp ray core design patterns doc [8/n]: pass large arg by value (#28660)
db2ce69 [Datasets] Add initial aggregate benchmark (#28486)
9c2abf9 [KubeRay][Operator] Improve migration notes (#28672)
45d7cd2 [core] Support generators to allow tasks to return a dynamic number of objects (#28291)

Reproduction script

NA

Issue Severity

No response

[rickyyx]

Unstable test so P1.

[rickyyx]

Solved