Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[<Ray component: util>] PlacementGroup bundle index not effect when num_gpus<1 #48685

Closed
Hanqer opened this issue Nov 11, 2024 · 1 comment
Closed
Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core P1 Issue that should be fixed within a few weeks

Comments

@Hanqer
Copy link

Hanqer commented Nov 11, 2024

What happened + What you expected to happen

import ray 
import os 

from ray.util.placement_group import PlacementGroup, PlacementGroupSchedulingStrategy, placement_group

ray.init() 

@ray.remote(num_gpus=1)
class Test():
    def __init__(self, id):
        self.id = id 

    def test(self):
        print(f"Actor ID: {self.id}, CUDA_VISIBLE_DEVICES: {os.environ['CUDA_VISIBLE_DEVICES']}")

if __name__ == "__main__":
    pg = placement_group([{"GPU": 1, "CPU": 1} for _ in range(4)])
    ray.get(pg.ready())

    actors = [
        Test.options(
            num_gpus=0.5,
            num_cpus=0.5,
            scheduling_strategy=PlacementGroupSchedulingStrategy(
                placement_group=pg, placement_group_bundle_index=i
            ),
        ).remote(i)
        for i in range(4)]

    for actor in actors:
        ray.get(actor.test.remote())

I think this should output the cuda_devices: 0 1 2 3, since each actor requires 0.5 gpus, and each bundle has 1 gpu resouces. Total four actors are evenly require bundle with placement_group_bundle_index of bundle list with four items. I holp this code place each actor on 0.5 gpus but with different 4 devieces. However, this result in two actors on gpu devices 0, and two actors on gpu devices 1. I want to knwo why, thx!

Versions / Dependencies

ray==2.12.0

Reproduction script

import ray 
import os 

from ray.util.placement_group import PlacementGroup, PlacementGroupSchedulingStrategy, placement_group

ray.init() 

@ray.remote(num_gpus=1)
class Test():
    def __init__(self, id):
        self.id = id 

    def test(self):
        print(f"Actor ID: {self.id}, CUDA_VISIBLE_DEVICES: {os.environ['CUDA_VISIBLE_DEVICES']}")

if __name__ == "__main__":
    pg = placement_group([{"GPU": 1, "CPU": 1} for _ in range(4)])
    ray.get(pg.ready())

    actors = [
        Test.options(
            num_gpus=0.5,
            num_cpus=0.5,
            scheduling_strategy=PlacementGroupSchedulingStrategy(
                placement_group=pg, placement_group_bundle_index=i
            ),
        ).remote(i)
        for i in range(4)]

    for actor in actors:
        ray.get(actor.test.remote())

Issue Severity

High: It blocks me from completing my task.

@Hanqer Hanqer added bug Something that is supposed to be working; but isn't triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Nov 11, 2024
@jcotant1 jcotant1 added core Issues that should be addressed in Ray Core and removed core Issues that should be addressed in Ray Core labels Nov 11, 2024
@rynewang
Copy link
Contributor

This has been fixed in master in #48088 (not released yet, aim ray 2.40)

(Test pid=99523) Actor ID: 0, CUDA_VISIBLE_DEVICES: 0
(Test pid=99520) Actor ID: 1, CUDA_VISIBLE_DEVICES: 1
(Test pid=99522) Actor ID: 2, CUDA_VISIBLE_DEVICES: 2
(Test pid=99525) Actor ID: 3, CUDA_VISIBLE_DEVICES: 3

@jjyao jjyao added P1 Issue that should be fixed within a few weeks and removed triage Needs triage (eg: priority, bug/not-bug, and owning component) labels Nov 25, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't core Issues that should be addressed in Ray Core P1 Issue that should be fixed within a few weeks
Projects
None yet
Development

No branches or pull requests

4 participants