Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[data] ObjectRefs passed to map UDF are not automatically deref'ed #49207

Open
raulchen opened this issue Dec 11, 2024 · 0 comments
Open

[data] ObjectRefs passed to map UDF are not automatically deref'ed #49207

raulchen opened this issue Dec 11, 2024 · 0 comments
Labels
bug Something that is supposed to be working; but isn't data Ray Data-related issues P1 Issue that should be fixed within a few weeks

Comments

@raulchen
Copy link
Contributor

When users pass an ObjectRef as a UDF parameter. It's supposed to be automatically deref'ed

def map_fn(arg):
  # The following is broken right now.
  #  assert arg == 1
  # Instead, users need to use
  arg = ray.get(arg)

arg_ref = ray.put(1)
ds.map(map_fn, fn_args=[arg_ref])

This bug hurts both user experience and performance.
Because the UDF arguments are not passed as the Ray task arguments.
So Ray Scheduler doesn't know if the arguments are ready or not, before scheduling the tasks.

@raulchen raulchen added bug Something that is supposed to be working; but isn't P1 Issue that should be fixed within a few weeks data Ray Data-related issues labels Dec 11, 2024
raulchen added a commit that referenced this issue Dec 11, 2024
## Why are these changes needed?

This PR enables passing kwargs to map tasks, which will be accessible
via `TaskContext.kwargs`.

This is a prerequisite to fixing
#49207. And optimization rules
can use this API to pass additional arguments to the map tasks.

---------

Signed-off-by: Hao Chen <chenh1024@gmail.com>
simonsays1980 pushed a commit to simonsays1980/ray that referenced this issue Dec 12, 2024
## Why are these changes needed?

This PR enables passing kwargs to map tasks, which will be accessible
via `TaskContext.kwargs`.

This is a prerequisite to fixing
ray-project#49207. And optimization rules
can use this API to pass additional arguments to the map tasks.

---------

Signed-off-by: Hao Chen <chenh1024@gmail.com>
ujjawal-khare pushed a commit to ujjawal-khare-27/ray that referenced this issue Dec 17, 2024
## Why are these changes needed?

This PR enables passing kwargs to map tasks, which will be accessible
via `TaskContext.kwargs`.

This is a prerequisite to fixing
ray-project#49207. And optimization rules
can use this API to pass additional arguments to the map tasks.

---------

Signed-off-by: Hao Chen <chenh1024@gmail.com>
Signed-off-by: ujjawal-khare <ujjawal.khare@dream11.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something that is supposed to be working; but isn't data Ray Data-related issues P1 Issue that should be fixed within a few weeks
Projects
None yet
Development

No branches or pull requests

1 participant