Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Data] Add environment variable support for Ray Data execution callbacks. #51449

Open
wants to merge 1 commit into
base: master
Choose a base branch
from

Conversation

JDarDagran
Copy link
Contributor

Why are these changes needed?

This change adds support for loading Ray Data execution callbacks from an environment variable (RAY_DATA_EXECUTION_CALLBACKS). Users can now specify a comma-separated list of callback class paths, which will be dynamically imported and added to the DataContext.
This allows for easy instrumentation and monitoring of Ray Data executions without modifying application code, which is useful for deb ugging and observability in production environments.

Checks

  • I've signed off every commit(by using the -s flag, i.e., git commit -s) in this PR.
  • I've run scripts/format.sh to lint the changes in this PR.
  • I've included any doc changes needed for https://docs.ray.io/en/master/.
    • I've added any new APIs to the API Reference. For example, if I added a
      method in Tune, I've added it in doc/source/tune/api/ under the
      corresponding .rst file.
  • I've made sure the tests are passing. Note that there might be a few flaky tests, see the recent failures at https://flakey-tests.ray.io/
  • Testing Strategy
    • Unit tests
    • Release tests
    • This PR is not tested :(

@JDarDagran JDarDagran requested a review from a team as a code owner March 18, 2025 14:18
@JDarDagran JDarDagran force-pushed the data/add-execution-callback-from-env-var branch from 7f64748 to f67abdf Compare March 18, 2025 14:21
This change adds support for loading Ray Data execution callbacks from an
environment variable (`RAY_DATA_EXECUTION_CALLBACKS`). Users can now specify
a comma-separated list of callback class paths, which will be dynamically
imported and added to the DataContext.

This allows for easy instrumentation and monitoring of Ray Data executions
without modifying application code, which is useful for debugging and
observability in production environments.

Signed-off-by: Jakub Dardzinski <kuba0221@gmail.com>
@JDarDagran JDarDagran force-pushed the data/add-execution-callback-from-env-var branch from f67abdf to eded6dd Compare March 18, 2025 14:36
@jcotant1 jcotant1 added the data Ray Data-related issues label Mar 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
data Ray Data-related issues
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants