[Data] Enable execution plan optimizer for supported Ray Data APIs #36294
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 13 commits
June 9, 2023 17:27
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
|
Failing CI and release tests look unrelated to this PR to me. |
raulchen
approved these changes
Jun 21, 2023
python/ray/data/_internal/plan.py
Outdated
| @@ -136,6 +136,8 @@ def __init__( | |||
| # determined by the config at the time it was created. | |||
| self._context = copy.deepcopy(DataContext.get_current()) | |||
|
|
|||
| self._skip_optimizer_pipeline = not self._context.optimizer_enabled | |||
There was a problem hiding this comment.
Nit, what about naming this variable "_generated_from_pipeline"? This will be easier to reason about. Because it only indicates whether this plan is generated from a pipeline, and doesn't nothing to do with the optimizer here.
There was a problem hiding this comment.
Good point, updated.
added 2 commits
June 21, 2023 15:04
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
raulchen
reviewed
Jun 22, 2023
python/ray/data/_internal/plan.py
Outdated
| # Whether the corresponding dataset is generated from a pipeline. | ||
| # Currently, when this is True, this skips the new execution plan optimizer. | ||
| # TODO(scottjlee): remove this once we remove DatasetPipeline. | ||
| self._generated_from_pipeline = not self._context.optimizer_enabled |
There was a problem hiding this comment.
This should be true regardless of the optimizer flag. Because in _get_execution_dag, we will check the optimizer flag.
added 4 commits
June 22, 2023 10:32
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
Signed-off-by: Scott Lee <sjl@anyscale.com>
8 tasks
arvind-chandra
pushed a commit
to lmco/ray
that referenced
this pull request
Aug 31, 2023
…ay-project#36294) This PR further expands support for general use of the execution plan optimizer, including all currently existing Ray Data APIs, with the major exception of `DatasetPipeline`. In the case where a DatasetPipeline is used, the `ExecutionPlan` will include a flag to skip the new optimizer path and fall back to the legacy plan optimizer. In addition, this PR implements several small patches to fully enable the new execution plan optimizer on existing Ray Data APIs, such as data lineage serialization. Signed-off-by: Scott Lee <sjl@anyscale.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Why are these changes needed?
This PR further expands support for general use of the execution plan optimizer, including all currently existing Ray Data APIs, with the major exception of
DatasetPipeline. In the case where a DatasetPipeline is used, theExecutionPlanwill include a flag to skip the new optimizer path and fall back to the legacy plan optimizer.In addition, this PR implements several small patches to fully enable the new execution plan optimizer on existing Ray Data APIs, such as data lineage serialization.
Related issue number
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.