Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[FEA] Memory Budget Execution for Stateful Modifying Expressions #7261

Open
revans2 opened this issue Dec 5, 2022 · 0 comments
Open

[FEA] Memory Budget Execution for Stateful Modifying Expressions #7261

revans2 opened this issue Dec 5, 2022 · 0 comments
Labels
feature request New feature or request reliability Features to improve reliability or bugs that severly impact the reliability of the plugin

Comments

@revans2
Copy link
Collaborator

revans2 commented Dec 5, 2022

Is your feature request related to a problem? Please describe.
After #7260 finishes we need to have a good way to deal with Expressions that need to keep state or modify the input to their children.

For these we need a way for the expression to call back into the execution framework to execute one or more of their children with a new batch. We might need to pass in an instance of a class or interface to allow for this. We also need help in splitting the data. In cases where state is maintained in the task we will not be able to split on an input row boundary because we might be in the middle of an array for a higher order function, or in the case of if/else the rows on one batch may not coincide with the rows for another batch, or we might want to keep multiple batches split in the same way. This is where it might be best to just split the input to this sub-tree and redo computation. But that is going to require us to inform the Expression to drop all of its state. I don’t like this as a long term solution, but it should be rare and finding a way to handle the handoffs makes it really difficult to do.

@revans2 revans2 added feature request New feature or request ? - Needs Triage Need team to review and classify labels Dec 5, 2022
@sameerz sameerz added the reliability Features to improve reliability or bugs that severly impact the reliability of the plugin label Dec 6, 2022
@mattahrens mattahrens changed the title [FEA] Memory Budget Execution for Staeful Modifying Expressions [FEA] Memory Budget Execution for Stateful Modifying Expressions Dec 8, 2022
@mattahrens mattahrens removed the ? - Needs Triage Need team to review and classify label Dec 8, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
feature request New feature or request reliability Features to improve reliability or bugs that severly impact the reliability of the plugin
Projects
None yet
Development

No branches or pull requests

3 participants