[FEA] Memory Budget Execution for Stateful Modifying Expressions #7261
Labels
feature request
New feature or request
reliability
Features to improve reliability or bugs that severly impact the reliability of the plugin
Is your feature request related to a problem? Please describe.
After #7260 finishes we need to have a good way to deal with Expressions that need to keep state or modify the input to their children.
For these we need a way for the expression to call back into the execution framework to execute one or more of their children with a new batch. We might need to pass in an instance of a class or interface to allow for this. We also need help in splitting the data. In cases where state is maintained in the task we will not be able to split on an input row boundary because we might be in the middle of an array for a higher order function, or in the case of if/else the rows on one batch may not coincide with the rows for another batch, or we might want to keep multiple batches split in the same way. This is where it might be best to just split the input to this sub-tree and redo computation. But that is going to require us to inform the Expression to drop all of its state. I don’t like this as a long term solution, but it should be rare and finding a way to handle the handoffs makes it really difficult to do.
The text was updated successfully, but these errors were encountered: