[data] [streaming] Support a streaming_repartition() operator

[ericl]

In several use cases, it is useful to change the block size of datasets in a streaming way. The current repartition() operator is an all-to-all operator and is incompatible with streaming.

We could implement a general purpose streaming_repartition() operator that supports repartitioning in a few streaming-compatible ways:

• Splitting/coalescing blocks into a certain number of rows
• Splitting/coalescing blocks into a certain in-memory byte size
• Splitting/coalescing blocks into K pieces

This could be implemented as a new PhysicalOperator that implements the online repartitioning. This could also replace the current SplitBlocks mechanism from #36352

[luxunxiansheng]

Suppose I have 20 big size files and I implement a specfic datasource for it. I would like to load the datset by read_datasource with a parallesim ,say , 200. Now I see the splitblocks function to split each block to many smaller blocks. My question is , how does the splitblocks work? It will split a single big file in each row into many many binary parts and then to coalesce them somewhere in the downstream?

[ericl]

SplitBlocks works within the read task to split the read output into multiple smaller pieces. These will remain as smaller individual blocks for the remainder of the computation unless the dataset is explicitly repartitioned.

Ray Data will automatically insert SplitBlocks to ensure the desired/autodetected parallelism is met after a read.

[anrooo]

Also been looking for this feature. We have pipelines where processing single rows can take many minutes where we need greater control over block size at various parts of our pipelines.

[drcege]

@srinathk10 Why is this closed; does #50179 provide a true streaming split? I mean, without the need to materialize the whole dataset.

Because the documentation still says this operation requires all inputs to be materialized in object store for it to execute, could you please clarify what the exact meaning of all inputs here refers to: the entire dataset or only the input of the current node .

Can I understand that in the case of shuffle=False, this is a streaming split (the whole dataset is not materialized)?

[srinathk10]

https://docs.ray.io/en/latest/data/api/doc/ray.data.Dataset.repartition.html
@drcege Thanks for pointing out inconsistency in the documentation. I will put out a patch for this one.

Yes, #50179 does handle streaming repartition when target_num_rows_per_block is set.