[Datasets] Support select_columns to select subset of columns

[c21]

Description

Datasets have add_column() and drop_columns() to add or drop columns. But it's not flexible enough when user wants to select a subset of existing columns. We can provide a new select_columns API to do it, and also deprecate existing add/drop_column API.

Use case

Help better UX when user manipulates a subset of columns.

[c21]

In addition, as discussed, lazy-first execution + indexing on columns names would be a great UX boost too, but it's not urgent.

[c21]

Just FYI one more user request in https://ray-distributed.slack.com/archives/C02PHB3SQHH/p1664802732112309 . This should be prioritized.

[chongxiaoc]

anyone has bandwidth in near future can feel free to grab this task.

[heyitsmui]

took a quick look, @c21 for select_columns can we just do something like ds.map_batches(lambda batch: batch.filter(items=[...])) to select the columns if batch_format==pandas.DataFrame and ds.map_batches(lambda batch: batch.select(...)) if batch_format==pyarrow.Table? (and address lazy-first separately)

[jianoaix]

Now this should be implemented even easier with this Block API select(): https://sourcegraph.com/github.com/ray-project/ray@master/-/blob/python/ray/data/block.py?L279

[heyitsmui]

ah so something like map_batches(lambda batch: BlockAccessor.for_block(batch).select(...)) and block api should handle both batch_formats

[heyitsmui]

@jianoaix can you take a look at this draft PR: 11457e8? have some quick questions on there for you as well, thanks!