[data] Implement Dataset.distinct #36655
Conversation
Signed-off-by: Hao Chen <chenh1024@gmail.com>
Signed-off-by: Hao Chen <chenh1024@gmail.com>
python/ray/data/dataset.py
Outdated
| "`distinct` currently only suports Datasets with one single column, " | ||
| "please apply `select_columns` before `distinct`." | ||
| ) | ||
| return self.groupby(columns[0]).count().drop_columns(["count()"]) |
There was a problem hiding this comment.
shall we add a TODO to implement an aggregate function for distinct, so we don't need to calculate count?
There was a problem hiding this comment.
instead of drop_columns(["count()"]), can we call select_columns(columns[0]), so we don't rely on the implicit naming of count()?
There was a problem hiding this comment.
shall we add a TODO to implement an aggregate function for distinct, so we don't need to calculate count?
I considered this initially. but considering count is already very cheap, this is probably no big benefit to implement a standalone distinct function.
|
Oh one more thing - please update the API reference doc to include this new API, thanks. |
Signed-off-by: Hao Chen <chenh1024@gmail.com>
@c21 I added a new item in |
Signed-off-by: Philipp Moritz <pcmoritz@gmail.com>
Implement `Dataset.distinct`. Currently this API only supports Datasets with one single column. This is because `groupby` doesn't support multiple columns yet. Co-authored-by: Philipp Moritz <pcmoritz@gmail.com> Signed-off-by: e428265 <arvind.chandramouli@lmco.com>
Why are these changes needed?
Implement
Dataset.distinct. Currently this API only supports Datasets with one single column. This is becausegroupbydoesn't support multiple columns yet.Related issue number
Closes #32984
Checks
git commit -s) in this PR.scripts/format.shto lint the changes in this PR.method in Tune, I've added it in
doc/source/tune/api/under thecorresponding
.rstfile.