[Datasets] CSV reading of unspecified extension is not robust

[holdenk]

What happened + What you expected to happen

Trying to load a CSV file fails because the extension does not match.

fs = fsspec.filesystem('https')
ds = ray.data.read_csv(
    "https://https://gender-pay-gap.service.gov.uk/viewing/download-data/2021",
    filesystem=fs)

Results in

(_prepare_read pid=2326326) Filter: FileExtensionFilter(extensions=['.csv'], allow_if_no_extensions=False)

There is no easy way to override this that I can see.

Versions / Dependencies

ray @ https://s3-us-west-2.amazonaws.com/ray-wheels/latest/ray-3.0.0.dev0-cp38-cp38-manylinux2014_x86_64.whl
Python 3.8.5

Reproduction script

fs = fsspec.filesystem('https')
ds = ray.data.read_csv(
    "https://https://gender-pay-gap.service.gov.uk/viewing/download-data/2021",
    filesystem=fs)

Issue Severity

High: It blocks me from completing my task.

[clarkzinzow]

@holdenk Have you tried setting partition_filter=None in the read_csv call?

fs = fsspec.filesystem('https')
ds = ray.data.read_csv(
    "https://https://gender-pay-gap.service.gov.uk/viewing/download-data/2021",
    filesystem=fs,
    partition_filter=None)

Docs: https://docs.ray.io/en/master/data/package-ref.html#ray.data.read_csv

[pcmoritz]

^ The above seems to work but is pretty counter-intuitive / not clear how to find this flag for a user -- @clarkzinzow @matthewdeng @c21 @jianoaix are there things we can do here to make the UX less painful?

[jianoaix]

We can probably improve the documentation, and include the examples of how to read target subset of files, to make the API usage more clear.

[holdenk]

That does help. I think, from my perspective, even just making the naming consistent, e.g. I looked how to set a FileExtensionFilter after I ran into this error, but the CSV reader calls the file extension filter a partition_filter (which... doesn't make sense to me? it's not filtering partitions).

[c21]

I think we should improve the error message here. A better error message should look like

Did not find any available files to read from input directory or paths.
Please double check the input files are having expected extensions: [.csv], or set `partition_filter` field to None to disable file filtering.

A little bit more background: we have two types of partition filter: (1).Hive partition, (2).file directory. So I agree the naming is kind of confusing, but it's really just filtering out files based on file extension in directory.

Additional question: @holdenk - curious what's the file extension in your case for thoses CSV files? Is it no file extension? nvm, tried your link in example, it's a CSV file without extension. thought it was a directory.

[thatcort]

I don't think filtering files based on an assumed file extension is OK. For example, if the file is tab-delimited, will it look for a .tsv file instead of .csv? Best to just let the user specify the file name, no?