Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

group_files does not support FSFile objects (TypeError: unhasshable type 'FSFile') #1604

Closed
gerritholl opened this issue Mar 19, 2021 · 0 comments · Fixed by #1606
Closed
Assignees
Labels

Comments

@gerritholl
Copy link
Member

Feature Request

Is your feature request related to a problem? Please describe.

When I pass FSFile objects to Multiscene.from_files, satpy.readers.group_files fails with ValueError:

import satpy
from fsspec.implementations.local import LocalFileSystem

fns = ['/media/nas/x21308/abi/ABI-L1B-RadC/2019/023/18/OR_ABI-L1b-RadC-M3C01_G16_s20190231812167_e20190231814540_c20190231814581.nc',
  '/media/nas/x21308/abi/ABI-L1B-RadC/2019/023/18/OR_ABI-L1b-RadC-M3C01_G16_s20190231822167_e20190231824540_c20190231824584.nc',
  '/media/nas/x21308/abi/ABI-L1B-RadC/2019/023/18/OR_ABI-L1b-RadC-M3C01_G16_s20190231832167_e20190231834540_c20190231834583.nc']

lfs = LocalFileSystem()

fsfs = [satpy.readers.FSFile(fn, fs=lfs) for fn in fns]

satpy.readers.group_files(fsfs)

fails with:

Traceback (most recent call last):
  File "/home/gholl/checkouts/protocode/mwe/typeerror-fsfile.py", line 12, in <module>
    satpy.readers.group_files(fsfs)
  File "/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/satpy/readers/__init__.py", line 90, in group_files
    reader_files = _assign_files_to_readers(
  File "/data/gholl/miniconda3/envs/py39/lib/python3.9/site-packages/satpy/readers/__init__.py", line 122, in _assign_files_to_readers
    files_to_sort = set(files_to_sort)
TypeError: unhashable type: 'FSFile'

Since group_files is used by MultiScene.from_files, the same exception occurs when trying to pass FSFile objects to Multiscene.from_files.

Describe the solution you'd like

I would like to be able to pass FSFile objects to MultiScene.from_files.

The most reasonable implementation would probably to make FSFile objects hashable. Since both filenames and filesystems are hashable, this should not be a problem.

Describe any changes to existing user workflow

No backward compatibility problems foreseen.

Additional context

I'm not sure if I could solve this differently.

@gerritholl gerritholl self-assigned this Mar 19, 2021
@gerritholl gerritholl added the bug label Mar 19, 2021
gerritholl added a commit to gerritholl/satpy that referenced this issue Mar 19, 2021
FSFile objects had been inadvertently made unhashable by
pytroll#1582 .  Make FSFile objects
hashable again by implementing a __hash__ method.

- [x] Fixes pytroll#1604
- [x] Fixes pytroll#1605
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant