Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DataQuery matches against all unmodified DataIDs #1806

Open
djhoese opened this issue Aug 25, 2021 · 4 comments · May be fixed by #1807
Open

DataQuery matches against all unmodified DataIDs #1806

djhoese opened this issue Aug 25, 2021 · 4 comments · May be fixed by #1807
Assignees
Labels
bug component:dep_tree Dependency tree and dataset loading

Comments

@djhoese
Copy link
Member

djhoese commented Aug 25, 2021

Describe the bug
I'm working on the nucaps reader and am getting exceptions of "TooManyResults". This happens when the DependencyTree is trying to load composite dependencies so it searches for the wavelength for one composite in the reader datasets. The DataID matching is matching against all DataIDs even though none of them actually match because it is only using the modifiers key set to an empty tuple.

To Reproduce

In [9]: from satpy.dataset import ModifierTuple

In [10]: from satpy import DataID, DataQuery

In [11]: query1 = DataQuery(wavelength=1.6, modifiers=())

In [12]: query2 = DataQuery(wavelength=1.6)

In [13]: data_id = DataID({"name": {"required": True}, "level": {}, "modifiers": {"default": [], "type": ModifierTuple}}, name="test", modifiers=tuple())

In [14]: query1.filter_dataids([data_id])
Out[14]: [DataID(name='test', modifiers=())]

In [16]: query2.filter_dataids([data_id])
Out[16]: []

Expected behavior
No match and just ignore the composite when checking if it can be loaded.

Actual results
See above. When loading the actual Scene I see:

WARNING  : Too many possible datasets to load for DataQuery(wavelength=0.85, modifiers=())

Additional context
The nucaps reader does not define wavelength in its DataID. I'm going to try and define it and see if that fixes this.

@djhoese djhoese added bug component:dep_tree Dependency tree and dataset loading labels Aug 25, 2021
@djhoese
Copy link
Member Author

djhoese commented Aug 25, 2021

Adding this to the reader yaml fixes it:

    wavelength:
      type: !!python/name:satpy.dataset.dataid.WavelengthRange

What are your thoughts @mraspaud? It would seem every reader with a custom DataID needs this defined.

@mraspaud
Copy link
Member

That's not good, and not the way I envisioned this being used. The wavelength should not be needed for custom DataIDs to work.

@mraspaud
Copy link
Member

I'm pretty sure that's because the wavelength is being hardcoded somewhere in the DataID comparisons, but unfortunately I don't have time to look a this before the week end

@djhoese
Copy link
Member Author

djhoese commented Aug 26, 2021

I'll see if I can figure it out. Now looking at it it isn't quite clear to me why adding wavelength to the reader's identifiers fixes this. 🤔 Actually maybe it makes sense...I'll see what I can do.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug component:dep_tree Dependency tree and dataset loading
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants