modifier NIREmissivePartFromReflectance triggers early dask computation

[gerritholl]

Describe the bug

Calling the modifier NIREmissivePartFromReflectance in satpy.modifiers.spectral leads to early computing of the arrays involved.

To Reproduce

import dask.config
from satpy.tests.utils import CustomScheduler
from satpy import Scene
from glob import glob
from satpy.modifiers.spectral import NIREmissivePartFromReflectance
seviri_files = glob("/media/nas/x21308/scratch/SEVIRI/202103300900/H-000*")
sc = Scene(filenames={"seviri_l1b_hrit": seviri_files})
sc.load(["IR_039", "IR_108", "IR_134"])
mod = NIREmissivePartFromReflectance(name="nir")
with dask.config.set(scheduler=CustomScheduler(max_computes=0)):
    mod([sc["IR_039"], sc["IR_108"]], sc["IR_134"])

Expected behavior

I expect that the code completes without computing any dask arrays and without raising any exceptions.

Actual results

It fails with RuntimeError:

Traceback (most recent call last):
  File "/data/gholl/checkouts/protocode/mwe/split-39-early-compute.py", line 11, in <module>
    mod([sc["IR_039"], sc["IR_108"]], sc["IR_134"])
  File "/data/gholl/checkouts/satpy/satpy/modifiers/spectral.py", line 162, in __call__
    return self._get_emissivity_as_dataarray(projectables, optional_datasets)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/checkouts/satpy/satpy/modifiers/spectral.py", line 173, in _get_emissivity_as_dataarray
    emissivity = self._get_emissivity_as_dask(da_nir, da_tb11, da_tb13_4, da_sun_zenith, _nir.attrs)
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/checkouts/satpy/satpy/modifiers/spectral.py", line 186, in _get_emissivity_as_dask
    return reflectance_3x_calculator.emissive_part_3x()
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/pyspectral/near_infrared_reflectance.py", line 186, in emissive_part_3x
    return self.radiance2tb(self._e3x)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/pyspectral/radiance_tb_conversion.py", line 250, in radiance2tb
    return radiance2tb(rad, self.rsr[self.bandname][self.detector]['central_wavelength'] * 1e-6)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/pyspectral/radiance_tb_conversion.py", line 262, in radiance2tb
    return rad2temp(wavelength, rad)
           ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/pyspectral/blackbody.py", line 51, in blackbody_rad2temp
    rad = np.array(radiance, dtype='float64')
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/dask/array/core.py", line 1700, in __array__
    x = self.compute()
        ^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/dask/base.py", line 342, in compute
    (result,) = compute(self, traverse=False, **kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/mambaforge/envs/py311/lib/python3.11/site-packages/dask/base.py", line 628, in compute
    results = schedule(dsk, keys, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/data/gholl/checkouts/satpy/satpy/tests/utils.py", line 288, in __call__
    raise RuntimeError("Too many dask computations were scheduled: "
RuntimeError: Too many dask computations were scheduled: 1

Environment Info:

• OS: openSUSE 15.3
• Satpy Version: v0.44.0-17-gbc32c9434
• pyspectral Version: 0.12.5

Additional context

This is due to a download bug/limitation in pyspectral: pytroll/pyspectral#201

[pnuu]

Fixing the blackbody module isn't enough. There is a a forced .compute() for the reflectances in https://github.com/pytroll/pyspectral/blob/main/pyspectral/near_infrared_reflectance.py#L290

I'll file an issue to Pyspectral.

[pnuu]

No, sorry, I read the code too hastily. The computation is triggered only if the inputs were something other than Dask arrays.

It's just the dtype that needs to be preserved, I'll write an issue about this (and propably implement it) instead.

[pnuu]

Closed by pytroll/pyspectral#203