MetopC script fails after update to SatPy 0.37

[lobsiger]

Describe the bug
The script fails under Satpy 0.37 but has always run fine under Satpy 0.35 and Satpy 0.36.
See description and error messages/debug output on the google list in this thread:

https://groups.google.com/g/pytroll/c/RHGLzoAuZjc

To Reproduce

I puplished this minimal code together with the 3 data files and area definition on wetransfer (link valid one week):

https://we.tl/t-yL5rE8grUk (31MB zip file)

from satpy import Scene
from satpy.writers import compute_writer_results
area = 'isleofman'
composites = ['ir108_3d']

files = ['./data/AVHR_xxx_1B_M03_20220810203703Z_20220810204003Z_N_O_20220810211140Z',
         './data/AVHR_xxx_1B_M03_20220810204003Z_20220810204303Z_N_O_20220810211212Z',
         './data/AVHR_xxx_1B_M03_20220810204303Z_20220810204603Z_N_O_20220810211243Z']

scn = Scene(filenames = files, reader = 'avhrr_l1b_eps')
scn.load(composites, generate = False)
new_scn = scn.resample(area)

save_objects = []
for composite in composites:
    save_objects.append(new_scn.save_dataset(composite, filename = './imgs/MetopC-{name}.png', compute = False))
compute_writer_results(save_objects)

Expected behavior
This script should save an image of the UK in subdirectory ./imgs. It works under SatPy 0.36. fails with SatPy 0.37

Actual results

...
Traceback (most recent call last):
  File "issue037.py", line 24, in <module>
    compute_writer_results(save_objects)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/satpy/writers/__init__.py", line 541, in compute_writer_results
    da.compute(delayeds)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/base.py", line 598, in compute
    results = schedule(dsk, keys, **kwargs)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/threaded.py", line 89, in get
    results = get_async(
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/local.py", line 511, in get_async
    raise_exception(exc, tb)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/local.py", line 319, in reraise
    raise exc
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/local.py", line 224, in execute_task
    result = _execute_task(task, data)
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/dask/core.py", line 119, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
  File "/home/eumetcast/miniconda3/envs/pytroll/lib/python3.8/site-packages/PIL/Image.py", line 2943, in fromarray
    arr = obj.__array_interface__
AttributeError: 'Array' object has no attribute '__array_interface__'
(pytroll) eumetcast@kallisto:~/issue037$

Screenshots

https://groups.google.com/g/pytroll/c/RHGLzoAuZjc

Environment Info:

• OS: 0.37 fails on both GNU/Linux and Windows 10, SatPY 0.36 still works on GNU/Linux
• Satpy Version:

Name Version Build Channel

satpy 0.37.0 pyhd8ed1ab_0 conda-forge

• PyResample Version:

Name Version Build Channel

pyresample 1.23.0 py38h47df419_1 conda-forge

Full output of command "conda list" published on:
https://groups.google.com/g/pytroll/c/RHGLzoAuZjc

• Readers and writers dependencies (when relevant): [run from satpy.utils import check_satpy; check_satpy()]

Additional context
Original MetopC-area.py is part of a whole distribution Kit 4.0 that includes scripts for many GEO and LEO satellites.
Whenever possible I use generate=False for speed. This has worked fine up to now and has been extensively tested.
Most code of my Kit 4.0 is in 2 modules LEOstuff.py and GEOstuff.py. I expect other sat scripts to fail now as well ...

[mraspaud]

@lobsiger Thanks for filing this issue with all the details! I could reproduce the error thanks to your script and data files, I will start debugging this.

[mraspaud]

Using git bisect I can see that dc28b1d is the first bad commit. Also the script works fine with another composite (overview) so I suspect the problem lies in the handling of the 3d enhancement after the enhancement refactoring.

[lobsiger]

Martin,

I doubt that this is a 3d enhancement problem. The script also fails with 'ir_cloud_day'. Both use AVHRR channel '4'. Interestingly here the script worked with composites=['4']. IIRC my original script also failed with composites=['natural_with_night_fog'].
It might of course be some more general enhancement problem after 'enhancement refactoring' ...

[mraspaud]

@lobsiger the 3d effect was still buggy, and I fixed it in #2174, and now ir108_3d works.
I will check if I get the problem with ir_cloud_day.

[lobsiger]

@mraspaud I was was probably wrong with 'natural_with_night_fog'. Maybe it didn't work because I had a default list of composites=['ir108_3d', 'natural_with_night_fog']. I just tested with satpy 0.37 for different DAY and NIG passes this list:

composites = ['cloud_phase_distinction', 'cloud_phase_distinction_raw', 'cloudtop', 'day_microphysics', 'green_snow',
'natural_color', 'natural_color_raw', 'natural_enh', 'natural_with_night_fog', 'night_fog', 'night_microphysics', 'overview',
'overview_raw', 'snow']

But 'ir_cloud_day' (like ir108_3d') does NOT work. That's verified.

Ernst

[mraspaud]

@lobsiger thanks for confirming this. I checked and both ir_cloud_day and ir108_3d now work with my PR mentionned above. Do you want to try it out?

[lobsiger]

@mraspaud TBH I do not know how to update just your fix with conda and/or pip. I have no Satpy devel version setup here.
Years back I knew at least how to update from the github master branch. But recently I tried that and it didn't work anymore.
Do you expect me to copy your fix with an editor (I doubt it) ?

[gerritholl]

@lobsiger Try pip install git+https://github.com/mraspaud/satpy.git@fix-3d-enhancement

[lobsiger]

@gerritholl thanks for that. I have done that under GNU/Linux Satpy 0.37 and the two ir composites now do work!

I also tried just now

pip install git+https://github.com/pytroll/pycoast.git@main

on my Windows 10 setup and it seems to work. Can you confirm that this is the way to directly install from the 'main' branch? As I said above I have tried that before, but it didn't work at that time. I now suspect that then I have used branch name 'master' instead of 'main'. Has this branch been renamed somewhere in the past or is this due to my aging brain?

The point is that I fixed with PRs a couple of things in pycoast this spring but the pycoast conda package is still at 1.5.0. I'am up to date on GNU/Linux but could not update my Windows 10 install easily (probably because of that silly 'master'/'main' thing).

[gerritholl]

@lobsiger That would install the main branch, yes. I think you can leave out @main and then somehow pip or git will magically figure out which branch is the reference branch, so it should work in case you don't know if it's main, master, or something else. Satpy and most other pytroll projects renamed from master to main some time in early 2021, I think.

[lobsiger]

@gerritholl thanks for that. Renaming explains my 'master'/'main' problem. Is there any chance that I can easily screw up my install if I switch form conda to pip back and forth? As I said I have now updated my Windows 10 and a conda list pycoast says:

pycoast 1.5.0+50.gb2ad512 pypi_0 pypi

Will conda understand when a pycoast v. 1.6.0 (with Daves PR 63) comes out anytime this fall that I update from conda-forge?

[mraspaud]

@lobsiger I would be very careful with mixing pypi packages with conda because it can make the environment difficult to update and prone to unexplainable errors/dependency problems.
So I use pip just to install packages that are either not available from conda, or for testing.

[mraspaud]

also, I consider this a quite serious bug, so when my PR is reviewed and merged, I will try to make a release ASAP.