fails to load all composites for abi_l1b

[raybellwaves]

Describe the bug
fails to load all composites for GOES-16 data (abi_l1b reader).

Moved convo from slack (https://pytroll.slack.com/archives/C0LNH7LMB/p1600825423000100?thread_ts=1600574120.039700&cid=C0LNH7LMB)

To Reproduce

See https://gist.github.com/raybellwaves/c9ef25bbb9819b3b88c1201d11340b00

Expected behavior
Load all datasets

Actual results

---------------------------------------------------------------------------
ValueError                                Traceback (most recent call last)
<ipython-input-28-d99dafe80c78> in <module>
----> 1 scn_reampled = scn.resample(scn['C01'].attrs['area'])

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/scene.py in resample(self, destination, datasets, generate, unload, resampler, reduce_data, **resample_kwargs)
   1126         # multiple resolutions, etc.)
   1127         if generate:
-> 1128             keepables = new_scn._generate_composites()
   1129         else:
   1130             # don't lose datasets that we may need later for generating

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/scene.py in _generate_composites(self, nodes)
    895             required_nodes = self._wishlist - set(self._datasets.keys())
    896             nodes = set(self._dependency_tree.trunk(nodes=required_nodes)) - set(self._datasets.keys())
--> 897         return self._read_composites(nodes)
    898 
    899     def _remove_failed_datasets(self, keepables):

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/scene.py in _read_composites(self, compositor_nodes)
    871         keepables = set()
    872         for item in compositor_nodes:
--> 873             self._generate_composite(item, keepables)
    874         return keepables
    875 

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/scene.py in _generate_composite(self, comp_node, keepables)
    811         try:
    812             delayed_prereq = False
--> 813             prereq_datasets = self._get_prereq_datasets(
    814                 comp_node.name,
    815                 prereqs,

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/scene.py in _get_prereq_datasets(self, comp_id, prereq_nodes, keepables, skip)
    761             if prereq_id not in self._datasets and prereq_id not in keepables \
    762                     and not prereq_node.is_leaf:
--> 763                 self._generate_composite(prereq_node, keepables)
    764 
    765             if prereq_node is self._dependency_tree.empty_node:

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/scene.py in _generate_composite(self, comp_node, keepables)
    845 
    846         try:
--> 847             composite = compositor(prereq_datasets,
    848                                    optional_datasets=optional_datasets,
    849                                    **self.attrs)

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/composites/__init__.py in __call__(self, projectables, *args, **kwargs)
   1654 
   1655         # Get merged metadata
-> 1656         attrs = combine_metadata(foreground, background)
   1657         if attrs.get('sensor') is None:
   1658             # sensor can be a set

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/dataset/metadata.py in combine_metadata(average_times, *metadata_objects)
     50     shared_keys = _shared_keys(info_dicts)
     51 
---> 52     return _combine_shared_info(shared_keys, info_dicts, average_times)
     53 
     54 

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/dataset/metadata.py in _combine_shared_info(shared_keys, info_dicts, average_times)
     78         if 'time' in key and isinstance(values[0], datetime) and average_times:
     79             shared_info[key] = average_datetimes(values)
---> 80         elif _are_values_combinable(values):
     81             shared_info[key] = values[0]
     82     return shared_info

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/dataset/metadata.py in _are_values_combinable(values)
    113         # checked any_arrays so this false positive should have no impact
    114         return _all_list_of_arrays_equal(values)
--> 115     return _all_values_equal(values)
    116 
    117 

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/dataset/metadata.py in _all_values_equal(values)
    176 def _all_values_equal(values):
    177     try:
--> 178         return _pairwise_all(nan_allclose, values)
    179     except TypeError:
    180         return _pairwise_all(eq, values)

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/satpy/dataset/metadata.py in _pairwise_all(func, values)
    141 def _pairwise_all(func, values):
    142     for value in values[1:]:
--> 143         if not func(values[0], value):
    144             return False
    145     return True

<__array_function__ internals> in allclose(*args, **kwargs)

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/numpy/core/numeric.py in allclose(a, b, rtol, atol, equal_nan)
   2187 
   2188     """
-> 2189     res = all(isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))
   2190     return bool(res)
   2191 

<__array_function__ internals> in isclose(*args, **kwargs)

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/numpy/core/numeric.py in isclose(a, b, rtol, atol, equal_nan)
   2288     yfin = isfinite(y)
   2289     if all(xfin) and all(yfin):
-> 2290         return within_tol(x, y, atol, rtol)
   2291     else:
   2292         finite = xfin & yfin

~/local/bin/anaconda3/envs/test_env/lib/python3.8/site-packages/numpy/core/numeric.py in within_tol(x, y, atol, rtol)
   2274     def within_tol(x, y, atol, rtol):
   2275         with errstate(invalid='ignore'):
-> 2276             return less_equal(abs(x-y), atol + rtol * abs(y))
   2277 
   2278     x = asanyarray(a)

ValueError: operands could not be broadcast together with shapes (4,) (0,) 

Screenshots
If applicable, add screenshots to help explain your problem.

Environment Info:

• OS: Linux
• Satpy Version: 0.23.0
• PyResample Version: 1.16.0
• Readers and writers dependencies (when relevant): [run from satpy.config import check_satpy; check_satpy()]

[djhoese]

Based on the error message (which I edited into your message above, you missed the ValueError), I wonder if this is something from one of the channels conflicting with the night time background (static) image. Could you print out the attributes for that?

You could do:

scn.load([... the stuff you're loading ...], generate=False)
print(scn.keys())

To see all the loaded datasets that are going into the composites that may be causing issues.

[djhoese]

This issue is decently old and was waiting on more information from the poster. A lot has changed since this was originally posted so I'm hoping it is fixed now. That said, the error in the notebook (in the gist you linked) and as mentioned on slack, seems to be from metadata from some of the readers being loaded as numpy arrays and the composites are trying to merge them together.

Given the lack of response I'm going to close this in hopes that it is fixed now. If someone runs into this again or if someone can provide more information then we can reopen.