scene.coarsest_area and scene.resample not working on MSG1/MSG2 in satpy 0.29 #2134
Comments
|
Can you try replacing this line: The sandwich composite uses the HRV channel, which does not cover the full SEVIRI disk. The extra |
|
Still does not seems to work. Switched to satpy 0.36.0. In [1]: from satpy import Scene In [2]: scn = Scene(reader='seviri_l1b_native',filenames=['MSG1-SEVI-MSG15-0100-NA-20220101075742.422000000Z-NA.nat'], reader_kwargs={'fill_disk':True}) In [3]: scn.load(['ir_sandwich']) In [4]: lscn = scn.resample(scn.coarsest_area(), resampler='native')ValueError Traceback (most recent call last) ~/anaconda3/lib/python3.7/site-packages/satpy/scene.py in resample(self, destination, datasets, generate, unload, resampler, reduce_data, **resample_kwargs) ~/anaconda3/lib/python3.7/site-packages/satpy/scene.py in _resampled_scene(self, new_scn, destination_area, reduce_data, **resample_kwargs) ~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in resample_dataset(dataset, destination_area, **kwargs) ~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in resample(source_area, data, destination_area, resampler, **kwargs) ~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in resample(self, data, cache_dir, mask_area, **kwargs) ~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in resample(self, data, cache_dir, mask_area, **kwargs) ~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in compute(self, data, expand, **kwargs) ~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in expand_reduce(cls, d_arr, repeats) ~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in aggregate(d, y_size, x_size) ValueError: Aggregation requires arrays with shapes and chunks divisible by the factor Additionally I tried running this: scn.load(['IR_108', 'HRV']) In [20]: scn['IR_108'].shape scn.to_xarray_dataset() To avoid creating the large chunks, set the option
In [15]: scn['IR_108'].data In [16]: scn['HRV'].data |
|
I wonder if it is the 3711 chunk size of the HRV first dimension. That isn't divisble by 3 ("the factor") and also a little confusing what each chunk must be. For your last line could you do |
|
scn['HRV'].data.chunks |
|
Hacky workaround that may work, right before calling Does that fix things? |
|
Issue still persists |
|
after you do the above hack, can you print the chunks again? I'm very confused at this point. |
|
In [1]: from satpy import Scene In [3]: scn.load(['IR_108', 'HRV']) In [4]: scn["HRV"].data =scn['HRV'].data.rechunk(3712) In [5]: scn["IR_108"].data.chunks In [6]: scn["HRV"].data.chunks |
|
|
Ah yes, I suppose that does make sense. Oops. You could try a chunk size of 11136 or really anything divisible by 3 I guess, but you'd want an even chunking so maybe 2784 which would give you 4 chunks in each dimension, but is still divisible by 3. |
|
I will also try to debug tonight. |
|
Following code snippet worked: from satpy import Scene Point here is 3712 is not divisible by 3, 3711 is divisible by 3. |
|
I'm not sure we should close this yet. The workaround I gave you is more of a hack. The original use case/code is supposed to work or at least we want it to work. Let's see what some of the SEVIRI experts have to say about this. |
|
FYI, this is a duplicate of #1595 |
|
@ghansham Not a solution to the issue, but why do you want to resample onto the coarsest grid? The purpose of the |
|
I feel the products downloaded using the scripts mentioned in the following link: have some issues mentioned in this bug (variable chunk size) specifically for high resolution bands |
|
Closing as duplicate of #1595. |
|
I will try the one you mentioned.
…On Tue, 28 Jun, 2022, 18:44 strandgren, ***@***.***> wrote:
@ghansham <https://github.com/ghansham> Not a solution to the issue, but
why do you want to resample onto the coarsest grid? The purpose of the
ir_sandwich composite is mainly to exploit the higher spatial resolution
of the HRV channel. If you instead use the colorized_ir_clouds composite
you should get similar output, but for the full disk and with no need to
resample.
—
Reply to this email directly, view it on GitHub
<#2134 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AAYXFJAUPAZ3GHY4AZYNTM3VRL3BRANCNFSM5Z6KDKXQ>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Describe the bug
While resampling the scene to coarsest_area using resampler='native', it fails.
I was trying to generate the ir_sandwich, when I got stuck here.
To Reproduce
Expected behavior
It should have resampled without any error
Actual results
Text output of actual results or error messages including full tracebacks if applicable.
In [18]: lscn = scn.resample(scn.coarsest_area(), resampler='native')
ValueError Traceback (most recent call last)
in
----> 1 lscn = scn.resample(scn.coarsest_area(), resampler='native')
~/anaconda3/lib/python3.7/site-packages/satpy/scene.py in coarsest_area(self, datasets)
259
260 """
--> 261 return self._compare_areas(datasets=datasets, compare_func=min)
262
263 def min_area(self, datasets=None):
~/anaconda3/lib/python3.7/site-packages/satpy/scene.py in _compare_areas(self, datasets, compare_func)
207 if not all(isinstance(x, type(areas[0]))
208 for x in areas[1:]):
--> 209 raise ValueError("Can't compare areas of different types")
210 elif isinstance(areas[0], AreaDefinition):
211 first_crs = areas[0].crs
ValueError: Can't compare areas of different types
In [19]: lscn = scn.resample(scn._datasets['IR_108'].attrs['area'], resampler='native')
ValueError Traceback (most recent call last)
in
----> 1 lscn = scn.resample(scn._datasets['IR_108'].attrs['area'], resampler='native')
~/anaconda3/lib/python3.7/site-packages/satpy/scene.py in resample(self, destination, datasets, generate, unload, resampler, reduce_data, **resample_kwargs)
811 new_scn = self.copy(datasets=datasets)
812 self._resampled_scene(new_scn, destination, resampler=resampler,
--> 813 reduce_data=reduce_data, **resample_kwargs)
814
815 # regenerate anything from the wishlist that needs it (combining
~/anaconda3/lib/python3.7/site-packages/satpy/scene.py in _resampled_scene(self, new_scn, destination_area, reduce_data, **resample_kwargs)
771 kwargs = resample_kwargs.copy()
772 kwargs['resampler'] = resamplers[source_area]
--> 773 res = resample_dataset(dataset, destination_area, **kwargs)
774 new_datasets[ds_id] = res
775 if ds_id in new_scn._datasets:
~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in resample_dataset(dataset, destination_area, **kwargs)
1380
1381 fill_value = kwargs.pop('fill_value', get_fill_value(dataset))
-> 1382 new_data = resample(source_area, dataset, destination_area, fill_value=fill_value, **kwargs)
1383 new_attrs = new_data.attrs
1384 new_data.attrs = dataset.attrs.copy()
~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in resample(source_area, data, destination_area, resampler, **kwargs)
1343 res = [resampler_instance.resample(ds, **kwargs) for ds in data]
1344 else:
-> 1345 res = resampler_instance.resample(data, **kwargs)
1346
1347 return res
~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in resample(self, data, cache_dir, mask_area, **kwargs)
959 cache_dir=cache_dir,
960 mask_area=mask_area,
--> 961 **kwargs)
962
963 @staticmethod
~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in resample(self, data, cache_dir, mask_area, **kwargs)
431
432 cache_id = self.precompute(cache_dir=cache_dir, **kwargs)
--> 433 return self.compute(data, cache_id=cache_id, **kwargs)
434
435 def _create_cache_filename(self, cache_dir=None, prefix='',
~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in compute(self, data, expand, **kwargs)
1046 repeats[x_axis] = x_repeats
1047
-> 1048 d_arr = self.expand_reduce(data.data, repeats)
1049 new_data = xr.DataArray(d_arr, dims=data.dims)
1050 return update_resampled_coords(data, new_data, target_geo_def)
~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in expand_reduce(cls, d_arr, repeats)
1012 y_size = 1. / repeats[0]
1013 x_size = 1. / repeats[1]
-> 1014 return cls.aggregate(d_arr, y_size, x_size)
1015 else:
1016 raise ValueError("Must either expand or reduce in both "
~/anaconda3/lib/python3.7/site-packages/satpy/resample.py in aggregate(d, y_size, x_size)
970 "more than 2 dimensions.")
971 if not (x_size.is_integer() and y_size.is_integer()):
--> 972 raise ValueError("Aggregation factors are not integers")
973 for agg_size, chunks in zip([y_size, x_size], d.chunks):
974 for chunk_size in chunks:
ValueError: Aggregation factors are not integers
Screenshots
If applicable, add screenshots to help explain your problem.
Environment Info:
from satpy.utils import check_satpy; check_satpy()]Readers
=======
abi_l1b: ok
abi_l1b_scmi: ok
abi_l2_nc: ok
acspo: ok
agri_l1: ok
ahi_hrit: ok
ahi_hsd: ok
ahi_l1b_gridded_bin: ok
ami_l1b: ok
amsr2_l1b: ok
amsr2_l2: ok
amsr2_l2_gaasp: ok
ascat_l2_soilmoisture_bufr: cannot find module 'satpy.readers.ascat_l2_soilmoisture_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'")))
avhrr_l1b_aapp: ok
avhrr_l1b_eps: ok
avhrr_l1b_gaclac: ok
avhrr_l1b_hrpt: ok
avhrr_l1c_eum_gac_fdr_nc: ok
caliop_l2_cloud: cannot find module 'satpy.readers.caliop_l2_cloud' (cannot import name 'Dataset' from 'satpy.dataset' (/home/rapid/anaconda3/lib/python3.7/site-packages/satpy/dataset/init.py))
clavrx: ok
cmsaf-claas2_l2_nc: ok
electrol_hrit: ok
fci_l1c_nc: ok
fci_l2_nc: ok
generic_image: ok
geocat: ok
ghrsst_l3c_sst: cannot find module 'satpy.readers.ghrsst_l3c_sst' (cannot import name 'Dataset' from 'satpy.dataset' (/home/rapid/anaconda3/lib/python3.7/site-packages/satpy/dataset/init.py))
glm_l2: ok
goes-imager_hrit: ok
goes-imager_nc: ok
gpm_imerg: ok
grib: ok
hsaf_grib: ok
hy2_scat_l2b_h5: ok
iasi_l2: ok
iasi_l2_so2_bufr: cannot find module 'satpy.readers.iasi_l2_so2_bufr' (('Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes.\n Error: ', ModuleNotFoundError("No module named 'eccodes'")))
jami_hrit: ok
li_l2: cannot find module 'satpy.readers.li_l2' (cannot import name 'Dataset' from 'satpy.dataset' (/home/rapid/anaconda3/lib/python3.7/site-packages/satpy/dataset/init.py))
maia: ok
mersi2_l1b: ok
mimicTPW2_comp: ok
mirs: ok
modis_l1b: ok
modis_l2: ok
msi_safe: ok
mtsat2-imager_hrit: ok
mviri_l1b_fiduceo_nc: ok
nucaps: ok
nwcsaf-geo: ok
nwcsaf-msg2013-hdf5: ok
nwcsaf-pps_nc: ok
olci_l1b: ok
olci_l2: ok
omps_edr: ok
safe_sar_l2_ocn: ok
sar-c_safe: ok
satpy_cf_nc: ok
scatsat1_l2b: cannot find module 'satpy.readers.scatsat1_l2b' (cannot import name 'Dataset' from 'satpy.dataset' (/home/rapid/anaconda3/lib/python3.7/site-packages/satpy/dataset/init.py))
seviri_l1b_hrit: ok
seviri_l1b_icare: ok
seviri_l1b_native: ok
seviri_l1b_nc: ok
seviri_l2_bufr: cannot find module 'satpy.readers.seviri_l2_bufr' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
seviri_l2_grib: cannot find module 'satpy.readers.seviri_l2_grib' (Missing eccodes-python and/or eccodes C-library installation. Use conda to install eccodes)
slstr_l1b: ok
slstr_l2: ok
smos_l2_wind: ok
tropomi_l2: ok
vaisala_gld360: ok
vii_l1b_nc: ok
vii_l2_nc: ok
viirs_compact: ok
viirs_edr_active_fires: ok
viirs_edr_flood: ok
viirs_l1b: ok
viirs_sdr: ok
virr_l1b: ok
Writers
/home/rapid/anaconda3/lib/python3.7/site-packages/pyninjotiff/tifffile.py:155: UserWarning: failed to import the optional _tifffile C extension module.
Loading of some compressed images will be slow.
Tifffile.c can be obtained at http://www.lfd.uci.edu/~gohlke/
"failed to import the optional _tifffile C extension module.\n"
awips_tiled: ok
cf: ok
geotiff: ok
mitiff: ok
ninjotiff: ok
simple_image: ok
Extras
cartopy: ok
geoviews: ok
Additional context
Add any other context about the problem here.
The text was updated successfully, but these errors were encountered: