MIMIC reader available_dataset_names returns 1d lat/lon fields

[joleenf]

Describe the bug
A mimic dataset loaded into the Scene object will include the data field (TPWGrid at the least) and the 1-D latitude and longitude fields. The latitude and longitude fields are 1-D versus the 2-D.

To Reproduce

from satpy import find_files_and_readers, Scene
mimic_files=find_files_and_readers(base_dir=<mimic_data_dir>)
scn = Scene(mimic_files)
keys = scn.available_dataset_names()
scn.load(keys)    # Therefore this fails
scn.save_datasets(filename="{name}_{start_time:%Y%m%d%H%M%S}.tif")  # also fails/see Actual Results for further information

Expected behavior
I am thinking that any dataset available from the available_dataset_names should load without error, or it should not be returned in the list of available_dataset_names. True?

Actual results

Odd, when debug_on is not set, there is an error with the load command, but not with debug_on.
Could not load dataset 'DatasetID(name='latitude', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=())': different number of dimensions on data and dims: 1 vs 2
Traceback (most recent call last):
File "/home/joleenf/miniconda3/envs/satpy/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 782, in _load_dataset_with_area
ds = self._load_dataset_data(file_handlers, dsid, **kwargs)
File "/home/joleenf/miniconda3/envs/satpy/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 667, in _load_dataset_data
proj = self._load_dataset(dsid, ds_info, file_handlers, **kwargs)
File "/home/joleenf/miniconda3/envs/satpy/lib/python3.8/site-packages/satpy/readers/yaml_reader.py", line 643, in _load_dataset
projectable = fh.get_dataset(dsid, ds_info)
File "/home/joleenf/miniconda3/envs/satpy/lib/python3.8/site-packages/satpy/readers/mimic_TPW2_nc.py", line 109, in get_dataset
data = xr.DataArray(data, dims=['y', 'x'])
File "/home/joleenf/miniconda3/envs/satpy/lib/python3.8/site-packages/xarray/core/dataarray.py", line 344, in init
coords, dims = _infer_coords_and_dims(data.shape, coords, dims)
File "/home/joleenf/miniconda3/envs/satpy/lib/python3.8/site-packages/xarray/core/dataarray.py", line 121, in _infer_coords_and_dims
raise ValueError(
ValueError: different number of dimensions on data and dims: 1 vs 2
The following datasets were not created and may require resampling to be generated: DatasetID(name='longitude', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=()), DatasetID(name='latitude', wavelength=None, resolution=None, polarization=None, calibration=None, level=None, modifiers=())

However, with debug_on, no error is created with the load, the error occurs in the save_datasets step:
----> 1 scn.save_datasets(filename="{name}_{start_time:%Y%m%d%H%M%S}.tif")

~/miniconda3/envs/satpy/lib/python3.8/site-packages/satpy/scene.py in save_datasets(self, writer, filename, datasets, compute, **kwargs)
1347 datasets = [ds for ds in datasets if ds is not None]
1348 if not datasets:
-> 1349 raise RuntimeError("None of the requested datasets have been "
1350 "generated or could not be loaded. Requested "
1351 "composite inputs may need to have matching "

RuntimeError: None of the requested datasets have been generated or could not be loaded. Requested composite inputs may need to have matching dimensions (eg. through resampling).

Environment Info:

• OS: Linux (only tested on Linux)
• Satpy Version: 0.22

[djhoese]

I agree this is not expected and would consider this a bug in the reader. For the odd debug_on error thing, is it possible you have this backwards? With debug_on it fails in the load step, with it off it fails in save_datasets? My thought was that the load error wasn't actually an error but instead part of a DEBUG log message or something.

What are the names of the 2D lon/lat variables?

I think the reader needs to be updated to either:

1. Properly create DataArray objects when the data is not 2D.
2. Not handle (in the file handler's .available_datasets method) any 1D datasets.

[joleenf]

I am sorry, I was mistaken, the error occurs with/without debug_on
The data I am currently working with only contains a tpwGrid, there are coarse resolution datasets which contain more fields:

1.) fine

dimensions:
	lat = 9001 ;
	lon = 18000 ;
variables:
	float lonArr(lon) ;
		string lonArr:units = "degrees_east" ;
	float latArr(lat) ;
		string latArr:units = "degrees_north" ;
	float tpwGrid(lat, lon) ;
		tpwGrid:least_significant_digit = 1LL ;
		string tpwGrid:units = "mm" ;

2.) coarse

dimensions:
	lat = 721 ;
	lon = 1440 ;
variables:
	float lonArr(lon) ;
		string lonArr:units = "degrees_east" ;
	float latArr(lat) ;
		string latArr:units = "degrees_north" ;
	float tpwGrid(lat, lon) ;
		tpwGrid:least_significant_digit = 1LL ;
		string tpwGrid:units = "mm" ;
	float tpwGridPrior(lat, lon) ;
		tpwGridPrior:least_significant_digit = 1LL ;
		string tpwGridPrior:units = "mm" ;
	float tpwGridSubseq(lat, lon) ;
		tpwGridSubseq:least_significant_digit = 1LL ;
		string tpwGridSubseq:units = "mm" ;
	float timeAwayGridPrior(lat, lon) ;
		timeAwayGridPrior:least_significant_digit = 0LL ;
		string timeAwayGridPrior:units = "minutes" ;
	float timeAwayGridSubseq(lat, lon) ;
		timeAwayGridSubseq:least_significant_digit = 0LL ;
		string timeAwayGridSubseq:units = "minutes" ;
	float footGridPrior(lat, lon) ;
		footGridPrior:least_significant_digit = 0LL ;
		string footGridPrior:units = "km" ;
	float footGridSubseq(lat, lon) ;
		footGridSubseq:least_significant_digit = 0LL ;
		string footGridSubseq:units = "km" ;
	ubyte satGridPrior(lat, lon) ;
		string satGridPrior:source_key = "Key: 0: None, 1: NOAA-N, 2: NOAA-P, 3: Metop-A, 4: Metop-B, 5: SNPP, 6: SSMI-17, 7: SSMI-18" ;
	ubyte satGridSubseq(lat, lon) ;
		string satGridSubseq:source_key = "Key: 0: None, 1: NOAA-N, 2: NOAA-P, 3: Metop-A, 4: Metop-B, 5: SNPP, 6: SSMI-17, 7: SSMI-18" ;

[djhoese]

After skimming through the reader, I think the reader should not advertise the lon/lat arrays. The reader is taking the lon/lat arrays and then making an AreaDefinition from them. In this case it doesn't make sense to provide this area (2D) with lonArr or latArr when the user requests them since they are 1D.

[joleenf]

@djhoese There are going to be a few more updates in this pull when it is ready. One thing is leading to another and I have found that ideally, in order for the process scn.load(scn.available_dataset_ids()); scn.save_datasets(...) to work, I need to add the xarray_kwargs command which keeps the original float32 time latency data as float32 rather than converting to timdelta64[ns]. In addition, I am added some of the metadata which I inadvertently omitted from the original netCDF files.