Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Sentinel-1 sar-c safe reader consumes too much memory and a lot of time #2830

Closed
yukaribbba opened this issue Jun 18, 2024 · 4 comments
Closed

Sentinel-1 sar-c safe reader consumes too much memory and a lot of time #2830

yukaribbba opened this issue Jun 18, 2024 · 4 comments
Assignees
@mraspaud

Comments

@yukaribbba
Copy link
Contributor

Describe the bug
Nearly all my 64GB memory and it still asks for more.

To Reproduce

files = find_files_and_readers(base_dir="C:/Users/45107/Downloads/Sat/Polar/sen1a_iw_20240614t095536_20240614t095558",
                               reader="sar-c_safe",
                               )
scn = Scene(filenames=files)
scn.load(["green-sar"])
scn.save_dataset("green-sar", filename="green.tif", fill_value=0, compress=None)

Actual results

C:\Users\45107\miniforge3\python.exe C:\Users\45107\PycharmProjects\Satellite\testcode.py 
[DEBUG: 2024-06-18 23:25:29 : satpy.readers.yaml_reader] Reading ('C:\\Users\\45107\\Documents\\GitHub\\satpy\\satpy\\etc\\readers\\sar-c_safe.yaml',)
[DEBUG: 2024-06-18 23:25:29 : rasterio.env] PROJ data files are available at built-in paths.
[DEBUG: 2024-06-18 23:25:29 : satpy.readers.yaml_reader] Reading ('C:\\Users\\45107\\Documents\\GitHub\\satpy\\satpy\\etc\\readers\\sar-c_safe.yaml',)
[WARNING: 2024-06-18 23:25:29 : satpy.readers.yaml_reader] Available datasets are unknown, returning all datasets...
[DEBUG: 2024-06-18 23:25:29 : satpy.composites.config_loader] Looking for composites config file sar-c.yaml
[DEBUG: 2024-06-18 23:25:29 : satpy.composites.config_loader] Looking for composites config file sar.yaml
[DEBUG: 2024-06-18 23:25:29 : satpy.readers.sar_c_safe] Reading measurement.
[DEBUG: 2024-06-18 23:25:29 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x0000015936E7D7C0>
[DEBUG: 2024-06-18 23:25:29 : rasterio.env] Starting outermost env
[DEBUG: 2024-06-18 23:25:29 : rasterio.env] No GDAL environment exists
[DEBUG: 2024-06-18 23:25:29 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x0000015936B1E410> created
[DEBUG: 2024-06-18 23:25:30 : rasterio._filepath] Installing FilePath filesystem handler plugin...
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] PROJ data files are available at built-in paths.
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x0000015936B1E410>.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x0000015936E7D7C0>
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Sharing flag: 0
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Dataset <open DatasetReader name='C:/Users/45107/Downloads/Sat/Polar/sen1a_iw_20240614t095536_20240614t095558/S1A_IW_GRDH_1SDV_20240614T095536_20240614T095558_054318_069B94_1707.SAFE/measurement/s1a-iw-grd-vv-20240614t095536-20240614t095558-054318-069b94-001.tiff' mode='r'> is started.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x0000015936E7D7C0>
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x0000015936B1E410> options
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x0000015936B1E410>.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exiting outermost env
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x0000015936E7D7C0>
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading noise data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading calibration data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Calibrating.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading longitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading latitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading measurement.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x0000015936FAC7A0>
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Starting outermost env
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] No GDAL environment exists
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x0000015936FAE020> created
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] PROJ data files are available at built-in paths.
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x0000015936FAE020>.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x0000015936FAC7A0>
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Sharing flag: 0
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Nodata success: 0, Nodata value: 0.000000
[DEBUG: 2024-06-18 23:25:30 : rasterio._base] Dataset <open DatasetReader name='C:/Users/45107/Downloads/Sat/Polar/sen1a_iw_20240614t095536_20240614t095558/S1A_IW_GRDH_1SDV_20240614T095536_20240614T095558_054318_069B94_1707.SAFE/measurement/s1a-iw-grd-vh-20240614t095536-20240614t095558-054318-069b94-002.tiff' mode='r'> is started.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x0000015936FAC7A0>
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x0000015936FAE020> options
[DEBUG: 2024-06-18 23:25:30 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x0000015936FAE020>.
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exiting outermost env
[DEBUG: 2024-06-18 23:25:30 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x0000015936FAC7A0>
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading noise data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading calibration data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Calibrating.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading longitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading latitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading measurement.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading noise data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading calibration data.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Calibrating.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading longitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Reading latitude.
[DEBUG: 2024-06-18 23:25:30 : satpy.readers.sar_c_safe] Constructing coordinate arrays.
[DEBUG: 2024-06-18 23:25:30 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:30 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:30 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:30 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:30 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:31 : satpy.scene] Unloading dataset: DataID(name='measurement', polarization='vv', resolution=80, calibration=<1>, quantity=<1>, modifiers=())
[DEBUG: 2024-06-18 23:25:31 : satpy.scene] Unloading dataset: DataID(name='measurement', polarization='vh', resolution=80, calibration=<1>, quantity=<1>, modifiers=())
[DEBUG: 2024-06-18 23:25:31 : satpy.scene] Unloading dataset: DataID(name='measurement', polarization='vv', resolution=80, calibration=<1>, quantity=<2>, modifiers=())
[DEBUG: 2024-06-18 23:25:31 : satpy.writers] Reading ['C:\\Users\\45107\\Documents\\GitHub\\satpy\\satpy\\etc\\writers\\geotiff.yaml']
[DEBUG: 2024-06-18 23:25:31 : satpy.writers] Adding enhancement configuration from file: C:\Users\45107\Documents\GitHub\satpy\satpy\etc\enhancements\generic.yaml
[DEBUG: 2024-06-18 23:25:31 : satpy.writers] Adding enhancement configuration from file: D:\satpy_config\enhancements\generic.yaml
[DEBUG: 2024-06-18 23:25:31 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:31 : satpy.writers] Data for DataID(name='green-sar', resolution=80) will be enhanced with options:
	[{'name': 'stretch', 'method': <function stretch at 0x0000015936F3A700>, 'kwargs': {'stretch': 'crude', 'min_stretch': [0.00109, -17.57, 0.00332], 'max_stretch': [0.0594, -3.3, 0.3]}}]
[DEBUG: 2024-06-18 23:25:31 : trollimage.xrimage] Applying stretch crude with parameters {'min_stretch': [0.00109, -17.57, 0.00332], 'max_stretch': [0.0594, -3.3, 0.3]}
[DEBUG: 2024-06-18 23:25:31 : rasterio.crs] Matched. confidence=100, c_code=b'4326', c_name=b'EPSG'
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Entering env context: <rasterio.env.Env object at 0x000001593702C980>
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Starting outermost env
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] No GDAL environment exists
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] New GDAL environment <rasterio._env.GDALEnv object at 0x0000015937073460> created
[DEBUG: 2024-06-18 23:25:31 : rasterio._env] GDAL_DATA found in environment.
[DEBUG: 2024-06-18 23:25:31 : rasterio._env] PROJ data files are available at built-in paths.
[DEBUG: 2024-06-18 23:25:31 : rasterio._env] Started GDALEnv: self=<rasterio._env.GDALEnv object at 0x0000015937073460>.
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Entered env context: <rasterio.env.Env object at 0x000001593702C980>
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Path: _UnparsedPath(path='green.tif'), mode: w, driver: GTiff
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Skipped delete for overwrite, dataset does not exist: 'green.tif'
[DEBUG: 2024-06-18 23:25:31 : rasterio._base] Nodata success: 1, Nodata value: 0.000000
[DEBUG: 2024-06-18 23:25:31 : rasterio._base] Nodata success: 1, Nodata value: 0.000000
[DEBUG: 2024-06-18 23:25:31 : rasterio._base] Nodata success: 1, Nodata value: 0.000000
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Exiting env context: <rasterio.env.Env object at 0x000001593702C980>
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Cleared existing <rasterio._env.GDALEnv object at 0x0000015937073460> options
[DEBUG: 2024-06-18 23:25:31 : rasterio._env] Stopped GDALEnv <rasterio._env.GDALEnv object at 0x0000015937073460>.
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Exiting outermost env
[DEBUG: 2024-06-18 23:25:31 : rasterio.env] Exited env context: <rasterio.env.Env object at 0x000001593702C980>
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Window: Window(col_off=0, row_off=14586, width=25294, height=473)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] IO window xoff=0.0 yoff=14586.0 width=25294.0 height=473.0
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Window: Window(col_off=0, row_off=14586, width=25294, height=473)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] IO window xoff=0.0 yoff=14586.0 width=25294.0 height=473.0
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Window: Window(col_off=0, row_off=13260, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] IO window xoff=0.0 yoff=13260.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] Window: Window(col_off=0, row_off=13260, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:31 : rasterio._io] IO window xoff=0.0 yoff=13260.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Window: Window(col_off=0, row_off=0, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] IO window xoff=0.0 yoff=0.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Window: Window(col_off=0, row_off=0, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] IO window xoff=0.0 yoff=0.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Window: Window(col_off=0, row_off=1326, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] IO window xoff=0.0 yoff=1326.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] Window: Window(col_off=0, row_off=1326, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:32 : rasterio._io] IO window xoff=0.0 yoff=1326.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Window: Window(col_off=0, row_off=2652, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] IO window xoff=0.0 yoff=2652.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Window: Window(col_off=0, row_off=2652, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] IO window xoff=0.0 yoff=2652.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Window: Window(col_off=0, row_off=3978, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] IO window xoff=0.0 yoff=3978.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] Window: Window(col_off=0, row_off=3978, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:33 : rasterio._io] IO window xoff=0.0 yoff=3978.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Window: Window(col_off=0, row_off=5304, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] IO window xoff=0.0 yoff=5304.0 width=25294.0 height=1326.0
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Output nodata value read from file: None
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Output nodata values: [None]
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] all_valid: True
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] mask_flags: ([<MaskFlags.all_valid: 1>],)
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Jump straight to _read()
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] Window: Window(col_off=0, row_off=5304, width=25294, height=1326)
[DEBUG: 2024-06-18 23:25:34 : rasterio._io] IO window xoff=0.0 yoff=5304.0 width=25294.0 height=1326.0
Traceback (most recent call last):
  File "C:\Users\45107\PycharmProjects\Satellite\testcode.py", line 24, in <module>
    scn.save_dataset("green-sar", filename="green.tif", fill_value=0, compress=None)
  File "C:\Users\45107\Documents\GitHub\satpy\satpy\scene.py", line 1231, in save_dataset
    return writer.save_dataset(self[dataset_id],
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\Documents\GitHub\satpy\satpy\writers\__init__.py", line 885, in save_dataset
    return self.save_image(img, filename=filename, compute=compute, fill_value=fill_value, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\Documents\GitHub\satpy\satpy\writers\geotiff.py", line 278, in save_image
    return img.save(filename, fformat="tif", driver=driver,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\trollimage\xrimage.py", line 248, in save
    return self.rio_save(filename, fformat=fformat, driver=driver,
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\trollimage\xrimage.py", line 432, in rio_save
    res = da.store(*to_store)
          ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\array\core.py", line 1229, in store
    compute_as_if_collection(Array, store_dsk, map_keys, **kwargs)
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\base.py", line 402, in compute_as_if_collection
    return schedule(dsk2, keys, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\threaded.py", line 90, in get
    results = get_async(
              ^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\local.py", line 512, in get_async
    raise_exception(exc, tb)
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\local.py", line 320, in reraise
    raise exc
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\local.py", line 225, in execute_task
    result = _execute_task(task, data)
             ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\core.py", line 127, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\optimization.py", line 1001, in __call__
    return core.get(self.dsk, self.outkey, dict(zip(self.inkeys, args)))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\core.py", line 157, in get
    result = _execute_task(task, cache)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\core.py", line 127, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\core.py", line 127, in <genexpr>
    return func(*(_execute_task(a, cache) for a in args))
                  ^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\core.py", line 127, in _execute_task
    return func(*(_execute_task(a, cache) for a in args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\miniforge3\Lib\site-packages\dask\utils.py", line 78, in apply
    return func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\45107\Documents\GitHub\satpy\satpy\readers\sar_c_safe.py", line 508, in intp
    return interpolator((grid_y, grid_x))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "interpnd.pyx", line 192, in scipy.interpolate.interpnd.NDInterpolatorBase.__call__
  File "interpnd.pyx", line 333, in scipy.interpolate.interpnd.LinearNDInterpolator._evaluate_double
  File "interpnd.pyx", line 349, in scipy.interpolate.interpnd.LinearNDInterpolator._do_evaluate
  File "interpnd.pyx", line 172, in scipy.interpolate.interpnd.NDInterpolatorBase._find_simplicies
  File "C:\Users\45107\miniforge3\Lib\site-packages\numpy\lib\function_base.py", line 962, in copy
    return array(a, order=order, subok=subok, copy=True)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
numpy.core._exceptions._ArrayMemoryError: Unable to allocate 16.0 GiB for an array with shape (33539844, 64) and data type float64

Environment Info:

  • OS: Windows
  • Satpy Version: 0.49.1 dev
@mraspaud mraspaud self-assigned this Jun 19, 2024
@mraspaud
Copy link
Member

Thanks for reporting this issue. Indeed, the sentinel1 data is very heavy, and I have to take small chunk sizes when working with that data on my laptop, for example:

DASK_ARRAY__CHUNK_SIZE=32MB my_s1_script.py

See more on how to tweak performance here: https://satpy.readthedocs.io/en/stable/faq.html#why-is-satpy-slow-on-my-powerful-machine

@yukaribbba
Copy link
Contributor Author

Thanks for the hint and it does work for me. I usually set DASK_ARRAY__CHUNK_SIZE to 2200*2200 (about 68MB.), which gives me satisfying performance on most other readers except this one. I wish we had more specific tips for adjust the value depending on readers. That'd be better...

@mraspaud
Copy link
Member

I agree, we should give more information when we can. Feel free to create a PR about it if you want.

@yukaribbba
Copy link
Contributor Author

Ok that's on my todo list. For now I'll just close this.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mraspaud mraspaud

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@mraspaud @yukaribbba