You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Is your feature request related to a problem? Please describe.
The xcube function xcube.core.geom.rasterize_features() seems to eagerly load large numpy arrays into memory often resulting in an OOM error.
xcube.core.geom.get_geometry_mask() may be affected too as it is effectively a wrapper for rasterio.features.geometry_mask() which may not use Dask at all (to be verified!).
Describe the solution you'd like
Instead they should produce lazy dask arrays.
The text was updated successfully, but these errors were encountered:
...which may not use Dask at all (to be verified!).
Verified, it does not use Dask at all and returns a numpy array. From rasterio docstring:
This function requires significant memory resources. The shapes
iterator will be materialized to a Python list and another C copy of
that list will be made. The `out` array will be copied and
additional temporary raster memory equal to 2x the smaller of `out`
data or GDAL's max cache size (controlled by GDAL_CACHEMAX, default
is 5% of the computer's physical memory) is required.
Is your feature request related to a problem? Please describe.
The xcube function
xcube.core.geom.rasterize_features()
seems to eagerly load large numpy arrays into memory often resulting in an OOM error.xcube.core.geom.get_geometry_mask()
may be affected too as it is effectively a wrapper forrasterio.features.geometry_mask()
which may not use Dask at all (to be verified!).Describe the solution you'd like
Instead they should produce lazy dask arrays.
The text was updated successfully, but these errors were encountered: