Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Split binary and array based reduction #123

Merged
merged 8 commits into from
Jan 20, 2020
Merged

Split binary and array based reduction #123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
c4ba859
Remove support for parameter `binary` in `aggregate_temporal` and `re…
m-mohr Jan 16, 2020
88378c6
merge_cubes works with binary reduction operators instead of list-bas…
m-mohr Jan 16, 2020
4277ec5
Added binary variants for reduce_dimension and aggregate_polygon, sli…
m-mohr Jan 16, 2020
c906d8e
Merge branch 'draft' into issue-94
m-mohr Jan 16, 2020
6e203b6
Fix process names
m-mohr Jan 16, 2020
55df485
Integrated suggestions from @mkadunc
m-mohr Jan 16, 2020
4571cae
Apply suggestions from code review
m-mohr Jan 16, 2020
6ba296b
Merge branch 'draft' into issue-94
m-mohr Jan 20, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 8 additions & 4 deletions CHANGELOG.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,8 +7,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
## Unreleased

### Added
- New processes:
- Processes:
- `add`
- `aggregate_polygon_binary`
- `all`
- `any`
- `array_apply`
Expand All @@ -20,6 +21,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `load_uploaded_files`
- `mask_polygon`
- `reduce_dimension`
- `reduce_dimension_binary`
- `rename_labels`
- Added further examples

Expand All @@ -40,6 +42,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- Parameter `bands` accepts common band names. [#77](https://github.com/Open-EO/openeo-processes/issues/77)
- Parameter `properties`: Callback parameter `value` renamed to `x`.
- `mask`: Clarifies behavior for missing dimensions in the mask. [#55](https://github.com/Open-EO/openeo-processes/issues/55)
- `merge_cubes` works with binary reduction operators instead of list-based reducers. [#94](https://github.com/Open-EO/openeo-processes/issues/94)
- `ndvi` and `normalized_difference`: Rewrite of the processes with a completely new behavior. [#60](https://github.com/Open-EO/openeo-processes/issues/60)
- `resample_spatial`: Default value of parameter `align` changed from `lower-left` to `upper-left`. [#61](https://github.com/Open-EO/openeo-processes/issues/61)
- The following operations work on two values instead on a sequence of values: `and`, `divide`, `multiply`, `or`, `subtract`, `xor`. [#85](https://github.com/Open-EO/openeo-processes/issues/85)
Expand All @@ -50,12 +53,13 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
- `filter_bbox`, `load_collection`, `resample_spatial`: PROJ definitions are deprecated in favor of EPSG codes, WKT2 and PROJJSON. [#58](https://github.com/Open-EO/openeo-processes/issues/58)

### Removed
- The following operations don't support `ignore_nodata` any longer: `and`, `divide`, `multiply`, `or`, `subtract`, `xor`. [#85](https://github.com/Open-EO/openeo-processes/issues/85)
- Removed support for vector data cubes, except for the processes `aggregate_poylgon` and `save_result`. [#68](https://github.com/Open-EO/openeo-processes/issues/68)
- The following processes don't support `ignore_nodata` any longer: `and`, `divide`, `multiply`, `or`, `subtract`, `xor`. [#85](https://github.com/Open-EO/openeo-processes/issues/85)
- The following processes don't support `binary` any longer: `aggregate_temporal`, `merge_cubes`, `resample_cube_temporal`. [#94](https://github.com/Open-EO/openeo-processes/issues/94)
- Support for vector data cubes, except for the processes `aggregate_poylgon` and `save_result`. [#68](https://github.com/Open-EO/openeo-processes/issues/68)
- `aggregate_polygon`: Doesn't allow returning a GeoJSON any longer.
- `filter_temporal` and `load_collection`: Temporal extents don't support time-only intervals any longer. [#88](https://github.com/Open-EO/openeo-processes/issues/88)
- `mask`: The mask parameter doesn't accept vectors (polygons) any longer. Use process `mask_polygon` instead. [#110](https://github.com/Open-EO/openeo-processes/issues/110)
- Removed processes:
- Processes:
- `find_collections`: Use `load_collection` and manual data discovery through the clients. [API#52](https://github.com/Open-EO/openeo-api/issues/52)
- `filter`: Use `filter_labels` instead.
- `output`: Use `debug` instead.
Expand Down
65 changes: 14 additions & 51 deletions aggregate_polygon.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,18 +1,16 @@
{
"id": "aggregate_polygon",
"summary": "Compute zonal statistics for polygons",
"description": "Aggregates zonal statistics for one or multiple polygons over the spatial dimensions.\n\nThe process considers all pixels for which the point at the pixel center intersects with the corresponding polygon (as defined in the Simple Features standard by the OGC).\n\nThe data cube must have been reduced to only contain two raster dimensions and a third dimension the values are aggregated for, for example the temporal dimension to get a time series. Otherwise this process fails with the `TooManyDimensions` error.\n\nThe number of total and valid pixels is returned together with the calculated values.",
"summary": "Zonal statistics for polygons",
"description": "Aggregates zonal statistics for one or multiple polygons over the spatial dimensions. This process passes a list of values to the reducer. In contrast, ``aggregate_polygon_binary()`` passes two values, which may be better suited for UDFs.\n\nThe process considers all pixels for which the point at the pixel center intersects with the corresponding polygon (as defined in the Simple Features standard by the OGC).\n\nThe data cube must have been reduced to only contain two spatial dimensions and a third dimension the values are aggregated for, for example the temporal dimension to get a time series. Otherwise this process fails with the `TooManyDimensions` error.\n\nThe number of total and valid pixels is returned together with the calculated values.",
"categories": [
"cubes",
"aggregate & resample"
],
"experimental": true,
"parameter_order": [
"data",
"polygons",
"reducer",
"name",
"binary"
"name"
],
"parameters": {
"data": {
Expand All @@ -38,38 +36,20 @@
"required": true
},
"reducer": {
"description": "A reducer to be applied on all values of each geometry. The reducer may be a single process such as ``mean()`` or consist of multiple sub-processes. By default, a reducer must accept an array as input. The process can also work on two values by setting the parameter `binary` to `true`.",
"schema": [
{
"title": "Unary behaviour",
"description": "Passes an array to the reducer.",
"type": "object",
"subtype": "process-graph",
"parameters": {
"data": {
"description": "An array with elements of any data type.",
"type": "array",
"items": {
"description": "Any data type."
}
}
}
},
{
"title": "Binary behaviour",
"description": "Passes two values to the reducer.",
"type": "object",
"subtype": "process-graph",
"parameters": {
"x": {
"description": "The first value. Any data type could be passed."
},
"y": {
"description": "The second value. Any data type could be passed."
"description": "A reducer to be applied on all values of each geometry. A reducer is a single process such as ``mean()`` or a set of processes, which computes a single value for a list of values, see the category 'reducer' for such processes.",
"schema": {
"type": "object",
"subtype": "process-graph",
"parameters": {
"data": {
"description": "An array with elements of any data type.",
"type": "array",
"items": {
"description": "Any data type."
}
}
}
],
},
"required": true
},
"name": {
Expand All @@ -78,13 +58,6 @@
"type": "string"
},
"default": "result"
},
"binary": {
"description": "Specifies whether the process should pass two values to the reducer or a list of values (default).\n\nIf the process passes two values, the reducer must be both associative and commutative as the execution may be executed in parallel and therefore the order of execution is arbitrary.\n\nThis parameter is especially useful for UDFs passed as reducers. Back-ends may still optimize and parallelize processes that work on list of values.",
"schema": {
"type": "boolean"
},
"default": false
}
},
"returns": {
Expand All @@ -100,11 +73,6 @@
}
},
"links": [
{
"href": "https://github.com/Open-EO/openeo-processes/issues/2",
"rel": "experimental",
"title": "More information about the experimental status of the process"
},
{
"href": "https://open-eo.github.io/openeo-api/glossary/#aggregation-and-resampling",
"rel": "about",
Expand All @@ -114,11 +82,6 @@
"href": "http://www.opengeospatial.org/standards/sfa",
"rel": "about",
"title": "Simple Features standard by the OGC"
},
{
"rel": "about",
"href": "https://en.wikipedia.org/wiki/Reduction_Operator",
"title": "Background information on reduction operators (binary reducers) by Wikipedia"
}
]
}
92 changes: 92 additions & 0 deletions aggregate_polygon_binary.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
{
"id": "aggregate_polygon_binary",
"summary": "Zonal statistics for polygons using binary aggregation",
"description": "Aggregates zonal statistics for one or multiple polygons over the spatial dimensions. This process passes two values to the reducer. In contrast, ``aggregate_polygon()`` passes a list of values.\n\nThe process considers all pixels for which the point at the pixel center intersects with the corresponding polygon (as defined in the Simple Features standard by the OGC).\n\nThe data cube must have been reduced to only contain two raster dimensions and a third dimension the values are aggregated for, for example the temporal dimension to get a time series. Otherwise this process fails with the `TooManyDimensions` error.\n\nThe number of total and valid pixels is returned together with the calculated values.",
"categories": [
"cubes",
"aggregate & resample"
],
"experimental": true,
"parameter_order": [
"data",
"polygons",
"reducer",
"name"
],
"parameters": {
"data": {
"description": "A data cube.",
"schema": {
"type": "object",
"subtype": "raster-cube"
},
"required": true
},
"polygons": {
"description": "One or more polygons to calculate zonal statistics for. Either specified as GeoJSON or vector data cube.\n\nFor GeoJSON this can be one of the following GeoJSON types:\n\n* A `Polygon` geometry,\n* a `GeometryCollection` containing Polygons,\n* a `Feature` with a `Polygon` geometry or\n* a `FeatureCollection` containing `Feature`s with a `Polygon` geometry.",
"schema": [
{
"type": "object",
"subtype": "geojson"
},
{
"type": "object",
"subtype": "vector-cube"
}
],
"required": true
},
"reducer": {
"description": "A reduction operator to be applied consecutively on tuples of values. It must be both associative and commutative as the execution may be executed in parallel and therefore the order of execution is arbitrary. The reduction operator may be a single process such as ``multiply()`` or consist of multiple sub-processes.",
"schema": {
"type": "object",
"subtype": "process-graph",
"parameters": {
"x": {
"description": "The first value. Any data type could be passed."
},
"y": {
"description": "The second value. Any data type could be passed."
}
}
},
"required": true
},
"name": {
"description": "The new dimension name to be used for storing the results. Defaults to `result`.",
"schema": {
"type": "string"
},
"default": "result"
}
},
"returns": {
"description": "A vector data cube with the computed results. The vector data cube implicitly gets restricted to the bounds of the polygons as if ``filter_polygon()`` would have been used with the same values for the corresponding parameters immediately before this process.\n\nThe computed value is stored in dimension with the name that was specified in the parameter `name`.\n\nThe computation also stores information about the total count of pixels (valid + invalid pixels) and the number of valid pixels (see ``is_valid()``) in each geometry. These values are stored as attributes of the result value with the attribute names `total_count` and `valid_count`.",
"schema": {
"type": "object",
"subtype": "vector-cube"
}
},
"exceptions": {
"TooManyDimensions": {
"message": "The number of dimensions must be reduced to three for 'aggregate_polygon'."
}
},
"links": [
{
"href": "https://open-eo.github.io/openeo-api/glossary/#aggregation-and-resampling",
"rel": "about",
"title": "Aggregation explained in the openEO glossary"
},
{
"href": "http://www.opengeospatial.org/standards/sfa",
"rel": "about",
"title": "Simple Features standard by the OGC"
},
{
"rel": "about",
"href": "https://en.wikipedia.org/wiki/Reduction_Operator",
"title": "Background information on reduction operators (binary reducers) by Wikipedia"
}
]
}
55 changes: 12 additions & 43 deletions aggregate_temporal.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,8 +12,7 @@
"intervals",
"reducer",
"labels",
"dimension",
"binary"
"dimension"
],
"parameters": {
"data": {
Expand Down Expand Up @@ -87,38 +86,20 @@
"required": true
},
"reducer": {
"description": "A reducer to be applied on all values along the specified dimension. The reducer may be a single process such as ``mean()`` or consist of multiple sub-processes. By default, a reducer must accept an array as input. The process can also work on two values by setting the parameter `binary` to `true`.",
"schema": [
{
"title": "Unary behaviour",
"description": "Passes an array to the reducer.",
"type": "object",
"subtype": "process-graph",
"parameters": {
"data": {
"description": "An array with elements of any data type.",
"type": "array",
"items": {
"description": "Any data type."
}
}
}
},
{
"title": "Binary behaviour",
"description": "Passes two values to the reducer.",
"type": "object",
"subtype": "process-graph",
"parameters": {
"x": {
"description": "The first value. Any data type could be passed."
},
"y": {
"description": "The second value. Any data type could be passed."
"description": "A reducer to be applied on all values along the specified dimension. A reducer is a single process such as ``mean()`` or a set of processes, which computes a single value for a list of values, see the category 'reducer' for such processes.",
"schema": {
"type": "object",
"subtype": "process-graph",
"parameters": {
"data": {
"description": "An array with elements of any data type.",
"type": "array",
"items": {
"description": "Any data type."
}
}
}
],
},
"required": true
},
"labels": {
Expand Down Expand Up @@ -159,13 +140,6 @@
]
},
"default": null
},
"binary": {
"description": "Specifies whether the process should pass two values to the reducer or a list of values (default).\n\nIf the process passes two values, the reducer must be both associative and commutative as the execution may be executed in parallel and therefore the order of execution is arbitrary.\n\nThis parameter is especially useful for UDFs passed as reducers. Back-ends may still optimize and parallelize processes that work on list of values.",
"schema": {
"type": "boolean"
},
"default": false
}
},
"returns": {
Expand Down Expand Up @@ -259,11 +233,6 @@
"href": "https://open-eo.github.io/openeo-api/glossary/#aggregation-and-resampling",
"rel": "about",
"title": "Aggregation explained in the openEO glossary"
},
{
"rel": "about",
"href": "https://en.wikipedia.org/wiki/Reduction_Operator",
"title": "Background information on reduction operators (binary reducers) by Wikipedia"
}
]
}
2 changes: 1 addition & 1 deletion apply_dimension.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
{
"id": "apply_dimension",
"summary": "Apply a process to pixels along a dimension",
"description": "Applies a process to all pixel values along a dimension of a raster data cube. For example, if the temporal dimension is specified the process will work on a time series of pixel values.\n\nThe process ``reduce_dimension()`` also applies a process to pixel values along a dimension, but removes the dimension afterwards. The process ``apply()`` applies a process to each pixel value in the data cube.\n\nThe target dimension is the source dimension if not specified otherwise in the `target_dimension` parameter. The pixel values in the target dimension get replaced by the computed pixel values. The name, type and reference system are preserved.\n\nThe dimension labels are preserved when the target dimension is the source dimension and the number of pixel values in the source dimension is equal to the number of values computed by the process. Otherwise, the dimension labels will be incrementing integers starting from zero, which can be changed using ``rename_labels()`` afterwards. The number of labels will equal to the number of values computed by the process.",
"description": "Applies a process to all pixel values along a dimension of a raster data cube. For example, if the temporal dimension is specified the process will work on a time series of pixel values.\n\nThe process ``reduce_dimension()`` also applies a process to pixel values along a dimension, but drops the dimension afterwards. The process ``apply()`` applies a process to each pixel value in the data cube.\n\nThe target dimension is the source dimension if not specified otherwise in the `target_dimension` parameter. The pixel values in the target dimension get replaced by the computed pixel values. The name, type and reference system are preserved.\n\nThe dimension labels are preserved when the target dimension is the source dimension and the number of pixel values in the source dimension is equal to the number of values computed by the process. Otherwise, the dimension labels will be incrementing integers starting from zero, which can be changed using ``rename_labels()`` afterwards. The number of labels will equal to the number of values computed by the process.",
"categories": [
"cubes"
],
Expand Down
Loading