Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Geohex aggregation on geo_shape field #91956

Merged
merged 21 commits into from
Dec 22, 2022
Merged

Conversation

iverase
Copy link
Contributor

@iverase iverase commented Nov 28, 2022

Implementation of geohex aggregation on geo_shape fields. It projects the H3 grid system into a equirectangular projection (see H3CartesianUtil) and uses it to rasterise the geometries indexed on a geo_shape field. One important side effect of this projection is that the relationship between the cell containing a pole and its children maybe lost so it requires special treatment.

Most the rasterization logic is inside AbstractGeoHexGridTiler. It is important to understand that children of an H3 bin are not fully contain in the parent and an H3 bin intersects other h3 bin in the next level which are not in the children set. Those bins can be found using H3 method H3.h3ToNoChildrenIntersecting. In order to make sure we visit all necessary cells we add the following call when navigating down the H3 hierarchy:

// H3 cells do intersect with other cells that are not part of the children cells. If the parent cell of those
// cells is disjoint, they will not be visited, therefore visit them here.
for (long child : H3.h3ToNoChildrenIntersecting(h3)) {
   if (relateTile(geoValue, H3.h3ToParent(child)) == GeoRelation.QUERY_DISJOINT) {
      valueIndex = setValuesByRecursion(values, geoValue, child, precision + 1, valueIndex);
   }
}

@iverase iverase added >feature :Analytics/Geo Indexing, search aggregations of geo points and shapes v8.7.0 labels Nov 28, 2022
@iverase iverase marked this pull request as draft November 28, 2022 09:27
@elasticsearchmachine
Copy link
Collaborator

Hi @iverase, I've created a changelog YAML for you.

@iverase iverase marked this pull request as ready for review December 5, 2022 14:46
@elasticsearchmachine elasticsearchmachine added the Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) label Dec 5, 2022
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-analytics-geo (Team:Analytics)

@iverase
Copy link
Contributor Author

iverase commented Dec 13, 2022

@elasticmachine update branch

Copy link
Contributor

@craigtaverner craigtaverner left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Mostly I like this a lot. I have, however, many comments, questions and suggestions. I'm approving this as a very good first step, with the expectation that we do more work before making this feature available. In particular increasing test coverage and fixing any bugs found by that process.

@iverase iverase merged commit d448012 into elastic:main Dec 22, 2022
@iverase iverase deleted the GeohexGeoshape branch December 22, 2022 17:55
} else {
// points are solved on the spherical geometry
final H3LatLonGeometry geometry = new H3LatLonGeometry(id);
if (fieldType instanceof GeoPointFieldMapper.GeoPointFieldType pointFieldType) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could just have used the shared interface GeoShapeQueryable

nreese added a commit to elastic/kibana that referenced this pull request Jan 5, 2023
…on/apis/maps/get_grid_tile.js (#148346)

Fixes #148122

elastic/elasticsearch#91956 added Geohex
aggregation on geo_shape field and also changed how resolutions are
mapped on vector tiles. 85264a33fffffff has resolution 5 and
84264a3ffffffff has resolution 4. The new resolution mappings are in
https://github.com/elastic/elasticsearch/blob/main/x-pack/plugin/vector-tile/src/main/java/org/elasticsearch/xpack/vectortile/rest/GridAggregation.java#L122.
Resolution mapping was changed to avoid hitting bucket limits which are
easier to hit with large shapes and hex binning on geo_shape.

Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
craigtaverner added a commit to craigtaverner/elasticsearch that referenced this pull request Jan 17, 2023
The feature to add support for geohex_grid aggregations over geo_shape
fields was added in elastic#91956.
This is the associated documentation for that.
craigtaverner added a commit that referenced this pull request Jan 24, 2023
* Documentation for geohex_grid over geo_shape

The feature to add support for geohex_grid aggregations over geo_shape
fields was added in #91956.
This is the associated documentation for that.

* Update docs/reference/aggregations/bucket/geohexgrid-aggregation.asciidoc

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>

* Fix explanation for geo_point vs geo_shape proj

When aggregating geohex over geoshape we use requirectangular because
underlying lucene index indexes and searches the polygons in that way.

* Correct spelling

According to grammarly, "therefor" is not an alternative spelling
of "therefore". We should use the conjunctive form here.

See https://www.grammarly.com/blog/therefore-vs-therefor/

Co-authored-by: Abdon Pijpelink <abdon.pijpelink@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
:Analytics/Geo Indexing, search aggregations of geo points and shapes >feature release highlight Team:Analytics Meta label for analytical engine team (ESQL/Aggs/Geo) v8.7.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants