[exporter/elasticsearch] Encode geo location attributes as geo_point in Elasticsearch

[carsonip]

Component(s)

exporter/elasticsearch

Is your feature request related to a problem? Please describe.

Geo location attributes geo.location.lat and geo.location.lon (SemConv) are stored as separate attributes in ES.

Describe the solution you'd like

They should be mapped as geo_point.

Describe alternatives you've considered

No response

Additional context

No response

[carsonip]

/label -needs-triage

[github-actions]

Pinging code owners:

• exporter/elasticsearch: @JaredTan95 @carsonip @lahsivjar

See Adding Labels via Comments if you do not have permissions to add labels yourself.

[carsonip]

@felixbarny @gregkalapos

• should geo.location.lon and geo.location.lat be removed after creating a new geo.location from the two of them?
• WDYT about using the format Geopoint expressed as a [Well-Known Text](https://docs.opengeospatial.org/is/12-063r5/12-063r5.html) POINT with the format: "POINT(lon lat)" in https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html ?

[felixbarny]

should geo.location.lon and geo.location.lat be removed after creating a new geo.location from the two of them?

Hm, I think so, yes. We're not losing information by combining them.

WDYT about using the format Geopoint expressed as a Well-Known Text POINT with the format: "POINT(lon lat)" in https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html ?

An advantage of that could be that it's easier to auto-detect geo fields without having to rely on naming conventions. But that would also require SemConv to adopt that format. The disadvantage is that it's more heavy in terms of bandwidth and storage, especially for backends that don't have native support for geo points. They'd need to store it as a string rather than an array of double values.

So overall, I'd lean towards storing it similar to Well-Known-Text, but as a double array ([lon, lat]).

We could then either introduce a naming convention (*.geo.location) or introduce type hints in OTLP.

[carsonip]

We're not losing information by combining them.

I haven't tried, but will querying for geo.location.lon return the same value after merging lon and lat into geo.location?

[carsonip]

I haven't tried, but will querying for geo.location.lon return the same value after merging lon and lat into geo.location?

Answering my own question, the answer is yes.

{
        "_index": ".ds-logs-foo.otel-default-2024.11.29-000001",
        "_id": "tw3Dd5MBgsycazFvRM-K",
        "_score": 1,
        "_source": {
          "@timestamp": "2024-11-29T11:50:00.000Z",
          "attributes": {
            "geo.location": {
              "lat": 2.1999999694526196,
              "lon": 1.0999999847263098
            }
          },
          "data_stream": {
            "type": "logs"
          }
        },
        "fields": {
          "geo.location": [
            {
              "coordinates": [
                1.0999999847263098,
                2.1999999694526196
              ],
              "type": "Point"
            }
          ],
          "geo.location.lon": [
            1.1
          ],
          "geo.location.lat": [
            2.2
          ]
        }
      }

[gregkalapos]

@felixbarny @gregkalapos

• should geo.location.lon and geo.location.lat be removed after creating a new geo.location from the two of them?

Yeah +1 on not storing those - also nice to see, we still can query only one if needed.

• WDYT about using the format Geopoint expressed as a [Well-Known Text](https://docs.opengeospatial.org/is/12-063r5/12-063r5.html) POINT with the format: "POINT(lon lat)" in https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html ?

In which layer do you mean? If we really want to introduce that to attributes, then that'd be a new data type, so I'm with Felix on this, that'd be very hard with little benefit. But the ES exporter is still free to ingest it in whatever format we feel best from https://www.elastic.co/guide/en/elasticsearch/reference/current/geo-point.html.

[carsonip]

In which layer do you mean?

I've ended up implemented using [lon, lat] in #36594 . Felix made a good call about storage size.