Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/elasticsearch] [chore] add docs for logs_dynamic_id #37472

22 changes: 21 additions & 1 deletion exporter/elasticsearchexporter/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,7 +143,7 @@ This can be customised through the following settings:
- `date_format`(default=`%Y.%m.%d`): Time format (based on strftime) to generate the second part of the Index name.

- `logs_dynamic_id` (optional): Dynamically determines the document ID to be used in Elasticsearch based on a log record attribute.
- `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document.
- `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document. See [Setting a document id dynamically](#setting-a-document-id-dynamically).

### Elasticsearch document mapping

Expand Down Expand Up @@ -349,6 +349,26 @@ Otherwise, it is mapped to an empty string ("").

In case the record contains `timestamp`, this value is used. Otherwise, the `observed timestamp` is used.

## Setting a document id dynamically

The `logs_dynamic_id` setting allows users to set the document ID dynamically based on a log record attribute.
Besides the ability to control the document ID, this setting also works as a deduplication mechanism, as Elasticsearch will refuse to index a document with the same ID.

The log record attribute `elasticsearch.document_id` can be set explicitly by a processor based on the log record.

As an example, the `transform` processor can create this attribute dynamically:

```yaml
processors:
transform/es-doc-id:
error_mode: ignore
log_statements:
- context: log
condition: attributes["event_name"] != null && attributes["event_creation_time"] != null
statements:
- set(attributes["elasticsearch.document_id"], Concat(["log", attributes["event_name"], attributes["event_creation_time"], "-"))
```

## Known issues

### version_conflict_engine_exception
Expand Down