diff --git a/exporter/elasticsearchexporter/README.md b/exporter/elasticsearchexporter/README.md index d78d5cba8421..0c5b4b0dbd52 100644 --- a/exporter/elasticsearchexporter/README.md +++ b/exporter/elasticsearchexporter/README.md @@ -143,7 +143,7 @@ This can be customised through the following settings: - `date_format`(default=`%Y.%m.%d`): Time format (based on strftime) to generate the second part of the Index name. - `logs_dynamic_id` (optional): Dynamically determines the document ID to be used in Elasticsearch based on a log record attribute. - - `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document. + - `enabled`(default=false): Enable/Disable dynamic ID for log records. If `elasticsearch.document_id` exists and is not an empty string in the log record attributes, it will be used as the document ID. Otherwise, the document ID will be generated by Elasticsearch. The attribute `elasticsearch.document_id` is removed from the final document. See [Setting a document id dynamically](#setting-a-document-id-dynamically). ### Elasticsearch document mapping @@ -349,6 +349,26 @@ Otherwise, it is mapped to an empty string (""). In case the record contains `timestamp`, this value is used. Otherwise, the `observed timestamp` is used. +## Setting a document id dynamically + +The `logs_dynamic_id` setting allows users to set the document ID dynamically based on a log record attribute. +Besides the ability to control the document ID, this setting also works as a deduplication mechanism, as Elasticsearch will refuse to index a document with the same ID. + +The log record attribute `elasticsearch.document_id` can be set explicitly by a processor based on the log record. + +As an example, the `transform` processor can create this attribute dynamically: + +```yaml +processors: + transform/es-doc-id: + error_mode: ignore + log_statements: + - context: log + condition: attributes["event_name"] != null && attributes["event_creation_time"] != null + statements: + - set(attributes["elasticsearch.document_id"], Concat(["log", attributes["event_name"], attributes["event_creation_time"], "-")) +``` + ## Known issues ### version_conflict_engine_exception