Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Ingest error metadata fields #75653

Merged
merged 1 commit into from
Jul 23, 2021
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 30 additions & 4 deletions docs/reference/ingest.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -323,7 +323,7 @@ template's index pattern should match `logs-<dataset-name>-*`.
--
You can create this template using {kib}'s <<manage-index-templates,**Index
Management**>> feature or the <<indices-put-template,create index template
API>>.
API>>.

For example, the following request creates a template matching `logs-my_app-*`.
The template uses a component template that contains the
Expand Down Expand Up @@ -550,7 +550,7 @@ PUT _ingest/pipeline/my-pipeline
"description": "Use geo_point dynamic template for address field",
"field": "_dynamic_templates",
"value": {
"address": "geo_point"
"address": "geo_point"
}
}
}
Expand All @@ -560,8 +560,8 @@ PUT _ingest/pipeline/my-pipeline

The set processor above tells ES to use the dynamic template named `geo_point`
for the field `address` if this field is not defined in the mapping of the index
yet. This processor overrides the dynamic template for the field `address` if
already defined in the bulk request, but has no effect on other dynamic
yet. This processor overrides the dynamic template for the field `address` if
already defined in the bulk request, but has no effect on other dynamic
templates defined in the bulk request.

WARNING: If you <<create-document-ids-automatically,automatically generate>>
Expand Down Expand Up @@ -717,6 +717,32 @@ PUT _ingest/pipeline/my-pipeline
----
// TEST[s/\.\.\./{"lowercase": {"field":"my-keyword-field"}}/]

Additional information about the pipeline failure may be available in the
document metadata fields `on_failure_message`, `on_failure_processor_type`,
`on_failure_processor_tag`, and `on_failure_pipeline`. These fields are
accessible only from within an `on_failure` block.

The following example uses the error metadata fields to provide additional
information on the document about the failure.

[source,console]
----
PUT _ingest/pipeline/my-pipeline
{
"processors": [ ... ],
"on_failure": [
{
"set": {
"description": "Record error information",
"field": "error_information",
"value": "Processor {{ _ingest.on_failure_processor_type }} with tag {{ _ingest.on_failure_processor_tag }} in pipeline {{ _ingest.on_failure_pipeline }} failed with message {{ _ingest.on_failure_message }}"
}
}
]
}
----
// TEST[s/\.\.\./{"lowercase": {"field":"my-keyword-field"}}/]

[discrete]
[[conditionally-run-processor]]
=== Conditionally run a processor
Expand Down