Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[DOC] Add new documentation for date index name processor #5398

Merged
merged 27 commits into from
Nov 15, 2023
Merged
Changes from 18 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
9a75346
Add new processor documentation
vagimeli Oct 25, 2023
5b086b8
Add new processor documentation
vagimeli Oct 25, 2023
db4159b
Add new processor documentation
vagimeli Oct 30, 2023
538e1fe
Merge branch 'main' into date-index-processor
vagimeli Oct 30, 2023
7527382
Merge branch 'main' into date-index-processor
vagimeli Oct 30, 2023
2e517e3
Merge branch 'main' into date-index-processor
vagimeli Nov 2, 2023
f02dd15
Merge branch 'main' into date-index-processor
vagimeli Nov 6, 2023
318e1f7
Address tech review feedback
vagimeli Nov 8, 2023
353349d
Address tech review feedback
vagimeli Nov 8, 2023
af6429e
Merge branch 'main' into date-index-processor
vagimeli Nov 8, 2023
6ad1a3f
Revise intro and examples to use date math index name use cases
vagimeli Nov 10, 2023
06e7db4
Add date math example
vagimeli Nov 13, 2023
7ac6b41
Update _ingest-pipelines/processors/date-index-name.md
vagimeli Nov 13, 2023
cfb9a44
Merge branch 'main' into date-index-processor
vagimeli Nov 13, 2023
e044ea5
Add date math example
vagimeli Nov 13, 2023
b85b7d4
Add date math example
vagimeli Nov 13, 2023
83a032c
Update processor example to use date math index expression
vagimeli Nov 13, 2023
1e62b04
Merge branch 'main' into date-index-processor
vagimeli Nov 13, 2023
f5dbdc5
Update processor example to use date math index expression
vagimeli Nov 13, 2023
150dc27
Update _ingest-pipelines/processors/date-index-name.md
vagimeli Nov 14, 2023
d33706a
Update _ingest-pipelines/processors/date-index-name.md
vagimeli Nov 14, 2023
68579e0
Update _ingest-pipelines/processors/date-index-name.md
vagimeli Nov 14, 2023
4ea66b0
Update _ingest-pipelines/processors/date-index-name.md
vagimeli Nov 14, 2023
90111c9
Update _ingest-pipelines/processors/date-index-name.md
vagimeli Nov 14, 2023
e393bdb
Update _ingest-pipelines/processors/date-index-name.md
vagimeli Nov 14, 2023
f8284df
Merge branch 'main' into date-index-processor
vagimeli Nov 14, 2023
bba5595
Fix dead link
vagimeli Nov 15, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
154 changes: 154 additions & 0 deletions _ingest-pipelines/processors/date-index-name.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,154 @@
---
layout: default
title: Date index name
parent: Ingest processors
nav_order: 55
---

# Date index name

The `date_index_name` processor is used to point documents to the correct time-based index based on the date or timestamp field within the document. The processor sets the `_index` metadata field to a [date math]({{site.url}}{{site.baseurl}}/field-types/supported-field-types/date/#date-math) index name expression. Then the processor fetches the date or timestamp from the `field` field in the document being processed and formats it into a date math index name expression. The extracted date, `index_name_prefix` value, and `date_rounding` value are then combined to create the date math index expression. For example, if the `field` field contains the value `2023-10-30T12:43:29.000Z` and `index_name_prefix` is set to `week_index-` and `date_rounding` is set to `w`, then the date math index name expression is `week_index-2023-10-30`. You can use the `date_formats value` field to specify how the date in the date math index expression should be formatted.

The following is the syntax for the `date_index_name` processor:

```json
{
"date_index_name": {
"field": "your_date_field or your_timestamp_field",
"date_rounding": "rounding_value"
}
}
```
{% include copy-curl.html %}

## Configuration parameters

The following table lists the required and optional parameters for the `date_index_name` processor.

Parameter | Required/Optional | Description |
|-----------|-----------|-----------|
`field` | Required | The date or timestamp field in the incoming document. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). |
`date_rounding` | Required | The rounded date format within the index name . Valid values are `y` (year), `M` (month), `w` (week), `d` (day), `h` (hour), `m` (minute), and `s` (second). |
`date_formats` | Optional | An array of date formats to parse the date or timestamp field. Default is `yyyy-MM-dd'T'HH:mm:ss.SSSXX`. |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
`index_name_format` | Optional | The date format. Default is empty value. | Supports [template snippets]({site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
`index_name_prefix` | Optional | The index name prefix to add before the date. Supports [template snippets]({site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets).
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
`description` | Optional | A brief description of the processor. |
`if` | Optional | A condition for running this processor. |
`ignore_failure` | Optional | If set to `true`, failures are ignored. Default is `false`. |
`locale` | `locale` | Optional | The locale to use when parsing the date. Default is `ENGLISH`. Supports [template snippets]({site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved
`on_failure` | Optional | A list of processors to run if the processor fails. |
`tag` | Optional | An identifier tag for the processor. Useful for debugging to distinguish between processors of the same type. |
`timezone` | Optional | The time zone to use when parsing the date. Default is `UTC`. Supports [template snippets]({site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). |
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

## Using the processor

Follow these steps to use the processor in a pipeline.

**Step 1: Create a pipeline.**

The following query creates a pipeline, named `date-index-name1`, that uses the `date_index_name` processor to index logs into monthly indexes:

```json
PUT /_ingest/pipeline/date-index-name1
{
"description": "Create weekly index pipeline",
"processors": [
{
"date_index_name": {
"field": "date_field",
"index_name_prefix": "week_index-",
"date_rounding": "w",
"date_formats": ["YYYY-MM-DD"]
}
}
]
}
```
{% include copy-curl.html %}

**Step 2 (Optional): Test the pipeline.**

It is recommended that you test your pipeline before you ingest documents.
{: .tip}

To test the pipeline, run the following query:

```json
POST _ingest/pipeline/date-index-name1/_simulate
{
"docs": [
{
"_index": "testindex1",
"_id": "1",
"_source": {
"date_field": "2023-10-30"
}
}
]
}
```
{% include copy-curl.html %}

#### Response

The following example response confirms that the pipeline is working as expected:

```json
{
"docs": [
{
"doc": {
"_index": "<week_index-{2023-10-01||/w{yyyy-MM-dd|UTC}}>",
"_id": "1",
"_source": {
"date_field": "2023-10-30"
},
"_ingest": {
"timestamp": "2023-11-13T18:23:10.408593092Z"
}
}
}
]
}
```

**Step 3: Ingest a document.**

The following query ingests a document into an index named `testindex1`:

```json
PUT testindex1/_doc/1?pipeline=date-index-name1
{
"date_field": "2023-10-30"
}
```
{% include copy-curl.html %}

#### Response

The request indexes the document into the index `week_index-2023-10-23` and will continue to index documents into weekly indexes because the pipeline rounds by week.
vagimeli marked this conversation as resolved.
Show resolved Hide resolved

```json
{
"_index": "week_index-2023-10-30",
"_id": "1",
"_version": 4,
"result": "updated",
"_shards": {
"total": 2,
"successful": 1,
"failed": 0
},
"_seq_no": 3,
"_primary_term": 1
}
```

**Step 4 (Optional): Retrieve the document.**

To retrieve the document, run the following query:

```json
GET week_index-2023-10-30/_doc/1
```
{% include copy-curl.html %}
vagimeli marked this conversation as resolved.
Show resolved Hide resolved