[DOC] Add kv processor content gap doc #5781

vagimeli · 2023-12-04T22:46:09Z

Description

Adds documentation about kv processor; addresses content gap

Issues Resolved

#5651

Checklist

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license and subject to the Developers Certificate of Origin.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli · 2023-12-05T16:09:11Z

@reta @gaobinlong Please take a look at the draft kv processor documentation. We particularly need your input for a real-life example of a kv processor ingest pipeline (see the section Using the processor steps 1-3). Thank you for your review!

For reference, if you'd like to see recent documentation for other processors, go to https://opensearch.org/docs/latest/ingest-pipelines/processors/index-processors/.

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

_ingest-pipelines/processors/kv.md

reta · 2023-12-05T19:55:24Z

We particularly need your input for a real-life example of a kv processor ingest pipeline (see the section Using the processor steps 1-3).

@gaobinlong do you mind providing those, I believe you should have them ready, thank you

_ingest-pipelines/processors/kv.md

Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli · 2023-12-08T21:08:47Z

@reta @gaobinlong Thank you for your review. I've addressed your feedback. I'll move this forward for doc and editorial review and then get the PR merged. Thank you!

hdhalter

Looks good! Just a few suggestions/nits.

_ingest-pipelines/processors/kv.md

hdhalter · 2023-12-11T22:45:37Z

_ingest-pipelines/processors/kv.md

+
+The following table lists the required and optional parameters for the `lowercase` processor.
+
+| Name  | Required/Optional  | Description  |


For consistency with the other processors, should the headings be: Parameter | Required | Description |

(Although, sometimes "Required/Optional" is the heading. See https://opensearch.org/docs/latest/ingest-pipelines/processors/date-index-name/ )

We are updating the tables to now read Required/Optional. The first round originally said Required, but those need updating. I'll sync them this week. Good comment. Thanks!

_ingest-pipelines/processors/kv.md

_ingest-pipelines/processors/lowercase.md

Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli · 2023-12-11T23:10:50Z

@hdhalter I accepted your suggestions. I'll update the first round of processors that used Required instead of Required/Optional and Name instead of Parameter. Thanks for the review.

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

natebower

@vagimeli Please see my comments and changes and let me know if you have any questions. Thanks!

_ingest-pipelines/processors/kv.md

natebower · 2023-12-12T13:31:49Z

_ingest-pipelines/processors/kv.md

+
+# KV processor
+
+The `kv` processor automatically extracts specific event fields or messages that are in a`key=value` format. This structured format organizes your data by grouping it together based on keys and values. It's helpful for analyzing, visualizing, and using data such as user behavior analytics, performance optimizations, or security investigations. 


Last sentence: I suggest just "It's helpful in use cases such as user behavior analytics...". Otherwise, "...and using data, such as user behavior analytics, performance optimization, or security investigation data."

natebower · 2023-12-12T13:32:22Z

_ingest-pipelines/processors/kv.md

+The `kv` processor automatically extracts specific event fields or messages that are in a`key=value` format. This structured format organizes your data by grouping it together based on keys and values. It's helpful for analyzing, visualizing, and using data such as user behavior analytics, performance optimizations, or security investigations. 
+
+## Example
+The following is the syntax for the `kv` processor: 


Please add a line between the heading and the following text.

natebower · 2023-12-12T13:33:37Z

_ingest-pipelines/processors/kv.md

+
+| Parameter  | Required/Optional  | Description  |
+|---|---|---|
+`field`  | Required  | The name of the field that contains the data to be parsed. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). |


Suggested change

`field` | Required | The name of the field that contains the data to be parsed. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). |

`field` | Required | The name of the field containing the data to be parsed. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). |

natebower · 2023-12-12T13:39:03Z

_ingest-pipelines/processors/kv.md

+`exclude_keys` | Optional | The keys to exclude from the document. Default is `null`. |
+`include_keys` | Optional | The keys for filtering and inserting. Default is to include all keys. |
+`prefix` | Optional | The prefix to add to the extracted keys. Default is `null`. |
+`strip_brackets` | Optional | If set to `true`, strips brackets `()`, `<>,` or `[]` and quotes `'` or `"` from extracted values. Default is `false`.


Can we put parentheses around the examples?: ((), <>, or []) and quotes (' or ")

natebower · 2023-12-12T13:45:36Z

_ingest-pipelines/processors/kv.md

+```
+{% include copy-curl.html %}
+
+**Step 2 (Optional): Test the pipeline.**


Suggested change

**Step 2 (Optional): Test the pipeline.**

**Step 2 (Optional): Test the pipeline**

natebower · 2023-12-12T13:47:06Z

_ingest-pipelines/processors/kv.md

+```
+{% include copy-curl.html %}
+
+#### Response


Confirm heading level.

natebower · 2023-12-12T13:47:21Z

_ingest-pipelines/processors/kv.md

+}
+```
+
+**Step 3: Ingest a document.**


Suggested change

**Step 3: Ingest a document.**

**Step 3: Ingest a document**

natebower · 2023-12-12T13:47:36Z

_ingest-pipelines/processors/kv.md

+```
+{% include copy-curl.html %}
+
+**Step 4 (Optional): Retrieve the document.**


Suggested change

**Step 4 (Optional): Retrieve the document.**

**Step 4 (Optional): Retrieve the document**

_ingest-pipelines/processors/lowercase.md

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

* Add kv processor content gap doc --------- Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Andriy Redko <drreta@gmail.com> Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Co-authored-by: Nathan Bower <nbower@amazon.com> (cherry picked from commit 424956d) Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>

* Add kv processor content gap doc --------- (cherry picked from commit 424956d) Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Signed-off-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com> Co-authored-by: Andriy Redko <drreta@gmail.com> Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Co-authored-by: Nathan Bower <nbower@amazon.com>

* Add kv processor content gap doc --------- Signed-off-by: Melissa Vagi <vagimeli@amazon.com> Co-authored-by: Andriy Redko <drreta@gmail.com> Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Co-authored-by: Nathan Bower <nbower@amazon.com>

Add kv processor content gap doc

a0cdf21

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli added Content gap v2.12.0 labels Dec 4, 2023

vagimeli self-assigned this Dec 4, 2023

vagimeli requested review from hdhalter, kolchfa-aws, Naarcha-AWS, ananzh, AMoo-Miki and natebower as code owners December 4, 2023 22:46

vagimeli changed the title ~~[DOC} Add kv processor content gap doc~~ [DOC] Add kv processor content gap doc Dec 4, 2023

vagimeli mentioned this pull request Dec 4, 2023

[DOC] The parameter field and target_field in KV ingest processor now supports template snippets #5651

Closed

4 tasks

Merge branch 'main' into kv

7a58fbc

Add parameters section

388727f

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli added the 3 - Tech review PR: Tech review in progress label Dec 5, 2023

Add parameters section

eb203df

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

reta reviewed Dec 5, 2023

View reviewed changes

_ingest-pipelines/processors/kv.md Outdated Show resolved Hide resolved

gaobinlong reviewed Dec 6, 2023

View reviewed changes

_ingest-pipelines/processors/kv.md Outdated Show resolved Hide resolved

_ingest-pipelines/processors/kv.md Outdated Show resolved Hide resolved

_ingest-pipelines/processors/kv.md Outdated Show resolved Hide resolved

_ingest-pipelines/processors/kv.md Outdated Show resolved Hide resolved

vagimeli and others added 3 commits December 8, 2023 11:23

Update _ingest-pipelines/processors/kv.md

87a0936

Co-authored-by: Andriy Redko <drreta@gmail.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Address tech review feedback

5037f37

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Merge branch 'main' into kv

57bfb88

vagimeli added 4 - Doc review PR: Doc review in progress and removed 3 - Tech review PR: Tech review in progress labels Dec 8, 2023

Merge branch 'main' into kv

6857826

hdhalter approved these changes Dec 11, 2023

View reviewed changes

vagimeli and others added 2 commits December 11, 2023 16:09

Update _ingest-pipelines/processors/kv.md

f46eb5a

Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _ingest-pipelines/processors/kv.md

e4a8c69

Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli and others added 2 commits December 11, 2023 16:09

Update _ingest-pipelines/processors/lowercase.md

6c69453

Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _ingest-pipelines/processors/kv.md

74a5222

Co-authored-by: Heather Halter <HDHALTER@AMAZON.COM> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli added 2 commits December 11, 2023 16:12

Merge branch 'main' into kv

14f6a20

Copy edits

3639d82

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

natebower reviewed Dec 12, 2023

View reviewed changes

vagimeli and others added 3 commits December 12, 2023 10:54

Update _ingest-pipelines/processors/kv.md

6cb60fa

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Update _ingest-pipelines/processors/lowercase.md

e3e8fe0

Co-authored-by: Nathan Bower <nbower@amazon.com> Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

Address editorial feedback

37082cd

Signed-off-by: Melissa Vagi <vagimeli@amazon.com>

vagimeli merged commit 424956d into main Dec 12, 2023
4 checks passed

vagimeli deleted the kv branch December 12, 2023 18:23

vagimeli added the backport 2.11 PR: Backport label for 2.11 label Dec 12, 2023

opensearch-trigger-bot bot mentioned this pull request Dec 12, 2023

[Backport 2.11] [DOC] Add kv processor content gap doc #5845

Merged

hdhalter mentioned this pull request Dec 20, 2023

[META DOC] Ingest Processors Content Gaps #4647

Closed

45 tasks

hdhalter removed the 4 - Doc review PR: Doc review in progress label Dec 26, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[DOC] Add kv processor content gap doc #5781

[DOC] Add kv processor content gap doc #5781

vagimeli commented Dec 4, 2023

vagimeli commented Dec 5, 2023

reta commented Dec 5, 2023

vagimeli commented Dec 8, 2023

hdhalter left a comment

hdhalter Dec 11, 2023

vagimeli Dec 11, 2023

vagimeli commented Dec 11, 2023

natebower left a comment

natebower Dec 12, 2023

natebower Dec 12, 2023

natebower Dec 12, 2023

natebower Dec 12, 2023

natebower Dec 12, 2023

natebower Dec 12, 2023

natebower Dec 12, 2023

natebower Dec 12, 2023


		The following table lists the required and optional parameters for the `lowercase` processor.

		\| Name \| Required/Optional \| Description \|


		# KV processor

		The `kv` processor automatically extracts specific event fields or messages that are in a`key=value` format. This structured format organizes your data by grouping it together based on keys and values. It's helpful for analyzing, visualizing, and using data such as user behavior analytics, performance optimizations, or security investigations.

	`field` \| Required \| The name of the field that contains the data to be parsed. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). \|
	`field` \| Required \| The name of the field containing the data to be parsed. Supports [template snippets]({{site.url}}{{site.baseurl}}/ingest-pipelines/create-ingest/#template-snippets). \|

	Step 2 (Optional): Test the pipeline.
	Step 2 (Optional): Test the pipeline

	Step 4 (Optional): Retrieve the document.
	Step 4 (Optional): Retrieve the document

[DOC] Add kv processor content gap doc #5781

[DOC] Add kv processor content gap doc #5781

Conversation

vagimeli commented Dec 4, 2023

Description

Issues Resolved

Checklist

vagimeli commented Dec 5, 2023

reta commented Dec 5, 2023

vagimeli commented Dec 8, 2023

hdhalter left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vagimeli commented Dec 11, 2023

natebower left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment