Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support Data Streams in OpenSearch #690

Merged
merged 3 commits into from
May 17, 2021

Conversation

ketanv3
Copy link
Contributor

@ketanv3 ketanv3 commented May 12, 2021

Description

This PR aims to add the missing MetadataFieldMapper required to parse the _data_stream_timestamp metadata field mapping used to create data streams and to perform timestamp field validation on the ingested documents. This PR also allows users to change the default timestamp field name (from "@timestamp") as required using an index template.

Issues Resolved

Check List

  • New functionality includes testing.
    • All tests pass
  • New functionality has been documented.
    • New functionality has javadoc added
  • Commits are signed per the DCO using --signoff

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.
For more information on following Developer Certificate of Origin and signing off your commits, please check here.

Signed-off-by: Ketan Verma ketan9495@gmail.com

ketanv3 added 2 commits May 12, 2021 12:55
…Refactored tests.

Signed-off-by: Ketan Verma <ketan9495@gmail.com>
Signed-off-by: Ketan Verma <ketan9495@gmail.com>
@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success d9b2521

@opensearch-ci-bot
Copy link
Collaborator

✅   DCO Check Passed d9b2521

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Precommit success d9b2521

@ketanv3 ketanv3 marked this pull request as ready for review May 12, 2021 08:10
@nknize nknize added enhancement Enhancement or improvement to existing feature or request v1.0.0 Version 1.0.0 v2.0.0 Version 2.0.0 labels May 12, 2021
@nknize
Copy link
Collaborator

nknize commented May 13, 2021

start gradle check

Copy link
Collaborator

@nknize nknize left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good; one backwards compatibility addition needed

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success d9b2521
Log 179

Reports 179

Signed-off-by: Ketan Verma <ketan9495@gmail.com>
@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Wrapper Validation success 32cedcd

@opensearch-ci-bot
Copy link
Collaborator

✅   DCO Check Passed 32cedcd

@ketanv3 ketanv3 requested a review from nknize May 13, 2021 17:07
@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Precommit success 32cedcd

@nknize
Copy link
Collaborator

nknize commented May 13, 2021

start gradle check

@opensearch-ci-bot
Copy link
Collaborator

✅   Gradle Check success 32cedcd
Log 180

Reports 180

Copy link
Collaborator

@nknize nknize left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM! Nice to get this out there!

@nknize nknize merged commit ce12097 into opensearch-project:main May 17, 2021
@nknize nknize added the pending backport Identifies an issue or PR that still needs to be backported label May 17, 2021
@nknize
Copy link
Collaborator

nknize commented May 17, 2021

Thank you for working this enhancement. Can you open a backport PR to the 1.x branch? The pending label will stay applied until we merge to 1.x.

ketanv3 added a commit to ketanv3/OpenSearch that referenced this pull request May 17, 2021
This commit adds support for data streams by adding a DataStreamFieldMapper, and making timestamp 
field name configurable. Backwards compatibility is supported.

Signed-off-by: Ketan Verma <ketan9495@gmail.com>
@ketanv3
Copy link
Contributor Author

ketanv3 commented May 17, 2021

@nknize Created this backport PR – #713

dblock pushed a commit that referenced this pull request May 20, 2021
This commit adds support for data streams by adding a DataStreamFieldMapper, and making timestamp 
field name configurable. Backwards compatibility is supported.

Signed-off-by: Ketan Verma <ketan9495@gmail.com>
@adnapibar adnapibar removed the pending backport Identifies an issue or PR that still needs to be backported label May 21, 2021
ritty27 pushed a commit to ritty27/OpenSearch that referenced this pull request May 12, 2024
* Adding an example for bulk update operation

Signed-off-by: Karthik Subramanian <karthiks3000@yahoo.com>

* updated the changelog.md

Signed-off-by: Karthik Subramanian <karthiks3000@yahoo.com>

* changes made by spotlessApply

Signed-off-by: Karthik Subramanian <karthiks3000@yahoo.com>

* removing CommonUtil based on review comments

Signed-off-by: Karthik Subramanian <karthiks3000@yahoo.com>

* reverting files to original state

Signed-off-by: Karthik Subramanian <karthiks3000@yahoo.com>

* changes by spotlessApply

Signed-off-by: Karthik Subramanian <karthiks3000@yahoo.com>

* removing static fields and unnecessary waits

Signed-off-by: Karthik Subramanian <karthiks3000@yahoo.com>

* changes as per review comments

Signed-off-by: Karthik Subramanian <karthiks3000@yahoo.com>

* removing unnecessary changes

Signed-off-by: Karthik Subramanian <karthiks3000@yahoo.com>

---------

Signed-off-by: Karthik Subramanian <karthiks3000@yahoo.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement Enhancement or improvement to existing feature or request v1.0.0 Version 1.0.0 v2.0.0 Version 2.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature] Support Data Streams in OpenSearch
4 participants