Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[INLONG-7581][Sort] Support multiple-sink migration for Elasticsearch #7659

Closed
wants to merge 5 commits into from
Closed

[INLONG-7581][Sort] Support multiple-sink migration for Elasticsearch #7659

wants to merge 5 commits into from

Conversation

Yizhou-Yang
Copy link
Contributor

@Yizhou-Yang Yizhou-Yang commented Mar 21, 2023 •
edited
Loading

Prepare a Pull Request

Motivation

Support multiple sink migration for ES sink.

Modifications

mainly modified multipleElasticRowFunction for 2 key features:

  1. support 4 additional options for esloadnode
  2. parsing canal-json data

TODO:

  1. table-level metric
  2. dirty data
  3. runtime strategy+other features

Verifying this change

image

image

@Yizhou-Yang Yizhou-Yang changed the title [INLONG-7581][Sort] Support multiple-sink migration for Elastic Search [INLONG-7581][Sort] Support multiple-sink migration for Elastic Search 7 Mar 21, 2023
@Yizhou-Yang Yizhou-Yang marked this pull request as ready for review March 21, 2023 06:59
@healchow healchow changed the title [INLONG-7581][Sort] Support multiple-sink migration for Elastic Search 7 [INLONG-7581][Sort] Support multiple-sink migration for Elasticsearch 7 Mar 24, 2023
@Yizhou-Yang Yizhou-Yang changed the title [INLONG-7581][Sort] Support multiple-sink migration for Elasticsearch 7 [INLONG-7581][Sort] Support multiple-sink migration for Elasticsearch Mar 27, 2023
yunqingmoswu
yunqingmoswu reviewed Mar 27, 2023
View reviewed changes
inlong-sort/sort-connectors/elasticsearch-base/pom.xml Show resolved Hide resolved
...arch-base/src/main/java/org/apache/inlong/sort/elasticsearch/table/ElasticsearchOptions.java Show resolved Hide resolved
...n/java/org/apache/inlong/sort/elasticsearch/table/MultipleElasticsearchSinkFunctionBase.java Show resolved Hide resolved
inlong-sort/sort-core/src/test/java/org/apache/inlong/sort/parser/ESMultipleSinkTest.java Outdated
import java.util.stream.Collectors;

/**
* Test for {@link org.apache.inlong.sort.protocol.node.load.DorisLoadNode}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Comment error

@gong
Copy link
Contributor

gong commented Mar 27, 2023

You need rebase master

@Yizhou-Yang Yizhou-Yang force-pushed the feature-multiple-es branch from 1b0abca to 6d4aca3 Compare March 27, 2023 10:03
@Yizhou-Yang Yizhou-Yang reopened this Mar 27, 2023
@Yizhou-Yang Yizhou-Yang force-pushed the feature-multiple-es branch from 716a066 to 86fd01d Compare March 27, 2023 10:13
@Yizhou-Yang Yizhou-Yang reopened this Mar 27, 2023
Revert "rebase master"
206b753 
This reverts commit 01205f3.
@gong
Copy link
Contributor

gong commented Mar 27, 2023 •
edited
Loading

1、git pull upstream master
2、git checkout -b INLONG-7581
3、pick your modification into INLONG-7581
4、git push origin INLONG-7581
you can link issues number 7581

@Yizhou-Yang Yizhou-Yang deleted the feature-multiple-es branch March 28, 2023 05:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@yunqingmoswu yunqingmoswu yunqingmoswu left review comments

Assignees
No one assigned
Labels
component/sort
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature][Sort] Support multiple-sink migration for Elasticsearch
3 participants
@Yizhou-Yang @gong @yunqingmoswu