Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[INLONG-10704][Sort] Sort Add Oceanbase Support #10700

Merged
merged 6 commits into from
Jul 29, 2024

Conversation

xxsc0529
Copy link
Contributor

@xxsc0529 xxsc0529 commented Jul 23, 2024

Fixes #10704

Motivation

Added support for Oceanbase data sources to adapt to the Oceanbase ecosystem. close #10704

Modifications

It mainly adds oceanbase data source support and uses the oceanbase-client driver to connect.

It mainly adds oceanbase data source support and uses the oceanbase-client driver to connect. And write oceanbase's ExtractNode logic and LoadNode logic. The specific implementation logic is the same as mysql.Added support for oceanbase on the front-end page. Since the jdbc-inlong processing logic in the flink15 connection lacks the jdbc dialect, an error is reported. This is modified here.

Verifying this change

(Please pick either of the following options)

  • This change is a trivial rework/code cleanup without any test coverage.

  • This change is already covered by existing tests, such as:
    (please describe tests)
    Here's how the page test works
    image
    image
    image
    image

  • This change added tests and can be verified as follows:
    Please use the following test classes for testing org.apache.inlong.sort.parser.MySqlExtractNodeToOceanBaseLoadNodeTest
    (example:)

    • Added integration tests for end-to-end deployment with large payloads (10MB)
    • Extended integration test for recovery after broker failure

Documentation

  • Does this pull request introduce a new feature? (yes / no) yes
  • If yes, how is the feature documented? (not applicable / docs / JavaDocs / not documented) Data source types have been added to the README.md
  • If a feature is not applicable for documentation, explain why?
  • If a feature is not documented yet in this PR, please create a follow-up issue for adding the documentation

@xxsc0529 xxsc0529 changed the title [INLONG-10632][Sort] Sort Add Oceanbase Support [INLONG-10704][Sort] Sort Add Oceanbase Support Jul 23, 2024
vernedeng
vernedeng previously approved these changes Jul 24, 2024
@dockerzhang dockerzhang requested review from featzhang and EMsnap July 24, 2024 03:03
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This factory should named OceanBaseDialectFactory?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No, it was found in the test that MySQL and Oceanbase both need this dialect to find the corresponding factory that conforms to the JDBC protocol

Copy link
Contributor

@EMsnap EMsnap left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

please add test result with real data from any source to oceanbase

@xxsc0529
Copy link
Contributor Author

please add test result with real data from any source to oceanbase

What kind of test data is needed and where to put it

@XiaoYou201
Copy link
Contributor

please add test result with real data from any source to oceanbase

What kind of test data is needed and where to put it

You can refer to inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/src/test/java/org/apache/inlong/sort/tests/Mysql2StarRocksTest.java

or package your code to manager container. Then start a job to verify the oceanbase connector work properly.If all right, plz
provide some screenshots~

@xxsc0529
Copy link
Contributor Author

please add test result with real data from any source to oceanbase

What kind of test data is needed and where to put it

You can refer to inlong-sort/sort-end-to-end-tests/sort-end-to-end-tests-v1.15/src/test/java/org/apache/inlong/sort/tests/Mysql2StarRocksTest.java

or package your code to manager container. Then start a job to verify the oceanbase connector work properly.If all right, plz provide some screenshots~

The logic of Oceanbase and MySQL is the same, using MySQL-CDC, and the following is tested in the form of a remote debug project jar package through local source code
image
This step shows that the pre-configuration logic is passable, and the corresponding CDC can be found and flinksql can be generated,Below is the generated flinksql
FlinkSqlParseResult(tableEnv=org.apache.flink.table.api.bridge.java.internal.StreamTableEnvironmentImpl@42554211, createTableSqls=[CREATE TABLE table_mysqtooceanbase(
id INT,
user_name STRING)
WITH (
'inlong.metric.labels' = 'groupId=mysqtooceanbase&streamId=mysqtooceanbase&nodeId=mysqtooceanbase',
'topic' = 'mysqtooceanbase.mysqtooceanbase',
'properties.bootstrap.servers' = '127.0.0.1:9092',
'connector' = 'kafka-inlong',
'inlong-msg.debezium-json.schema-include' = 'false',
'inlong-msg.debezium-json.encode.decimal-as-plain-number' = 'true',
'inlong-msg.inner.format' = 'debezium-json',
'format' = 'inlong-msg',
'inlong-msg.debezium-json.timestamp-format.standard' = 'SQL',
'inlong-msg.debezium-json.ignore-parse-errors' = 'true',
'inlong-msg.debezium-json.map-null-key.literal' = 'null',
'inlong-msg.debezium-json.map-null-key.mode' = 'DROP',
'inlong-msg.ignore-parse-errors' = 'false',
'scan.startup.mode' = 'earliest-offset',
'properties.group.id' = 'kafka-local_mysqtooceanbase.mysqtooceanbase_consumer_group'
), CREATE TABLE table_oceanbase(
PRIMARY KEY (id) NOT ENFORCED,
id INT,
user_name STRING)
WITH (
'inlong.metric.labels' = 'groupId=mysqtooceanbase&streamId=mysqtooceanbase&nodeId=oceanbase',
'connector' = 'jdbc-inlong',
'url' = 'jdbc:mysql://Confidentiality:4848/test',
'username' = 'Confidentiality@tt1',
'password' = '123456',
'table-name' = 't_ds_user'
)], loadSqls=[INSERT INTO table_oceanbase
SELECT
id AS id,
user_name AS user_name
FROM table_mysqtooceanbase ])
Go to flink-ui to find that the task has also been submitted
image

@XiaoYou201
Copy link
Contributor

LGTM~

vernedeng
vernedeng previously approved these changes Jul 26, 2024
aloyszhang
aloyszhang previously approved these changes Jul 26, 2024
@xxsc0529 xxsc0529 dismissed stale reviews from aloyszhang and vernedeng via df02cb9 July 26, 2024 06:55
@dockerzhang dockerzhang merged commit 0b50e29 into apache:master Jul 29, 2024
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[Feature][Sort] sort should support oceanbase
6 participants