Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[INLONG-5046][Agent] Support collect data from PostgreSQL #5367

Merged
merged 19 commits into from
Aug 17, 2022
Merged

[INLONG-5046][Agent] Support collect data from PostgreSQL #5367

merged 19 commits into from
Aug 17, 2022

Conversation

iamsee123
Copy link
Contributor

Prepare a Pull Request

Motivation

Add the collecting postgreSQL data function to agent module and provide the necessary management ability for manager module.

Modifications

  • PostgreSqlReader: read data from postgres by Debezium
  • PostgreSqlSource: PostgreSql source, split PostgreSql source job into multi readers
  • PostgreSqlSnapshotBase: postgres snapshot
  • SnapshotModeConstants: add some snapshot mode about postgres

Verifying this change

  • This change added tests and can be verified as follows:
  • add the Unit test of PostgreSqlReader in TestPostgresReader
  • add the Unit test of PostgreSqlSnapshotBase in TestPostgresOffsetManager

@dockerzhang dockerzhang requested a review from EMsnap August 9, 2022 11:06
@dockerzhang
Copy link
Contributor

@haibo-duan PTAL, thanks.

healchow
healchow requested changes Aug 14, 2022
View reviewed changes
...agent-plugins/src/test/java/org/apache/inlong/agent/plugin/sources/PostgreSQLReaderTest.java Outdated Show resolved Hide resolved
...agent-plugins/src/test/java/org/apache/inlong/agent/plugin/sources/PostgreSQLReaderTest.java Outdated Show resolved Hide resolved
...agent-plugins/src/test/java/org/apache/inlong/agent/plugin/sources/PostgreSQLReaderTest.java Show resolved Hide resolved
...lugins/src/test/java/org/apache/inlong/agent/plugin/sources/PostgreSQLOffsetManagerTest.java Show resolved Hide resolved
...ns/src/main/java/org/apache/inlong/agent/plugin/sources/snapshot/PostgreSQLSnapshotBase.java Outdated Show resolved Hide resolved
...ns/src/main/java/org/apache/inlong/agent/plugin/sources/snapshot/PostgreSQLSnapshotBase.java Outdated Show resolved Hide resolved
...nt-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/PostgreSQLReader.java Outdated Show resolved Hide resolved
...nt-plugins/src/main/java/org/apache/inlong/agent/plugin/sources/reader/PostgreSQLReader.java Outdated Show resolved Hide resolved
@iamsee123
[INLONG-5046] adjust name with postgres,postgreSql into official name…
a50f321 
… postgreSQL and add annotation on PostgreSQLReaderTest.java method postgresLoadTest
@iamsee123
[INLONG-5046] adjust codeStyle to fit the requests
6ec8d70
healchow
healchow approved these changes Aug 16, 2022
View reviewed changes
Copy link
Member

@healchow healchow left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@dockerzhang dockerzhang merged commit e11d7d8 into apache:master Aug 17, 2022
@iamsee123 iamsee123 deleted the INLONG-5046 branch August 17, 2022 06:15
rhizoma-atractylodis pushed a commit to rhizoma-atractylodis/inlong that referenced this pull request Aug 17, 2022
@iamsee123 @rhizoma-atractylodis
[INLONG-5046][Agent] Support collect data from PostgreSQL (apache#5367)
b67a0af
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dockerzhang dockerzhang dockerzhang approved these changes

@EMsnap EMsnap EMsnap approved these changes

@healchow healchow healchow approved these changes

Assignees
No one assigned
Labels
component/agent
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

[Feature][Agent] Support collect data from PostgreSQL
4 participants
@iamsee123 @dockerzhang @EMsnap @healchow