Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Feature][Agent] Support collect data from PostgreSQL #5046

Closed
2 tasks done
Tracked by #4988
dockerzhang opened this issue Jul 14, 2022 · 5 comments · Fixed by #5367
Closed
2 tasks done
Tracked by #4988

[Feature][Agent] Support collect data from PostgreSQL #5046

dockerzhang opened this issue Jul 14, 2022 · 5 comments · Fixed by #5367
Assignees
Milestone

Comments

@dockerzhang
Copy link
Contributor

Description

you can refer to:
https://inlong.apache.org/docs/next/design_and_concept/how_to_write_plugin_agent

Use case

No response

Are you willing to submit PR?

  • Yes, I am willing to submit a PR!

Code of Conduct

@iamsee123
Copy link
Contributor

I want to do this issue

@iamsee123
Copy link
Contributor

iamsee123 commented Jul 25, 2022

Motivation

Add the collecting postgreSQL data function to agent module and provide the necessary management ability for manager module.

Changes

  • An agent module plugin with Reader and Source added
  • A manager plugin with extention of ExtractNode and LoadNode added

Design

This PR following the document Agent Plugin and Manager Plugin

  1. read wal log in reader by Debezium
  2. create postgreSQL wal log reader and add it to source
  3. add Enum type in manager class

Implementation

  • Add the ability of collecting data from PostgreSQL
    • PostgreSQLLogReader: read data from wal log by Debezium
    • PostgreSQLLogSource:implement split method
  • Add the management ability for manager module
    • WalLogSnapShot: store some info during reading
    • add type in TaskTypeEnum,SourceType

@EMsnap
Copy link
Contributor

EMsnap commented Jul 29, 2022

Thanks for contribution, the implementation looks fine.
But for manager, it use tasktype for different tasks, not ExtractNode. ExtractNode is used for sort module actually.

@EMsnap
Copy link
Contributor

EMsnap commented Jul 29, 2022

For manager -> agent, pls see AgentServiceImpl getTaskResult

@iamsee123
Copy link
Contributor

OK, Thank you. I will see that.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment