[Improve][Connector-V2][Jdbc-Source] Support for Decimal types as splict keys #4634

zhilinli123 · 2023-04-20T08:15:41Z

Purpose of this pull request

Check list

Code changed are covered with tests, or it does not need tests for reason:
If any new Jar binary package adding in your PR, please add License Notice according
New License Guide
If necessary, please update the documentation to describe the new feature. https://github.com/apache/incubator-seatunnel/tree/dev/docs
If you are contributing the connector code, please check that the following files are updated:
1. Update change log that in connector document. For more details you can refer to connector-v2
2. Update plugin-mapping.properties and add new connector information in it
3. Update the pom file of seatunnel-dist
Update the release-note.

ic4y · 2023-04-20T14:06:23Z

The current partition range is processed using Long, and bigint(20) will cause overflow, so it is necessary to modify the PartitionParameter related classes.

ic4y · 2023-04-21T02:34:30Z

Please update the document

partition_column [string]
The column name for parallelism's partition, only support numeric type.

zhilinli123 · 2023-04-21T02:40:46Z

This is the document for now

ic4y · 2023-04-21T03:34:33Z

From the design perspective, it is necessary to support any DecimalType with scale=0. Otherwise, DecimalType(20,0), DecimalType(30,0), etc., will need to be modified. So it's better to do it well in one go

ic4y · 2023-04-21T03:36:17Z

This is the document for now

It is necessary to specify the supported data types clearly, such as Int, Long, etc

hailin0 · 2023-04-21T06:22:55Z

...dbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlIT.java

@@ -60,7 +60,8 @@ public class JdbcMysqlIT extends AbstractJdbcIT {
            Lists.newArrayList(
                    "/jdbc_mysql_source_and_sink.conf",
                    "/jdbc_mysql_source_and_sink_parallel.conf",
-                    "/jdbc_mysql_source_and_sink_parallel_upper_lower.conf");
+                    "/jdbc_mysql_source_and_sink_parallel_upper_lower.conf",
+                    "/jdbc_mysql_source_and_sink_bu_parallel_upper_lower.conf");


What is bu?

BIGINT unsigned abbreviate

incomprehensible abbreviation

sorry for my oversight

…_column apache#4627

zhilinli123 · 2023-04-24T02:35:46Z

From the design perspective, it is necessary to support any DecimalType with scale=0. Otherwise, DecimalType(20,0), DecimalType(30,0), etc., will need to be modified. So it's better to do it well in one go

I have checked that Flink also uses Long type for storage. I think we do not need to change it for the time being CC @hailin0 @ic4y

ic4y · 2023-04-25T02:51:36Z

From the design perspective, it is necessary to support any DecimalType with scale=0. Otherwise, DecimalType(20,0), DecimalType(30,0), etc., will need to be modified. So it's better to do it well in one go

I have checked that Flink also uses Long type for storage. I think we do not need to change it for the time being CC @hailin0 @ic4y

If Flink also supports bigint(20), it might have a similar issue.

ic4y

Currently, there are two issues that need further discussion

1、The design needs to support any DecimalType with scale=0, instead of bigint(20).
2、Need to resolve the overflow issue of using Long to store partition boundaries

zhilinli123 · 2023-05-09T05:59:12Z

Currently, there are two issues that need further discussion

1、The design needs to support any DecimalType with scale=0, instead of bigint(20). 2、Need to resolve the overflow issue of using Long to store partition boundaries

@TyrantLucifer @TaoZex @EricJoy2048 @hailin0 @Hisoka-X All bosses have time to help look at this thanks

…unsigned-partition

…0) used as a partition_column apache#4634

ic4y · 2023-05-11T10:27:39Z

...c/src/main/java/org/apache/seatunnel/connectors/seatunnel/jdbc/source/JdbcSourceFactory.java

+        // Currently, only numeric types are supported for reading partition keys
+        return type.equals(BasicType.INT_TYPE)
+                || type.equals(BasicType.LONG_TYPE)
+                || type.equals(new DecimalType(20, 0));


The current implementation support any DecimalType with scale=0.

ic4y · 2023-05-11T10:58:38Z

...dbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlIT.java

@@ -64,6 +64,7 @@ public class JdbcMysqlIT extends AbstractJdbcIT {
    private static final String CREATE_SQL =
            "CREATE TABLE IF NOT EXISTS %s\n"
                    + "(\n"
+                    + "    `c_bigint_30`          BIGINT(40) unsigned   DEFAULT NULL,\n"


It is recommended to directly use DECIMAL(30,0) for partition field testing.

ic4y · 2023-05-11T11:06:49Z

...dbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlIT.java

@@ -194,11 +196,14 @@ Pair<String[], List<SeaTunnelRow>> initTestData() {
                };

        List<SeaTunnelRow> rows = new ArrayList<>();
+        // Define a large bigint data value to test for compatibility above bigint(20)
+        BigDecimal testBigDecimalValue = new BigDecimal("2844674407371055000");


The number 2844674407371055000 is within the storage range of a long. You should use a value that is outside of long's storage capacity, such as 899,999,999,999,999,999,999,999,999,999.

ic4y · 2023-05-11T11:07:46Z

...dbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlIT.java

        for (int i = 0; i < 100; i++) {
            byte byteArr = Integer.valueOf(i).byteValue();
            SeaTunnelRow row =
                    new SeaTunnelRow(
                            new Object[] {
+                                testBigDecimalValue.add(BigDecimal.valueOf(i)),


Do not delete the column c_bit_1.

Please have a look. Thank you. @ic4y

…0) used as a partition_column apache#4634

zhilinli123 · 2023-05-15T07:31:04Z

@ic4y @hailin0 Please check it out. Thank you

ic4y · 2023-05-16T02:39:45Z

Wait for the CI to finish running

zhilinli123 · 2023-05-16T08:17:38Z

CI success @ic4y @EricJoy2048 @hailin0 Please check it out. Thank you

hailin0 · 2023-05-19T08:05:59Z

...dbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlIT.java

+                    + "    `c_bigint_30`            BIGINT(40)  unsigned  DEFAULT NULL,\n"
+                    + "    `c_decimal_unsigned_30`  DECIMAL(30) unsigned  DEFAULT NULL,\n"
+                    + "    `c_decimal_30`           DECIMAL(30)           DEFAULT NULL,\n"


move to end

hailin0 · 2023-05-19T08:06:15Z

...dbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlIT.java

+                    "c_bigint_30",
+                    "c_decimal_unsigned_30",
+                    "c_decimal_30",


move to end

hailin0 · 2023-05-19T08:06:32Z

...dbc-e2e-part-1/src/test/java/org/apache/seatunnel/connectors/seatunnel/jdbc/JdbcMysqlIT.java

+                                bigintValue.add(BigDecimal.valueOf(i)),
+                                decimalValue.add(BigDecimal.valueOf(i)),
+                                decimalValue.add(BigDecimal.valueOf(i)),


move to end

hailin0 · 2023-05-19T08:06:47Z

...nector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/resources/jdbc_mysql_source_and_sink.conf

@@ -42,12 +42,12 @@ sink {
        user = "root"
        password = "Abc!@#135_seatunnel"

-        query = """insert into sink (c_bit_1, c_bit_8, c_bit_16, c_bit_32, c_bit_64, c_boolean, c_tinyint, c_tinyint_unsigned, c_smallint, c_smallint_unsigned,
+        query = """insert into sink (c_bigint_30,c_decimal_unsigned_30,c_decimal_30,c_bit_1, c_bit_8, c_bit_16, c_bit_32, c_bit_64, c_boolean, c_tinyint, c_tinyint_unsigned, c_smallint, c_smallint_unsigned,


move to end

hailin0 · 2023-05-19T08:07:03Z

...ctor-jdbc-e2e-part-1/src/test/resources/jdbc_mysql_source_and_sink_parallel_upper_lower.conf

@@ -46,12 +46,12 @@ sink {
        user = "root"
        password = "Abc!@#135_seatunnel"
        connection_check_timeout_sec = 100
-        query = """insert into sink (c_bit_1, c_bit_8, c_bit_16, c_bit_32, c_bit_64, c_boolean, c_tinyint, c_tinyint_unsigned, c_smallint, c_smallint_unsigned,
+        query = """insert into sink (c_bigint_30,c_decimal_unsigned_30,c_decimal_30,c_bit_1, c_bit_8, c_bit_16, c_bit_32, c_bit_64, c_boolean, c_tinyint, c_tinyint_unsigned, c_smallint, c_smallint_unsigned,


move to end

hailin0 · 2023-05-19T08:07:08Z

...bc-e2e/connector-jdbc-e2e-part-1/src/test/resources/jdbc_mysql_source_and_sink_parallel.conf

@@ -45,12 +45,12 @@ sink {
        user = "root"
        password = "Abc!@#135_seatunnel"
        connection_check_timeout_sec = 100
-        query = """insert into sink (c_bit_1, c_bit_8, c_bit_16, c_bit_32, c_bit_64, c_boolean, c_tinyint, c_tinyint_unsigned, c_smallint, c_smallint_unsigned,
+        query = """insert into sink (c_bigint_30,c_decimal_unsigned_30,c_decimal_30,c_bit_1, c_bit_8, c_bit_16, c_bit_32, c_bit_64, c_boolean, c_tinyint, c_tinyint_unsigned, c_smallint, c_smallint_unsigned,


move to end

Thanks for checking. I fixed it
@hailin0

…unsigned-partition

…0) used as a partition_column apache#4634

zhilinli123 · 2023-05-23T03:45:23Z

PTAL:@ic4y @Hisoka-X @hailin0 thks

ic4y

LGTM

zhilinli123 · 2023-05-25T09:52:39Z

PTAL:@EricJoy2048 @hailin0 thks
I have solved some code conflicts, please help to open CI, thank you

…unsigned-partition # Conflicts: # seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/resources/jdbc_mysql_source_and_sink.conf # seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/resources/jdbc_mysql_source_and_sink_parallel.conf # seatunnel-e2e/seatunnel-connector-v2-e2e/connector-jdbc-e2e/connector-jdbc-e2e-part-1/src/test/resources/jdbc_mysql_source_and_sink_parallel_upper_lower.conf

…0) used as a partition_column apache#4634

hailin0 · 2023-05-30T07:02:06Z

update title

zhilinli123 · 2023-05-30T07:39:38Z

update title
in this way?

davidzollo

+1

zhilinli123 · 2023-06-07T13:58:46Z

For the current pr- I will modify all source-documents related to boundary value types - in jdbc

…ict keys (apache#4634)

…ict keys (apache#4634) * [Improve][Connector-V2][Jdbc-Source]Support Compatible Mysql bigint(20) used as a partition_column apache#4634 Co-authored-by: zhilinli <lzl15844876351@163.com>

* [Feature][Connector V2] expose configurable options in Cassandra (apache#3681) * [Connector-V2][Paimon] Introduce paimon connector (apache#4178) * [Improve][Zeta] Improve Zeta operation max count and ignore NPE (apache#4787) * [Improve][Zeta] Improve Zeta operation max count and ignore NPE * [Improve][Zeta] Improve Zeta operation max count and ignore NPE * [Improve][Zeta] Cancel pipeline add retry to avoid cancel failed. (apache#4792) * [Hotfix][CDC] Fix chunk start/end parameter type error (apache#4777) Incorrect wrapping as Array<Array> types, but only Array type required * [Feature][Zeta] Add OSS support for Imap storage to cluster-mode type (apache#4683) * Add OSS/S3 to cluster-mode type apache#4621 * fixed bug & add e2e test * Wait for the node to start before scheduling & Move jar to parent pom & optimize writer * update LICENSE * [Hotfix][CI] Fix error repository name in ci config files (apache#4795) * [Feature][Json-format] support read format for pulsar (apache#4111) * [Improve][Connector-V2][Jdbc-Sink][Doc] Add the generate sink sql par… (apache#4797) * [Improve][Connector-V2][Jdbc-Sink][Doc] Add the generate sink sql parameter for the jdbc sinj document * [Docs][Connector-V2][Mysql] fix Mysql sink format doc (apache#4800) * [Hotfix][Connector][Jdbc] Fix sqlserver system table case sensitivity (apache#4806) * [Hotfix][Connector][Jdbc] Fix reconnect throw close statement exception (apache#4801) * [Hotfix][Connector-V2][Jdbc] Fix the error of extracting primary key column in sink (apache#4815) * [Feature][Connector-v2] Add Snowflake Source&Sink connector (apache#4470) --------- Co-authored-by: Eric <gaojun2048@gmail.com> Co-authored-by: hailin0 <wanghailin@apache.org> * [Hotfix][CI] Fix redundant modules run e2e tests when change jdbc module (apache#4824) * fix pom.xml code style (apache#4836) * [Chore] Format the.conf file using the same style (apache#4830) * [Hotfix][Zeta] Fix cpu load problem (apache#4828) * [Improve][Zeta] Reduce the number of IMAPs used by checkpointIdCounter (apache#4832) * [Bugfix][connector-v2][rabbitmq] Fix reduplicate ack msg bug and code style (apache#4842) --------- Co-authored-by: 毕博 <bibo@mafengwo.com> * [Improve][Zeta] async execute checkpoint trigger and other block method (apache#4846) * [Improve][Zeta] async execute checkpoint trigger * [Bug][Zeta] Fix zeta cannot normally recycle thread belong to abnormal tasks * [Improve][Zeta] Move `restoreState` add `addSplitsBack` execute by TaskExecuteService * [Improve][Zeta] Move `receivedReader` execute by TaskExecuteService * [Bug][Zeta] Fix task `notifyTaskStatusToMaster` failed when job not running or failed before run (apache#4847) * [Bug][Zeta] Fix task repeat notify failed when job not running * [Bug][Zeta] Fix notifyTaskStatusToMaster not release lock and NPE * [Improve][Zeta] Reduce the frequency of fetching data from imap (apache#4851) * [Improve][Zeta] Add Metaspace size default value to config file (apache#4848) * [Improve][Zeta] Speed up listAllJob function (apache#4852) * [Bug][Zeta] Fix TaskGroupContext always hold classloader so classloader can't recycle (apache#4849) * [Improve][Zeta] Fix engine runtime error (apache#4850) * [Hotfix][Zeta] Fix completePendingCheckpoint concurrent action (apache#4854) This operation does not allow concurrent execution * [Hotfix][Zeta] Fix master active bug (apache#4855) * [Bugfix][DAG] Fix the incorrect setting of transform parallelism (apache#4814) * [Hotfix][Zeta] fix pipeline state not right bug (apache#4823) * [BUG][Doris] Add a jobId to the doris label to distinguish between tasks (apache#4853) Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [Improve] Add a jobId to the doris label to distinguish between tasks (apache#4839) Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [Hotfix][Zeta] Fix IMap operation timeout bug (apache#4859) * [Bug][Zeta] Fix restoreComplete Future can't be completed when cancel task (apache#4863) * [Feature][SQL Transform]Add catalog support for SQL Transform plugin (apache#4819) * [improve][SelectDB] Add a jobId to the selectDB label to distinguish between tasks (apache#4864) Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [Hotfix][Connector-v2][kafka] Fix the short interval of pull data settings and revise the format (apache#4875) * [Bug][Connector-V2][Doris] update last checkpoint id when doing snapshot (apache#4881) * [Hotfix][Zeta] Fix deploy operation timeout but task already finished bug (apache#4867) * [Core][Docs]Remove incubator in README file (apache#4882) * [Bugfix][CDC Base] Solving the ConcurrentModificationException caused by snapshotState being modified concurrently. (apache#4877) * [improve][CDC base] Implement Sample-based Sharding Strategy with Configurable Sampling Rate (apache#4856) * [Improve][Zeta] Reduce the operation count of imap_running_job_metrics (apache#4861) * [Bug][Zeta] Fix TaskExecutionService will return not active ExecutionContext (apache#4869) * [Hotfix][Jdbc] Fix XA DataSource crash(Oracle/Dameng/SqlServer) (apache#4866) * [Bugfix] [Connector-V2] [File] Fix read temp file (apache#4876) Co-authored-by: wantao <wantao@inmyshow.com> * [Bug][Zeta] Fix TaskExecutionService synchronized lock will not release (apache#4886) * [Improve][Zeta] Move driver into lib directory and change operation count (apache#4845) * [hotfix][kafka] Fix the problem that the partition information cannot be obtained when kafka is restored (apache#4764) * [Bugfix][zeta] Fix the deadlock issue with JDBC driver loading (apache#4878) * [Chore] update 2.3.2 release-note.md (apache#4892) * [Improve][Connector-V2][Jdbc-Source] Support for Decimal types as splict keys (apache#4634) * [Improve][Connector-V2][Jdbc-Source]Support Compatible Mysql bigint(20) used as a partition_column apache#4634 Co-authored-by: zhilinli <lzl15844876351@163.com> * [Bug][connector-v2][doris] add streamload Content-type for doris URLdecode error (apache#4880) * [Chore] Change repository name from incubator-seatunnel to seatunnel (apache#4868) --------- Co-authored-by: Jia Fan <fanjiaeminem@qq.com> * [Improve][connector-V2-Neo4j]Supports neo4j sink batch write and update docs (apache#4841) * [Hotfix][connector-v2][e2e] Fix maven scope (apache#4901) * quick-start-seatunnel-engine.md (apache#4943) * fix error (apache#4888) * [Hotfix][Connector-V2][ClickhouseFile] Fix ClickhouseFile write file failed when field value is null (apache#4937) * Update ClickhouseFileSinkWriter.java Bug fix: When ClikchouseFileSinkerWriter writes to a temporary file, it does not check whether the field value is empty, so an exception will be thrown. Modified to write an empty string when a null value is encountered * Update ClickhouseFileSinkWriter.java repair code style * Update ClickhouseFileSinkWriter.java code style * [Improve][Zeta] Add an interface for batch retrieval of JobMetrics (apache#4576) * [Improve] Documentation and partial word optimization. (apache#4936) * code format * add cdc feature * fix cdc can not get driver error --------- Co-authored-by: gdliu3 <gdliu3@iflytek.com> * [Doc][Connector-V2] StarRocks `nodeUrls` property name fix (apache#4951) node_urls -> nodeUrls node_urls doesn't work * [Feature][E2E][FtpFile] add ftp file e2e test case (apache#4647) * [WIP][Feature][Connector-e2e] add ftp e2e test * Let e2e barely execute by excluding the commons-net jar package. * Resolve the maven conflict --------- Co-authored-by: hailin0 <wanghailin@apache.org> * [Hotfix][Connector-V2][StarRocks] Fix code style (apache#4966) * [Hotfix][Connector-v2][HbaseSink]Fix default timestamp (apache#4958) * [Doc]Change the transform website url (apache#4954) * [Docs][Connector-V2][Http]Reconstruct the Http connector document (apache#4962) Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Feature][connector-v2][mongodb] mongodb support cdc sink (apache#4833) * [Bug] [zeta][starter]fix bug (apache#4983) (apache#4984) Co-authored-by: wsstony <tonymao777@163.com> * fix redis nodes format error. (apache#4981) Co-authored-by: lightzhao <zhaolianyong777@gmail.com> * [Improve][CDC]Remove driver for cdc connector (apache#4952) * [Hotfix][Connector-V2][Mongodb] Fix document error content and remove redundant code (apache#4982) Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Improve][Connector-V2][OSS-Jindo] Optimize jindo oss connector (apache#4964) * [Improve][Connector-V2][Jindo-Oss] Optimize jindo-oss connector * [Improve][Connector-V2][Jindo-Oss] Update module name * [Hotfix][Connector-V2][StarRocks] Fix code style * [bugfix] Upgrade the key log output level(apache#4993) * [Feature][Zeta] Configuration files support user variable replacement (apache#4969) * [Feature][Transform-V2][SQL] Support 'select *' and 'like' clause for SQL Transform plugin (apache#4991) Co-authored-by: mcy <rewrma@163.com> * [Improve][CDC]change driver scope to provider (apache#5002) * [Hotfix][Connector-V2][Hive] Support user-defined hive-site.xml (apache#4965) * [Improve][Connector-v2][Mongodb]Optimize reading logic (apache#5001) Co-authored-by: chenqqq11 <chenzy15@ziroom.com> * [Feature][Connector-V2][Clickhouse] clickhouse writes with checkpoints (apache#4999) * [Hotfix][Connector-V2][Mongodb] Compatible with historical parameters (apache#4997) * Split updated modules integration test for part 4 (apache#5028) * [Hotfix] Fix the CI Job name error (apache#5032) * [Feature][CDC] Support disable/enable exactly once for INITIAL (apache#4921) * [bugfix][zeta] Fixed multi-table job data loss and latency issues (apache#149) (apache#5031) * [Hotfix][CDC] Fix jdbc connection leak for mysql (apache#5037) * [Bugfix][zeta] Fix cdc connection does not close (apache#4922) * Fix XA Transaction bug (apache#5020) * Set Up with Kubernetes, dockerfile document error in constructing docker image (apache#5022) Co-authored-by: yctan <1417983443@qq.com> * [Improve][Connector-v2][Mongodb]sink support transaction update/writing (apache#5034) * fix:the HdfsStorage can not delete checkpoint file apache#5046 (apache#5054) * [BugFix] [Connector-V2] [MySQL-CDC] serverId from int to long (apache#5033) (apache#5035) * [bugfix] change MySQL CDC serverId from int to long (apache#5033) * style: 🎨 optimize code style * [Feature][Connector-V2][cdc] Change the time zone to the default time zone (apache#5030) * [Bugfix][connector-cdc-mysql] Fix listener not released when BinlogClient reuse (apache#5011) * [Feature][Connector-V2][Jdbc] Add oceanbase dialect factory (apache#4989) --------- Co-authored-by: silenceland <silenceland23@163.com> Co-authored-by: changhuyan <877018069@qq.com> * [HotFix][Zeta] fix after the savepoint job is restored, the checkpoint file cannot be generated apache#4985 (apache#5051) * fix after the savepoint job is restored, the checkpoint file cannot be generated * fix class not found exception (apache#5063) * [Feature] update action config to support run CI on fork repo (apache#5065) * [Bugfix]fix clickhouse source connector read Nullable() type is not null,example:Nullable(Float64) while value is null the result is 0.0 (apache#5080) * [Feature][Connector-V2][Clickhouse] Add clickhouse connector time zone key,default system time zone (apache#5078) * Add clickhouse connector time zone key,default system time zone * Modify the document and add clickhouse server_time_zone configuration * [Chore] Modify repeat des (apache#5088) Co-authored-by: 80597928 <Lzl@qq.com> * [Docs] Add Value types in Java to Schema feature (apache#5087) * [Feature][Connector-V2] JDBC source support string type as partition key (apache#4947) * [HotFix] Fix code style (apache#5092) * [Docs][Zeta] Add savepoint doc (apache#5081) * [Feature][connector-v2][mongodbcdc]Support source mongodb cdc (apache#4923) * [Improve] Improve savemode api (apache#4767) * [Doc] Improve DB2 Source Vertica Source & DB2 Sink Vertica Sink document (apache#5102) * [Improve][Docs][Clickhouse] Reconstruct the clickhouse connector doc (apache#5085) --------- Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Pom]update version to 2.3.3-SNAPSHOT (apache#5043) * update version to 2.3.3-SNAPSHOT * update dependency version in know dependencies file * Add logs to find job restore from master active switch error * [Feature][Connector-V2][mysql cdc] Conversion of tinyint(1) to bool is supported (apache#5105) Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [Improve][Zeta] Add sleep for Task to reduce CPU cost (apache#5117) * [Feature][JDBC Sink] Add DM upsert support (apache#5073) --------- Co-authored-by: David Zollo <davidzollo365@gmail.com> * [Hotfix][Connector][Jdbc] Fix the problem of JdbcOutputFormat database connection leak (apache#4802) [Hotfix][Connector][Jdbc] Fix the problem of JdbcOutputFormat database connection leak * [Hotfix]Fix mongodb cdc e2e instability (apache#5128) Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Hotfix][Zeta] Fix task state memory leak (apache#5139) * [Hotfix][Zeta] Fix checkpoint error report without msg (apache#5137) * [Improve][Zeta] Improve CheckpointCoordinator notify complete when restore (apache#5136) * [Improve] Improve CheckpointCoordinator notify complete when restore * update * [Improve][Zeta] Improve CheckpointCoordinator log error when report error from task (apache#178) (apache#5134) * [Hotfix][Zeta] Fix MultipleTableJobConfigParser ignore env option (apache#5067) * [Fix][Zeta] Fix MultipleTableJobConfigParser ignore env option * update * [Improve][Connector[File] Optimize files commit order (apache#5045) Before using `HashMap` store files path, so every checkpoint file commit is out of order. Now switch to using `LinkedHashMap` to ensure that files are commit in the generated order * [Hotfix][Mongodb cdc] Solve startup resume token is negative (apache#5143) --------- Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Feature][connector][kafka] Support read debezium format message from kafka (apache#5066) * [Feature][CDC] Support tables without primary keys (with unique keys) (apache#163) (apache#5150) * [Feature][Connector-V2][CDC] Support string type shard fields. (apache#5147) * [feature][CDC base] Supports string type shard fields * Delete invalid code * [Feature][Connector-V2][File] Add cos source&sink (apache#4979) * [Feature][Connector-V2][File] Add cos sink * update doc&e2e and add pom file header * add e2e file header and config * add file-cos module into dist pom.xml * [Feature][Connector-V2][File] Add cos source --------- Co-authored-by: dengd1937 <dengd1803@gmail.com> * [Fix][Zeta] Fix SinkFlowLifeCycle without init lastCommitInfo (apache#5152) * [Hotfix][MongodbCDC]Refine data format to adapt to universal logic (apache#5162) Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Chore] Update bug-report.yml (apache#5160) * [Improve][CDC] support exactly-once of cdc and fix the BinlogOffset comparing bug (apache#5057) * [Improve][CDC] support exactly-once of cdc, fix the BinlogOffset comparing bug * [Improve][CDC] adjust code style * [Improve][CDC] fix ci error --------- Co-authored-by: happyboy1024 <296442618@qq.com> * [Docs][Connector-V2][Hudi] Reconstruct the Hudi connector document (apache#4905) * [Docs][Connector-V2][Hudi] Reconstruct the Hudi connector document --------- Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [Docs][Connector-V2][Doris] Reconstruct the Doris connector document (apache#4903) * [Docs][Connector-V2][Doris] Reconstruct the Doris connector document --------- Co-authored-by: zhouyao <yao.zhou@marketingforce.com> * [improve] [CDC Base] Add some split parameters to the optionRule (apache#5161) * [bugfix] [File Base] Fix Hadoop Kerberos authentication related issues. (apache#5171) * [CI] add code style check when docs changed (apache#5183) * [Bug][Translation][Spark] Fix SeaTunnelRowConvertor fail to convert when schema contains row type. (apache#5170) * [Improve][Zeta] Move checkpoint notify complete in checkpoint stage (apache#5185) * [Feature][Catalog] Add JDBC Catalog auto create table (apache#4917) * [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files. (apache#5153) * [Feature][Connector V2][File] Add config of 'file_filter_pattern', which used for filtering files. * [Improve][Connector-v2][Jdbc] check url not null throw friendly message (apache#5097) * check url not null throw friendly message * check jdbc source config * modify jdbc validate method --------- Co-authored-by: 80597928 <Lzl@qq.com> Co-authored-by: 80597928 <673421862@qq.com> * [bugfix][zeta] Fix the issue of two identical IDs appearing when executing seatunnel.sh -l as the job resumes (apache#5191) * [Improve][Docs][Kafka]Reconstruct the kafka connector document (apache#4778) * [Docs][Connector-V2][Kafka]Reconstruct the kafka connector document --------- Co-authored-by: chenzy15 <chenzy15@ziroom.com> * [Bug][Improve][LocalFileSink]Fix LocalFile Sink file_format_type. (apache#5118) * [Bug] [connector-v2] PostgreSQL versions below 9.5 are compatible use cdc sync problem (apache#5120) * [e2e] kafka e2e error (apache#5200) * [Hotfix][Connector-V2][JindoOssFile] Fix plugin-mapping.properties (apache#5215) Co-authored-by: tyrantlucifer <tyrantlucifer@gmail.com> * [Improve][Zeta] Don't trigger handleSaveMode when restore (apache#5192) * move imap storage file dependency packages to submodules (apache#5218) * [Hotfix][CI]Declare files that will always have UNIX line endings on checkout. (apache#5221) * [Hotfix][Connector-V2][Paimon] Bump paimon-bundle version to 0.4.0-incubating (apache#5219) * [Docs][Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL apache#4590 (apache#4757) * [Docs][Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL * [Docs] Fix Dockerfile and seatunnel-flink.yaml in Set Up with Kubernetes (apache#4793) * [Docs] update seatunnel-flink.yaml and Dockerfile to help the demo work * [Docs] update release-note apache#4788 --------- Co-authored-by: flynnxue <flynnxue@lilithgames.com> Co-authored-by: ic4y <83933160+ic4y@users.noreply.github.com> * [feature][doris] Doris factory type (apache#5061) * [feature][doris] Web need factory and data type convertor * [Fix] Update the Readme (apache#4968) Use the better description for the SeaTunnel Project * [CI] Split updated modules integration test for part 5 (apache#5208) * [CI] Split updated modules integration test for part 5 * [CI] Split updated modules integration test for part 5 * Split e2e * update json-smart * fix dm error * fix dm error * fix dm error * fix dm error * fix dm error * fix dm error * fix dm error * revert code * revert code * revert code --------- Co-authored-by: gdliu3 <gdliu3@iflytek.com> * [Feature][CDC][Zeta] Support schema evolution framework(DDL) (apache#5125) * Fixed IMap file storage e2e bug (apache#5237) * [Improve] [Connector-V2] Remove scheduler in JDBC sink apache#4736 (apache#5168) --------- Co-authored-by: gdliu3 <gdliu3@iflytek.com> * [Doc] [JDBC Oracle] Add JDBC Oracle Documentation (apache#5239) * [Feature][Zeta][REST-API]Add REST API To Submit Job (apache#5107) * [Fix] Update the project description (apache#4967) * Update the project description * [Feature][Zeta] Support history service record job execute error (apache#5114) * fix:hdfs Checkpoint Storage management fails to delete historical files * fix:hdfs Checkpoint Storage management fails to delete historical files * fix after the savepoint job is restored, the checkpoint file cannot be generated * [Feature][Zeta] Support history service record job execute error * Improve Jobstate-related class additions add serialVersionUID * add e2e test * [hotfix]Update .asf.yaml (apache#5242) * Update .asf.yaml * [Hotfix]Fix array index anomalies caused by apache#5057 (apache#5195) * [bugfix] [savepoint test] Turn on the testSavepoint test. (apache#5199) * [BUG][Connector-V2][Jdbc] support postgresql json type (apache#5194) * add Postgresql json type Co-authored-by: 80597928 <673421862@qq.com> * [Bugfix][cdc] Fix mysql bit column to java byte (apache#4817) * [Bugfix][AmazonDynamoDB] Fix the problem that all table data cannot be obtained (apache#5146) * [Docs][Connector][Source][jdbc]Change the line boundary store value type to BigDecimal (apache#4900) * [bug][jdbc][oracle]Fix the Oracle number type mapping problem (apache#5209) * [Bugfix][zeta] Fix the serialization issue of GetMetricsOperation during multi-node operation. (apache#5206) * [Hotfix][Zeta] Avoid Redundant Job Submissions by Checking Job Status (apache#5229) * [Bugfix][zeta] Fixed the issue of duplicated metrics caused by job fault tolerance or restore. (apache#5214) * [Imporve] [CDC Base] Add a fast sampling method that supports character types (apache#5179) * fixed zeta ci error (apache#5254) * [Doc][README] Remove useless github workflow, and adjust description of 'engineering structure'. (apache#4305) * [Feature][Zeta]The expiration time of a historical Job can be config (apache#5180) * fix:hdfs Checkpoint Storage management fails to delete historical files Co-authored-by: hailin0 <wanghailin@apache.org> * [bugfix] [e2e] Fixed a minor bug (apache#5274) * [Improve][SQL] Support use catalogTableName as SQL expression (apache#5273) * [Doc] Improve S3File Source & S3File Sink document (apache#5101) * Improve S3File Source & S3File Sink document * Fix style error (apache#5280) * Fix StarRocksJsonSerializer will transform array/map/row to string (apache#5281) * [Docs][Connector-V2][MyHours]Reconstruct the MyHours connector document (apache#5129) * [Docs][Connector-V2][MyHours]Reconstruct the MyHours connector document * fix format * fix format * [Improve][API & Zeta] Using connector custom serializer encode/decode states (apache#5238) * API: Using DefaultSerializer as connector sink default serializer * Zeta: Using connector custom serializer encode/decode states * [Feature][Connector-V2] connector-kafka source support data conversion extracted by kafka connect source (apache#4516) * Compatible kafka connect json apache#4137 * [Improve][CI/CD] Remove 'paths-ignore', enable the code style check for markdown files. (apache#5286) * [Bugfix][zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0. (apache#5263) * [Bugfix][zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0. * remove max-concurrent * [Feature][Connector-v2][RedisSink]Support redis to set expiration time. (apache#4975) * Support redis to set expiration time. * Set redis expire default value. * add e2e test. * add e2e test. * modify config file name. --------- Co-authored-by: lightzhao <zhaolianyong777@gmail.com> * [bugfix] Fix testGetErrorInfo case error (apache#5282) * [Feature][Zeta] Checkpoint support hdfs ha mode (apache#4942) * fix browser long type intercept (apache#5267) Co-authored-by: 80597928 <673421862@qq.com> * [Docs] remove `incubating` keyword in document (apache#5257) * [feature][web] hive add option because web need (apache#5154) * [feature][web] hive add option because web need * [feature][web] hive add option read_columns * [feature][web] required update optional * [bugfix] mvn spotless * fix conf * fix conf --------- Co-authored-by: liuli <m_liuli@163.com> * [Bug][flink-runtime][connectors-v2] Flink register table Environment The running mode is set to`job.mode` (apache#4826) * [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document (apache#5132) * [Docs][Connector-V2][StarRocks]Reconstruct the StarRocks connector document * [Improve][Connector-v2][HiveSink]remove drop partition when abort. (apache#4940) Co-authored-by: lightzhao <zhaolianyong777@gmail.com> Co-authored-by: liuli <m_liuli@163.com> Co-authored-by: ic4y <83933160+ic4y@users.noreply.github.com> * [Docs][Connector-V2][SelectDB-Cloud]Reconstruct the SelectDB-Cloud connector document (apache#5130) * [Docs][Connector-V2][SelectDB-Cloud]Reconstruct the SelectDB-Cloud connector document * fix codestyle --------- Co-authored-by: liuli <m_liuli@163.com> * [Docs][Connector-V2][HDFS]Refactor connector-v2 docs using unified format HDFS. (apache#4871) * Refactor connector-v2 docs using unified format HDFS. * add data type. * update. * add key feature. * add hdfs_site_path * 1.add data type. 2.add hdfs_site_path conf. * add data type. * add hdfs site conf. --------- Co-authored-by: lightzhao <zhaolianyong777@gmail.com> Co-authored-by: liuli <m_liuli@163.com> * [Improve] [Connector-V2] Remove scheduler in Tablestore sink (apache#5272) --------- Co-authored-by: gdliu3 <gdliu3@iflytek.com> * [BUG][Connector-V2][Mongo-cdc] Incremental data kind error in snapshot phase (apache#5184) * [BUG][Connector-V2][Mongo-cdc] Incremental data kind error in snapshot phase * [Hotfix] Fix com.google.common.base.Preconditions to seatunnel shade one (apache#5284) * [Merge] Fix merge conflict and fix jdbc fieldIde with compatibleMode confusion --------- Co-authored-by: Cason-ACE <35160064+cason0126@users.noreply.github.com> Co-authored-by: Tyrantlucifer <TyrantLucifer@gmail.com> Co-authored-by: hailin0 <wanghailin@apache.org> Co-authored-by: Xiaojian Sun <sunxiaojian926@163.com> Co-authored-by: Laglangyue <35491928+laglangyue@users.noreply.github.com> Co-authored-by: ZhilinLi <zhilinli0706@gmail.com> Co-authored-by: ic4y <83933160+ic4y@users.noreply.github.com> Co-authored-by: Hao Xu <sduxuhao@gmail.com> Co-authored-by: Eric <gaojun2048@gmail.com> Co-authored-by: Bibo <33744252+531651225@users.noreply.github.com> Co-authored-by: 毕博 <bibo@mafengwo.com> Co-authored-by: Carl-Zhou-CN <67902676+Carl-Zhou-CN@users.noreply.github.com> Co-authored-by: zhouyao <yao.zhou@marketingforce.com> Co-authored-by: Marvin <29311598@qq.com> Co-authored-by: monster <60029759+MonsterChenzhuo@users.noreply.github.com> Co-authored-by: gnehil <adamlee489@gmail.com> Co-authored-by: TaoZex <45089228+TaoZex@users.noreply.github.com> Co-authored-by: xiaofan2012 <41982310+xiaofan2022@users.noreply.github.com> Co-authored-by: wantao <wantao@inmyshow.com> Co-authored-by: Guangdong Liu <804167098@qq.com> Co-authored-by: zhilinli <lzl15844876351@163.com> Co-authored-by: zhaifengbing <ainizfb@126.com> Co-authored-by: dalong <60906603+alibabaMapengfei@users.noreply.github.com> Co-authored-by: FuYouJ <1247908487@qq.com> Co-authored-by: davidfans <136911434+davidfans@users.noreply.github.com> Co-authored-by: Fan Donglai <ddna_1022@163.com> Co-authored-by: gdliu3 <gdliu3@iflytek.com> Co-authored-by: DismalSnail <yepenghaobit@gmail.com> Co-authored-by: lightzhao <40714172+lightzhao@users.noreply.github.com> Co-authored-by: chenzy15 <chenzy15@ziroom.com> Co-authored-by: wssmao <39487209+wssmao@users.noreply.github.com> Co-authored-by: wsstony <tonymao777@163.com> Co-authored-by: lightzhao <zhaolianyong777@gmail.com> Co-authored-by: XiaoJiang521 <131635688+XiaoJiang521@users.noreply.github.com> Co-authored-by: mcy <rewrma@163.com> Co-authored-by: yctanGmail <138592845+yctanGmail@users.noreply.github.com> Co-authored-by: yctan <1417983443@qq.com> Co-authored-by: wu-a-ge <wfygowxf1@163.com> Co-authored-by: 司马琦昂 <bruce.maqiang@gmail.com> Co-authored-by: happyboy1024 <137260654+happyboy1024@users.noreply.github.com> Co-authored-by: He Wang <wanghechn@qq.com> Co-authored-by: silenceland <silenceland23@163.com> Co-authored-by: changhuyan <877018069@qq.com> Co-authored-by: Jarvis <liunaijie1996@163.com> Co-authored-by: 阿丙 <50567478+gaopeng666@users.noreply.github.com> Co-authored-by: jackyyyyyssss <127465317+jackyyyyyssss@users.noreply.github.com> Co-authored-by: 80597928 <Lzl@qq.com> Co-authored-by: Chengyu Yan <cheneyyin@hotmail.com> Co-authored-by: zhangchengming601 <86779821+zhangchengming601@users.noreply.github.com> Co-authored-by: lihjChina <237206177@qq.com> Co-authored-by: David Zollo <davidzollo365@gmail.com> Co-authored-by: EchoLee5 <39044001+EchoLee5@users.noreply.github.com> Co-authored-by: dengdi <114273849+dengd1937@users.noreply.github.com> Co-authored-by: dengd1937 <dengd1803@gmail.com> Co-authored-by: happyboy1024 <296442618@qq.com> Co-authored-by: FlechazoW <35768015+FlechazoW@users.noreply.github.com> Co-authored-by: 80597928 <673421862@qq.com> Co-authored-by: kun <66303359+Lifu12@users.noreply.github.com> Co-authored-by: Volodymyr <volodymyrduke@gmail.com> Co-authored-by: javalover123 <javalover123@foxmail.com> Co-authored-by: Volodymyr <770925351@qq.com> Co-authored-by: kksxf <flynnxue02@gmail.com> Co-authored-by: flynnxue <flynnxue@lilithgames.com> Co-authored-by: fang <56808812+zhibinF@users.noreply.github.com> Co-authored-by: gejinxin <844156709@qq.com> Co-authored-by: Wenjun Ruan <wenjun@apache.org> Co-authored-by: Koyfin <1040080742@qq.com> Co-authored-by: liuli <m_liuli@163.com>

[Bug] [connector-jdbc] Mysql bigint(20) cannot be used as a partition…

1af62b6

…_column apache#4627

zhilinli123 changed the title ~~[Bug] [connector-jdbc] Fix Mysql bigint(20) cannot be used as a partition_column bug~~ [Bug] [connector-jdbc] Fix Mysql bigint(20) unsigned cannot be used as a partition_column bug Apr 20, 2023

TyrantLucifer changed the title ~~[Bug] [connector-jdbc] Fix Mysql bigint(20) unsigned cannot be used as a partition_column bug~~ [Hotfix][Connector-V2][Jdbc] Fix Mysql bigint(20) unsigned cannot be used as a partition_column bug Apr 20, 2023

ic4y changed the title ~~[Hotfix][Connector-V2][Jdbc] Fix Mysql bigint(20) unsigned cannot be used as a partition_column bug~~ [Improve][Connector-V2][Jdbc-Source]Support Mysql bigint(20) used as a partition_column Apr 21, 2023

hailin0 reviewed Apr 21, 2023

View reviewed changes

[Bug] [connector-jdbc] Mysql bigint(20) cannot be used as a partition…

9107911

…_column apache#4627

ic4y added the discuss label Apr 26, 2023

ic4y requested changes Apr 27, 2023

View reviewed changes

zhilinli123 mentioned this pull request May 9, 2023

[OnlineMeeting&May.9]SeaTunnel community meeting Topic collect meeting #4724

Closed

Merge remote-tracking branch 'origin/dev' into fix-mysql-read-bigint-…

743cc1a

…unsigned-partition

zhilinli123 changed the title ~~[Improve][Connector-V2][Jdbc-Source]Support Mysql bigint(20) used as a partition_column~~ [Improve][Connector-V2][Jdbc-Source]Support Compatible Mysql bigint(20) used as a partition_column May 11, 2023

zhilinli123 added 2 commits May 11, 2023 11:53

Merge branch 'apache:dev' into fix-mysql-read-bigint-unsigned-partition

e44988a

[Improve][Connector-V2][Jdbc-Source]Support Compatible Mysql bigint(2…

14c8612

…0) used as a partition_column apache#4634

ic4y previously approved these changes May 11, 2023

View reviewed changes

github-actions bot added approved reviewed labels May 11, 2023

[Improve][Connector-V2][Jdbc-Source]Support Compatible Mysql bigint(2…

ac113da

…0) used as a partition_column apache#4634

zhilinli123 dismissed ic4y’s stale review via ac113da May 11, 2023 15:57

github-actions bot removed approved reviewed labels May 11, 2023

hailin0 reviewed May 19, 2023

View reviewed changes

zhilinli123 added 3 commits May 19, 2023 16:21

Merge remote-tracking branch 'origin/dev' into fix-mysql-read-bigint-…

3945a67

…unsigned-partition

[Improve][Connector-V2][Jdbc-Source]Support Compatible Mysql bigint(2…

975436e

…0) used as a partition_column apache#4634

Merge branch 'apache:dev' into fix-mysql-read-bigint-unsigned-partition

179af77

ic4y previously approved these changes May 23, 2023

View reviewed changes

github-actions bot added approved reviewed labels May 23, 2023

zhilinli123 dismissed ic4y’s stale review via b601b1c May 27, 2023 06:03

github-actions bot removed approved reviewed labels May 27, 2023

[Improve][Connector-V2][Jdbc-Source]Support Compatible Mysql bigint(2…

0496de0

…0) used as a partition_column apache#4634

zhilinli123 closed this May 27, 2023

zhilinli123 reopened this May 27, 2023

zhilinli123 changed the title ~~[Improve][Connector-V2][Jdbc-Source]Support Compatible Mysql bigint(20) used as a partition_column~~ [Improve][Connector-V2][Jdbc-Source] Support for Decimal types as splict keys May 30, 2023

ic4y approved these changes Jun 7, 2023

View reviewed changes

github-actions bot added approved reviewed labels Jun 7, 2023

davidzollo approved these changes Jun 7, 2023

View reviewed changes

ic4y merged commit d56bb1b into apache:dev Jun 7, 2023

zhilinli123 mentioned this pull request Jun 7, 2023

[Docs][Connector][Source][jdbc]Change the line boundary store value type to BigDecimal #4900

Merged

EricJoy2048 pushed a commit to EricJoy2048/incubator-seatunnel that referenced this pull request Jun 30, 2023

[Improve][Connector-V2][Jdbc-Source] Support for Decimal types as spl…

05cc51b

…ict keys (apache#4634)

[Improve][Connector-V2][Jdbc-Source] Support for Decimal types as splict keys #4634

[Improve][Connector-V2][Jdbc-Source] Support for Decimal types as splict keys #4634

Conversation

zhilinli123 commented Apr 20, 2023

Purpose of this pull request

Check list

ic4y commented Apr 20, 2023

ic4y commented Apr 21, 2023

zhilinli123 commented Apr 21, 2023 • edited Loading

ic4y commented Apr 21, 2023

ic4y commented Apr 21, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhilinli123 commented Apr 24, 2023

ic4y commented Apr 25, 2023

ic4y left a comment

Choose a reason for hiding this comment

zhilinli123 commented May 9, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhilinli123 commented May 15, 2023

ic4y commented May 16, 2023

zhilinli123 commented May 16, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zhilinli123 commented May 23, 2023

ic4y left a comment

Choose a reason for hiding this comment

zhilinli123 commented May 25, 2023 • edited Loading

hailin0 commented May 30, 2023

zhilinli123 commented May 30, 2023

davidzollo left a comment

Choose a reason for hiding this comment

zhilinli123 commented Jun 7, 2023

zhilinli123 commented Apr 21, 2023 •

edited

Loading

zhilinli123 commented May 16, 2023 •

edited

Loading

zhilinli123 commented May 25, 2023 •

edited

Loading