the sink-connector lightweight should retry on failures

[aadant]

I noticed that the data did not reconcile between CH and MySQL.

1714737 2024-02-15 10:28:37.820 [Sink Connector thread-pool-4] ERROR com.altinity.clickhouse.sink.connector.db.batch.PreparedStatementExecutor  - ******* ERROR inserting Batch *****************
java.sql.BatchUpdateException: Error writing request body to server, server ClickHouseNode [uri=http://host:8123/db, options={custom_settings=allow_experimental_object_type=1,insert_allow_materialized_columns=1,client_name=Sink Connector Lightweight}]@205662767
        at com.clickhouse.jdbc.SqlExceptionUtils.batchUpdateError(SqlExceptionUtils.java:107)
        at com.clickhouse.jdbc.internal.InputBasedPreparedStatement.executeAny(InputBasedPreparedStatement.java:154)
        at com.clickhouse.jdbc.internal.AbstractPreparedStatement.executeLargeBatch(AbstractPreparedStatement.java:85)
        at com.clickhouse.jdbc.internal.ClickHouseStatementImpl.executeBatch(ClickHouseStatementImpl.java:752)
        at com.altinity.clickhouse.sink.connector.db.batch.PreparedStatementExecutor.lambda$executePreparedStatement$0(PreparedStatementExecutor.java:156)
        at java.base/java.lang.Iterable.forEach(Iterable.java:75)
        at com.altinity.clickhouse.sink.connector.db.batch.PreparedStatementExecutor.executePreparedStatement(PreparedStatementExecutor.java:117)
        at com.altinity.clickhouse.sink.connector.db.batch.PreparedStatementExecutor.addToPreparedStatementBatch(PreparedStatementExecutor.java:88)
        at com.altinity.clickhouse.sink.connector.executor.ClickHouseBatchRunnable.flushRecordsToClickHouse(ClickHouseBatchRunnable.java:300)
        at com.altinity.clickhouse.sink.connector.executor.ClickHouseBatchRunnable.processRecordsByTopic(ClickHouseBatchRunnable.java:266)
        at com.altinity.clickhouse.sink.connector.executor.ClickHouseBatchRunnable.run(ClickHouseBatchRunnable.java:144)
        at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
        at java.base/java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305)
        at java.base/java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
        at java.base/java.lang.Thread.run(Thread.java:833)
1714741 2024-02-15 10:28:37.824 [Sink Connector thread-pool-4] ERROR com.altinity.clickhouse.sink.connector.db.batch.PreparedStatementExecutor  - **** ERROR: executing prepared statement
1714741 2024-02-15 10:28:37.824 [Sink Connector thread-pool-4] ERROR com.altinity.clickhouse.sink.connector.executor.ClickHouseBatchRunnable  - Error processing records for topic: embeddedconnector.

when rewinding to the replication position, the insert succeeded.

So failing batches should be retried (forever). Replication position should not make progress until the error is fixed.

Errors should not be snoozed but retried. Retrying transient failures will make replication more robust.

[Selfeer]

The issue was verified by the Altinity QA team and marked as qa-verified.

Build used for testing: altinityinfra/clickhouse-sink-connector:464-c979a8c2583ff701dd5b63efb8f36f7fff8111e7-lt

Automated TestFlows tests were added to the develop branch:
https://github.com/Altinity/clickhouse-sink-connector/blob/develop/sink-connector-lightweight/tests/integration/tests/retry_on_fail.py

Requirements for the ClickHouse Sink Connector Lightweight were updated.