Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use Hikari Connection Pool for handling Clickhouse connections #893

Open
wants to merge 11 commits into
base: 2.5.1
Choose a base branch
from
Open

Use Hikari Connection Pool for handling Clickhouse connections #893

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
eb46e45
Use Hikari Connection Pool for handling Clickhouse connections
subkanthi Oct 31, 2024
68dad63
Fixed imports
subkanthi Oct 31, 2024
af91740
Changes to use Hikari connection pool, Replaced ClickHouseConnection …
subkanthi Nov 8, 2024
c47d513
Set connection timeout in hikari pool to 50 seconds
subkanthi Nov 8, 2024
83cfcd4
Set connection pool to 100 in hikari
subkanthi Nov 8, 2024
03beb53
Merged conflicts with 2.5.1 branch.
subkanthi Jan 26, 2025
b8285ef
Merged conflicts with 2.5.1 branch.
subkanthi Jan 26, 2025
a5a2bb4
Changes to all integration tests to move the common logic of creating…
subkanthi Jan 27, 2025
ec2d634
Changes to all integration tests to move the common logic of creating…
subkanthi Jan 27, 2025
62ce23d
Changes to all integration tests to move the common logic of creating…
subkanthi Jan 27, 2025
00032c2
Changes to support one HikariDbSource for one database.
subkanthi Jan 27, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
38 changes: 29 additions & 9 deletions sink-connector-lightweight/dependency-reduced-pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -162,10 +162,6 @@
<artifactId>oracle-xe</artifactId>
<groupId>org.testcontainers</groupId>
</exclusion>
<exclusion>
<artifactId>junit-platform-launcher</artifactId>
<groupId>org.junit.platform</groupId>
</exclusion>
<exclusion>
<artifactId>junit-jupiter</artifactId>
<groupId>org.junit.jupiter</groupId>
Expand Down Expand Up @@ -289,10 +285,6 @@
<version>5.8.1</version>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>junit-platform-engine</artifactId>
<groupId>org.junit.platform</groupId>
</exclusion>
<exclusion>
<artifactId>junit-jupiter-api</artifactId>
<groupId>org.junit.jupiter</groupId>
Expand All @@ -317,10 +309,38 @@
<artifactId>junit-jupiter-api</artifactId>
<groupId>org.junit.jupiter</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-launcher</artifactId>
<version>1.8.2</version>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>apiguardian-api</artifactId>
<groupId>org.apiguardian</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-engine</artifactId>
<version>1.8.2</version>
<scope>test</scope>
<exclusions>
<exclusion>
<artifactId>junit-platform-launcher</artifactId>
<artifactId>opentest4j</artifactId>
<groupId>org.opentest4j</groupId>
</exclusion>
<exclusion>
<artifactId>junit-platform-commons</artifactId>
<groupId>org.junit.platform</groupId>
</exclusion>
<exclusion>
<artifactId>apiguardian-api</artifactId>
<groupId>org.apiguardian</groupId>
</exclusion>
</exclusions>
</dependency>
<dependency>
Expand Down
17 changes: 17 additions & 0 deletions sink-connector-lightweight/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -233,6 +233,11 @@
<!-- <scope>test</scope>-->
</dependency>

<dependency>
<groupId>com.zaxxer</groupId>
<artifactId>HikariCP</artifactId>
<version>6.0.0</version>
</dependency>
<dependency>
<groupId>org.antlr</groupId>
<artifactId>antlr4-runtime</artifactId>
Expand Down Expand Up @@ -334,6 +339,18 @@
<scope>test</scope>
</dependency>

<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-launcher</artifactId>
<version>1.8.2</version>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.junit.platform</groupId>
<artifactId>junit-platform-engine</artifactId>
<version>1.8.2</version>
<scope>test</scope>
</dependency>
<dependency> <!-- necessary for Java 9+ -->
<groupId>org.apache.tomcat</groupId>
<artifactId>annotations-api</artifactId>
Expand Down
32 changes: 18 additions & 14 deletions ...c/main/java/com/altinity/clickhouse/debezium/embedded/cdc/DebeziumChangeEventCapture.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,6 @@

import com.altinity.clickhouse.debezium.embedded.common.PropertiesHelper;
import com.altinity.clickhouse.debezium.embedded.config.SinkConnectorLightWeightConfig;
import com.altinity.clickhouse.debezium.embedded.ddl.parser.DDLParserService;
import com.altinity.clickhouse.debezium.embedded.ddl.parser.MySQLDDLParserService;
import com.altinity.clickhouse.debezium.embedded.parser.DebeziumRecordParserService;
import com.altinity.clickhouse.sink.connector.ClickHouseSinkConnectorConfig;
Expand All @@ -17,7 +16,6 @@
import com.altinity.clickhouse.sink.connector.executor.DebeziumOffsetManagement;
import com.altinity.clickhouse.sink.connector.model.ClickHouseStruct;
import com.altinity.clickhouse.sink.connector.model.DBCredentials;
import com.clickhouse.jdbc.ClickHouseConnection;
import com.google.common.annotations.VisibleForTesting;
import com.google.common.util.concurrent.ThreadFactoryBuilder;
import io.debezium.config.CommonConnectorConfig;
Expand All @@ -41,6 +39,7 @@
import org.json.simple.parser.ParseException;

import java.io.IOException;
import java.sql.Connection;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
import java.sql.SQLException;
Expand Down Expand Up @@ -89,7 +88,7 @@ public class DebeziumChangeEventCapture {
DebeziumEngine<ChangeEvent<SourceRecord, SourceRecord>> engine;

// Keep one clickhouse connection.
private ClickHouseConnection conn;
private Connection conn;

ClickHouseBatchWriter singleThreadedWriter;

Expand Down Expand Up @@ -197,7 +196,8 @@ private String getDatabaseName(SourceRecord sr) {
private BaseDbWriter createWriter(ClickHouseSinkConnectorConfig config, String databaseName) {
DBCredentials dbCredentials = parseDBConfiguration(config);
String jdbcUrl = BaseDbWriter.getConnectionString(dbCredentials.getHostName(), dbCredentials.getPort(), databaseName);
ClickHouseConnection conn = BaseDbWriter.createConnection(jdbcUrl, "Client_1", dbCredentials.getUserName(), dbCredentials.getPassword(), config);
Connection conn = BaseDbWriter.createConnection(jdbcUrl, BaseDbWriter.DATABASE_CLIENT_NAME, dbCredentials.getUserName(), dbCredentials.getPassword(),
databaseName, config);
return new BaseDbWriter(dbCredentials.getHostName(), dbCredentials.getPort(), databaseName, dbCredentials.getUserName(), dbCredentials.getPassword(), config, conn);
}

Expand Down Expand Up @@ -383,18 +383,19 @@ private void createDatabaseForDebeziumStorage(ClickHouseSinkConnectorConfig conf
DBCredentials dbCredentials = parseDBConfiguration(config);

String jdbcUrl = BaseDbWriter.getConnectionString(dbCredentials.getHostName(), dbCredentials.getPort(),
"system");
ClickHouseConnection conn = BaseDbWriter.createConnection(jdbcUrl, "Client_1", dbCredentials.getUserName(), dbCredentials.getPassword(), config);
BaseDbWriter.SYSTEM_DB);
Connection conn = BaseDbWriter.createConnection(jdbcUrl, BaseDbWriter.DATABASE_CLIENT_NAME,
dbCredentials.getUserName(), dbCredentials.getPassword(), BaseDbWriter.SYSTEM_DB, config);
BaseDbWriter writer = new BaseDbWriter(dbCredentials.getHostName(), dbCredentials.getPort(),
"system", dbCredentials.getUserName(),
BaseDbWriter.SYSTEM_DB, dbCredentials.getUserName(),
dbCredentials.getPassword(), config, conn);

Pair<String, String> tableNameDatabaseName = getDebeziumOffsetStorageDatabaseName(props);
String databaseName = tableNameDatabaseName.getRight();

String createDbQuery = String.format("create database if not exists %s", databaseName);
log.info("CREATING DEBEZIUM STORAGE Database: " + createDbQuery);
writer.executeQuery(createDbQuery);
writer.executeSystemQuery(createDbQuery);

break;
} catch (Exception e) {
Expand Down Expand Up @@ -422,13 +423,14 @@ private void createSchemaHistoryTable(ClickHouseSinkConnectorConfig config, Prop
DBCredentials dbCredentials = parseDBConfiguration(config);
String jdbcUrl = BaseDbWriter.getConnectionString(dbCredentials.getHostName(), dbCredentials.getPort(),
"system");
ClickHouseConnection conn = BaseDbWriter.createConnection(jdbcUrl, "Client_1",dbCredentials.getUserName(), dbCredentials.getPassword(), config);
Connection conn = BaseDbWriter.createConnection(jdbcUrl, BaseDbWriter.DATABASE_CLIENT_NAME,
dbCredentials.getUserName(), dbCredentials.getPassword(), BaseDbWriter.SYSTEM_DB, config);
BaseDbWriter writer = new BaseDbWriter(dbCredentials.getHostName(), dbCredentials.getPort(),
"system", dbCredentials.getUserName(),
dbCredentials.getPassword(), config, conn);

try {
writer.executeQuery(createSchemaHistoryTable);
writer.executeSystemQuery(createSchemaHistoryTable);
} catch(Exception e) {
log.error("Error creating schema history table", e);
}
Expand All @@ -449,7 +451,8 @@ private void createViewForShowReplicaStatus(ClickHouseSinkConnectorConfig config

String jdbcUrl = BaseDbWriter.getConnectionString(dbCredentials.getHostName(), dbCredentials.getPort(),
"system");
ClickHouseConnection conn = BaseDbWriter.createConnection(jdbcUrl, "Client_1",dbCredentials.getUserName(), dbCredentials.getPassword(), config);
Connection conn = BaseDbWriter.createConnection(jdbcUrl, BaseDbWriter.DATABASE_CLIENT_NAME,
dbCredentials.getUserName(), dbCredentials.getPassword(), BaseDbWriter.SYSTEM_DB, config);
BaseDbWriter writer = new BaseDbWriter(dbCredentials.getHostName(), dbCredentials.getPort(),
"system", dbCredentials.getUserName(),
dbCredentials.getPassword(), config, conn);
Expand All @@ -462,7 +465,7 @@ private void createViewForShowReplicaStatus(ClickHouseSinkConnectorConfig config
// Remove quotes.
formattedView = formattedView.replace("\"", "");
try {
writer.executeQuery(formattedView);
writer.executeSystemQuery(formattedView);
} catch(Exception e) {
log.error("**** Error creating VIEW **** " + formattedView);
}
Expand Down Expand Up @@ -539,8 +542,9 @@ public String getDebeziumStorageStatus(ClickHouseSinkConnectorConfig config, Pro
log.error("**** Connection to ClickHouse is not established, re-initiating ****");
String jdbcUrl = BaseDbWriter.getConnectionString(dbCredentials.getHostName(), dbCredentials.getPort(),
databaseName);
ClickHouseConnection conn = BaseDbWriter.createConnection(jdbcUrl, "Client_1",
dbCredentials.getUserName(), dbCredentials.getPassword(), config);
Connection conn = BaseDbWriter.createConnection(jdbcUrl, BaseDbWriter.DATABASE_CLIENT_NAME,
dbCredentials.getUserName(), dbCredentials.getPassword(),
BaseDbWriter.SYSTEM_DB, config);
writer = new BaseDbWriter(dbCredentials.getHostName(), dbCredentials.getPort(),
databaseName, dbCredentials.getUserName(),
dbCredentials.getPassword(), config, conn);
Expand Down
8 changes: 4 additions & 4 deletions ...ht/src/main/java/com/altinity/clickhouse/debezium/embedded/cdc/DebeziumOffsetStorage.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ public void deleteOffsetStorageRow(String offsetKey,

// String connectorName = config.getString("connector.name");
String debeziumStorageStatusQuery = String.format("delete from %s where offset_key='%s'" , tableName, offsetKey);
writer.executeQuery(debeziumStorageStatusQuery);
writer.executeSystemQuery(debeziumStorageStatusQuery);
}

/**
Expand All @@ -67,7 +67,7 @@ public void deleteSchemaHistoryTable(String offsetKey,

String debeziumStorageStatusQuery = String.format("delete from `%s` where JSONExtractRaw(JSONExtractRaw(history_data,'source'), 'server')='%s'" , tableName, offsetKey);
log.info("Deleting schema history table query: " + debeziumStorageStatusQuery);
writer.executeQuery(debeziumStorageStatusQuery);
writer.executeSystemQuery(debeziumStorageStatusQuery);
}
/**
* Function to get the latest timestamp of the record in the table
Expand All @@ -81,7 +81,7 @@ public String getDebeziumLatestRecordTimestamp(Properties props, BaseDbWriter wr
JdbcOffsetBackingStoreConfig.PROP_TABLE_NAME.name());

String debeziumLatestRecordTimestampQuery = String.format("select max(record_insert_ts) from %s" , tableName);
return writer.executeQuery(debeziumLatestRecordTimestampQuery);
return writer.executeSystemQuery(debeziumLatestRecordTimestampQuery);
}

public String getDebeziumStorageStatusQuery(
Expand All @@ -92,7 +92,7 @@ public String getDebeziumStorageStatusQuery(
String offsetKey = getOffsetKey(props);
// String connectorName = config.getString("connector.name");
String debeziumStorageStatusQuery = String.format("select offset_val from %s where offset_key='%s'" , tableName, offsetKey);
return writer.executeQuery(debeziumStorageStatusQuery);
return writer.executeSystemQuery(debeziumStorageStatusQuery);
}

/**
Expand Down
10 changes: 3 additions & 7 deletions ...y/clickhouse/debezium/embedded/ClickHouseDebeziumEmbeddedPostgresDecoderBufsDockerIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.utility.DockerImageName;

import java.sql.Connection;
import java.sql.ResultSet;
import java.util.HashMap;
import java.util.Map;
Expand Down Expand Up @@ -90,13 +91,8 @@
Thread.sleep(10000);//
Thread.sleep(50000);

String jdbcUrl = BaseDbWriter.getConnectionString(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(),
"public");
ClickHouseConnection chConn = BaseDbWriter.createConnection(jdbcUrl, "Client_1",
clickHouseContainer.getUsername(), clickHouseContainer.getPassword(), new ClickHouseSinkConnectorConfig(new HashMap<>()));

BaseDbWriter writer = new BaseDbWriter(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(),
"public", clickHouseContainer.getUsername(), clickHouseContainer.getPassword(), null, chConn);
BaseDbWriter writer = ITCommon.getDBWriter(clickHouseContainer);

Map<String, String> tmColumns = writer.getColumnsDataTypesForTable("tm");
Assert.assertTrue(tmColumns.size() == 22);
Assert.assertTrue(tmColumns.get("id").equalsIgnoreCase("UUID"));
Expand All @@ -106,7 +102,7 @@


int tmCount = 0;
ResultSet chRs = writer.getConnection().prepareStatement("select count(*) from tm").executeQuery();

Check failure on line 105 in sink-connector-lightweight/src/test/java/com/altinity/clickhouse/debezium/embedded/ClickHouseDebeziumEmbeddedPostgresDecoderBufsDockerIT.java

View workflow job for this annotation

GitHub Actions / JUnit Test Report

ClickHouseDebeziumEmbeddedPostgresDecoderBufsDockerIT.testDecoderBufsPlugin 

Code: 60. DB::Exception: Unknown table expression identifier 'tm' in scope SELECT count(*) FROM tm. (UNKNOWN_TABLE) (version 24.12.3.47 (official build))
Raw output 
java.sql.BatchUpdateException: 
Code: 60. DB::Exception: Unknown table expression identifier 'tm' in scope SELECT count(*) FROM tm. (UNKNOWN_TABLE) (version 24.12.3.47 (official build))

	at com.altinity.clickhouse.debezium.embedded.ClickHouseDebeziumEmbeddedPostgresDecoderBufsDockerIT.testDecoderBufsPlugin(ClickHouseDebeziumEmbeddedPostgresDecoderBufsDockerIT.java:105)
while(chRs.next()) {
tmCount = chRs.getInt(1);
}
Expand Down
10 changes: 3 additions & 7 deletions ...nity/clickhouse/debezium/embedded/ClickHouseDebeziumEmbeddedPostgresPgoutputDockerIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,7 @@
import org.testcontainers.junit.jupiter.Container;
import org.testcontainers.utility.DockerImageName;

import java.sql.Connection;
import java.sql.ResultSet;
import java.util.HashMap;
import java.util.Map;
Expand Down Expand Up @@ -94,13 +95,8 @@


// Create connection.
String jdbcUrl = BaseDbWriter.getConnectionString(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(),
"public");
ClickHouseConnection conn = BaseDbWriter.createConnection(jdbcUrl, "Client_1",
clickHouseContainer.getUsername(), clickHouseContainer.getPassword(), new ClickHouseSinkConnectorConfig(new HashMap<>()));

BaseDbWriter writer = new BaseDbWriter(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(),
"public", clickHouseContainer.getUsername(), clickHouseContainer.getPassword(), null, conn);
BaseDbWriter writer = ITCommon.getDBWriter(clickHouseContainer);

Map<String, String> tmColumns = writer.getColumnsDataTypesForTable("tm");
Assert.assertTrue(tmColumns.size() == 22);
Assert.assertTrue(tmColumns.get("id").equalsIgnoreCase("UUID"));
Expand All @@ -110,7 +106,7 @@


int tmCount = 0;
ResultSet chRs = writer.getConnection().prepareStatement("select count(*) from tm").executeQuery();

Check failure on line 109 in sink-connector-lightweight/src/test/java/com/altinity/clickhouse/debezium/embedded/ClickHouseDebeziumEmbeddedPostgresPgoutputDockerIT.java

View workflow job for this annotation

GitHub Actions / JUnit Test Report

ClickHouseDebeziumEmbeddedPostgresPgoutputDockerIT.testPgOutputPlugin 

Code: 60. DB::Exception: Unknown table expression identifier 'tm' in scope SELECT count(*) FROM tm. (UNKNOWN_TABLE) (version 24.12.3.47 (official build))
Raw output 
java.sql.BatchUpdateException: 
Code: 60. DB::Exception: Unknown table expression identifier 'tm' in scope SELECT count(*) FROM tm. (UNKNOWN_TABLE) (version 24.12.3.47 (official build))

	at com.altinity.clickhouse.debezium.embedded.ClickHouseDebeziumEmbeddedPostgresPgoutputDockerIT.testPgOutputPlugin(ClickHouseDebeziumEmbeddedPostgresPgoutputDockerIT.java:109)
while(chRs.next()) {
tmCount = chRs.getInt(1);
}
Expand Down
32 changes: 31 additions & 1 deletion ...nnector-lightweight/src/test/java/com/altinity/clickhouse/debezium/embedded/ITCommon.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,14 +2,17 @@

import com.altinity.clickhouse.debezium.embedded.common.PropertiesHelper;
import com.altinity.clickhouse.debezium.embedded.config.ConfigLoader;
import com.altinity.clickhouse.sink.connector.ClickHouseSinkConnectorConfig;
import com.altinity.clickhouse.sink.connector.db.BaseDbWriter;

import org.testcontainers.clickhouse.ClickHouseContainer;
import org.testcontainers.containers.*;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.SQLException;
import java.util.Properties;

import java.util.HashMap;
public class ITCommon {
static public Connection connectToMySQL(MySQLContainer mySqlContainer) {
Connection conn = null;
Expand Down Expand Up @@ -196,4 +199,31 @@ static public Properties getDebeziumPropertiesForSchemaOnly(MySQLContainer mySql
props.setProperty("replica.status.view", "CREATE VIEW IF NOT EXISTS %s.show_replica_status AS SELECT now() - fromUnixTimestamp(JSONExtractUInt(offset_val, 'ts_sec')) AS seconds_behind_source, toDateTime(fromUnixTimestamp(JSONExtractUInt(offset_val, 'ts_sec')), 'UTC') AS utc_time, fromUnixTimestamp(JSONExtractUInt(offset_val, 'ts_sec')) AS local_time FROM %s settings final=1");
return props;
}


static public BaseDbWriter getDBWriter(ClickHouseContainer clickHouseContainer) {

String jdbcUrl = BaseDbWriter.getConnectionString(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(),
"employees");
Connection connection = BaseDbWriter.createConnection(jdbcUrl, BaseDbWriter.DATABASE_CLIENT_NAME, clickHouseContainer.getUsername(),
clickHouseContainer.getPassword(), BaseDbWriter.SYSTEM_DB, new ClickHouseSinkConnectorConfig(new HashMap<>()));

BaseDbWriter writer = new BaseDbWriter(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(),
"employees", clickHouseContainer.getUsername(), clickHouseContainer.getPassword(), null, connection);

return writer;
}

static public BaseDbWriter getDBWriter(ClickHouseContainer clickHouseContainer, String databaseName) {

String jdbcUrl = BaseDbWriter.getConnectionString(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(),
databaseName);
Connection connection = BaseDbWriter.createConnection(jdbcUrl, BaseDbWriter.DATABASE_CLIENT_NAME, clickHouseContainer.getUsername(),
clickHouseContainer.getPassword(), BaseDbWriter.SYSTEM_DB, new ClickHouseSinkConnectorConfig(new HashMap<>()));

BaseDbWriter writer = new BaseDbWriter(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(),
databaseName, clickHouseContainer.getUsername(), clickHouseContainer.getPassword(), null, connection);

return writer;
}
}
10 changes: 1 addition & 9 deletions ...nector-lightweight/src/test/java/com/altinity/clickhouse/debezium/embedded/MariaDBIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -126,15 +126,7 @@ public void testMultipleDatabases() throws Exception {
conn.close();

// Create connection to clickhouse and validate if the tables are replicated.
String jdbcUrl = BaseDbWriter.getConnectionString(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(),
"system");
ClickHouseConnection chConn = BaseDbWriter.createConnection(jdbcUrl, "Client_1",
clickHouseContainer.getUsername(), clickHouseContainer.getPassword(), new ClickHouseSinkConnectorConfig(new HashMap<>()));

BaseDbWriter writer = new BaseDbWriter(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(),
"system", clickHouseContainer.getUsername(), clickHouseContainer.getPassword(), null, chConn);
// query clickhouse connection and get data for test_table1 and test_table2

BaseDbWriter writer = ITCommon.getDBWriter(clickHouseContainer);

ResultSet rs = writer.executeQueryWithResultSet("SELECT * FROM employees.audience");
// Validate the data
Expand Down
6 changes: 1 addition & 5 deletions ...ght/src/test/java/com/altinity/clickhouse/debezium/embedded/MySQLGenerateColumnsTest.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -97,14 +97,10 @@
conn.prepareStatement("insert into contacts(first_name, last_name, email) values('John', 'Doe', 'john.doe@gmail.com')").execute();
Thread.sleep(20000);

String jdbcUrl = BaseDbWriter.getConnectionString(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(), "employees");
ClickHouseConnection connection = BaseDbWriter.createConnection(jdbcUrl, "client_1", clickHouseContainer.getUsername(), clickHouseContainer.getPassword(), new ClickHouseSinkConnectorConfig(new HashMap<>()));

BaseDbWriter writer = new BaseDbWriter(clickHouseContainer.getHost(), clickHouseContainer.getFirstMappedPort(),
"employees", clickHouseContainer.getUsername(), clickHouseContainer.getPassword(), null, connection);
BaseDbWriter writer = ITCommon.getDBWriter(clickHouseContainer);
Map<String, String> columnsToDataTypeMap = writer.getColumnsDataTypesForTable("contacts");

Assert.assertTrue(columnsToDataTypeMap.get("id").equalsIgnoreCase("Int32"));

Check failure on line 103 in sink-connector-lightweight/src/test/java/com/altinity/clickhouse/debezium/embedded/MySQLGenerateColumnsTest.java

View workflow job for this annotation

GitHub Actions / JUnit Test Report

MySQLGenerateColumnsTest.testMySQLGeneratedColumns 

Cannot invoke "String.equalsIgnoreCase(String)" because the return value of "java.util.Map.get(Object)" is null
Raw output 
java.lang.NullPointerException: Cannot invoke "String.equalsIgnoreCase(String)" because the return value of "java.util.Map.get(Object)" is null
	at com.altinity.clickhouse.debezium.embedded.MySQLGenerateColumnsTest.testMySQLGeneratedColumns(MySQLGenerateColumnsTest.java:103)
Assert.assertTrue(columnsToDataTypeMap.get("first_name").equalsIgnoreCase("String"));
Assert.assertTrue(columnsToDataTypeMap.get("last_name").equalsIgnoreCase("String"));
Assert.assertTrue(columnsToDataTypeMap.get("fullname").equalsIgnoreCase("Nullable(String)"));
Expand Down
Loading
Loading