Add scylladb module #8002

mkorolyov · 2023-12-26T10:32:34Z

ScyllaDB is an open-source distributed NoSQL wide-column data store. It gains wide popularity as a drop-in replacement for Cassandra but a significantly faster one.

This PR adds the ScyllaDB module.

guy9 · 2024-01-07T11:12:43Z

Thanks @mkorolyov , interested in this as well

docs/modules/databases/scylladb.md

modules/scylladb/build.gradle

mykaul · 2024-01-14T09:32:03Z

I wish there was a way NOT to provide your own scylla.yaml, but merge your defaults with the default scylla.yaml - the end result is that you might be missing some new defaults if you bring your own scylla.yaml.

fix typos in docs Co-authored-by: Yaniv Kaul <yaniv.kaul@scylladb.com>

mkorolyov · 2024-01-16T09:34:29Z

I wish there was a way NOT to provide your own scylla.yaml, but merge your defaults with the default scylla.yaml - the end result is that you might be missing some new defaults if you bring your own scylla.yaml.

This looks like a separate feature for merging config files. Usually what I used to see in a majority of the applications is that you are using the default config file and overriding it with ENVs or flags or providing your own config file. Providing own config file which will still be merged with some default looks like implicit behavior which in my opinion will confuse users. WDYT?

guy9 · 2024-01-21T14:02:11Z

Any updates on merging this?

mykaul · 2024-01-23T16:26:44Z

I wish there was a way NOT to provide your own scylla.yaml, but merge your defaults with the default scylla.yaml - the end result is that you might be missing some new defaults if you bring your own scylla.yaml.

This looks like a separate feature for merging config files. Usually what I used to see in a majority of the applications is that you are using the default config file and overriding it with ENVs or flags or providing your own config file. Providing own config file which will still be merged with some default looks like implicit behavior which in my opinion will confuse users. WDYT?

Yes, my ask is exactly this - let's use the default Scylla YAML that comes with Scylla, and allow overriding values in it.

fruch · 2024-01-24T13:55:57Z

modules/scylladb/src/main/java/org/testcontainers/containers/ScyllaDBContainer.java

+        this.enableJmxReporting = false;
+
+        withEnv("CASSANDRA_SNITCH", "GossipingPropertyFileSnitch");
+        withEnv("JVM_OPTS", "-Dcassandra.skip_wait_for_gossip_to_settle=0 -Dcassandra.initial_token=0");


those are not relevant to scylla, you need commandline option for skip_wait_for_gossip_to_settle

fruch · 2024-01-24T13:56:32Z

modules/scylladb/src/main/java/org/testcontainers/containers/ScyllaDBContainer.java

+        addExposedPort(CQL_PORT);
+        this.enableJmxReporting = false;
+
+        withEnv("CASSANDRA_SNITCH", "GossipingPropertyFileSnitch");


I think this needs to be checked, scylla docker doesn't use this variable

vbazhmin · 2024-06-07T15:59:36Z

Hello folks. Any updates on this one ? ;)

eddumelendez · 2024-08-01T02:17:08Z

Hi, I'll review the PR once the other comments have been addressed. It is great seeing Scylla team reviewing the PR.

fruch · 2024-08-01T21:34:22Z

Hi, I'll review the PR once the other comments have been addressed. It is great seeing Scylla team reviewing the PR.

hopefully it won't be dragged like it's counterpart:
testcontainers/testcontainers-python#167

eddumelendez · 2024-08-14T01:26:23Z

@fruch LMK once this is ready to review and will do it as soon as possible. I still see some comments that hasn't been resolved. So, that's the reason why I haven't reviewed yet.

eddumelendez · 2024-10-16T03:08:41Z

Hi, I know this is still in progress because I see some comments unresolved. But, JFTR it would be nice to add an integration test enabling ssl and add it to the docs.

eddumelendez · 2024-12-17T23:28:02Z

I'm updating the PR based on testcontainers/testcontainers-go#2919

mkorolyov · 2025-01-03T18:36:14Z

Hi @mykaul,

I've looked into implementing your suggestion, and it seems there are a few challenges with achieving this functionality:

ScyllaDB Configuration Dependency:
The ScyllaDB service requires a single configuration file, typically located at /etc/scylla/scylla.yaml in the official Docker image. It's straightforward to replace this file in the container using a volume mount, e.g., -v ~/master_scylla.yaml:/etc/scylla/scylla.yaml.
Merging Custom Configuration:
To merge a custom partial configuration with the default one, we would need to process and generate the final configuration file before the container starts and then mount it to the container.
Extracting Default Configuration:
One potential approach is to extract the default configuration file directly from the Docker image without starting the container. This can be done using Docker commands, such as:
```
docker create --name temp-container scylladb/scylla  
docker cp temp-container:/etc/scylla/scylla.yaml ./scylla.yaml  
docker rm temp-container  
```

While this method works, it feels like a bit of a hack to me, and I’m not entirely comfortable with introducing such a workaround into the workflow.

Maintaining Up-to-Date Configurations:
Alternatively, the latest default ScyllaDB configuration file can be found in their repository here. However, storing a snapshot of this file in the testcontainers-java repository introduces a maintenance issue: it would become outdated as soon as updates are made upstream.
Dynamic Fetching Concerns:
Dynamically fetching the latest configuration file from the ScyllaDB repository during container setup (e.g., via HTTP calls) is also problematic and may not align with project standards or be acceptable to the maintainers.

Given these constraints, do you have other suggestions for achieving this functionality? Specifically, how can we ensure the availability of an up-to-date configuration file while maintaining simplicity and compatibility with the existing workflow?

Looking forward to your thoughts!

mkorolyov · 2025-01-03T19:01:42Z

@eddumelendez Hey, thanks for stepping in and polishing PR. Just asking, why you removed custom config file related changes?

mykaul · 2025-01-05T07:35:04Z

@mkorolyov - thanks for looking into this. I think your point no. 1 makes a convincing argument - it's trivial to just pass a yaml file, it's quite standard, so anyone using the Scylla testcontainer can do it. Let's just document it.

mkorolyov · 2025-01-05T21:00:17Z

@mykaul Hey, added possibility to easily replace config file for test container and documented it.

@eddumelendez Hey, added ssl integration test and documented it.

Could you pls take a look so we could proceed with it?

Appreciate your help and time!

eddumelendez · 2025-01-06T15:10:02Z

modules/scylladb/src/test/java/org/testcontainers/scylladb/ScyllaDBContainerTest.java

+    }
+
+    @Test
+    public void testSslConfiguration() {


@mkorolyov this test fails. Lacking ssl configuration? default image doesn't provide those env vars

it was committed by mistake. cleaned that up, thanks!

eddumelendez · 2025-01-06T15:46:36Z

modules/scylladb/src/test/java/org/testcontainers/scylladb/ScyllaDBContainerTest.java

+    }
+
+    @Test
+    public void testSimpleSsl()


Thanks for adding this! I think cql cli command would not work unless is configured properly. We should configure it properly when ssl is enabled https://opensource.docs.scylladb.com/stable/operating-scylla/security/gen-cqlsh-file.html

@eddumelendez yes, if ssl enabled on the server for client connections in order to cqlsh to work we need to configure ssl for it too, but it is not used here, in test containers, right? I didn't get your point, could you pls clarify what else you would like to see in this PR? thanks!

Users can execute commands using via execInContainer method provided by Testcontainers API. Testcontainers should provide a proper configuration for the container to run properly. In this case, external clients will connect properly because the sdk client is configured but not inside the container. I think we should fix that.

I've added a test example how to make cqlsh work with same ssl setup.

Thanks! WDYT about adding withSsl(MountableFile certificate, MountableFile keyfile, MountableFile truststore) and copy those files as part of ScyllaDBContainer. If so, then create the cqlshrc.

ScyllaDBContainer scylladb = new ScyllaDBContainer(SCYLLADB_IMAGE) .withConfigurationOverride("scylla-test-ssl") .withSsl(certificate, keyfile, truststore)

@eddumelendez Sure, added withSsl. Also used SSL_CERTFILE env variable instead of rewriting cqlshrc file as there could be issues with the connection hostname specified in the file and also it simplifies setup.

mkorolyov · 2025-01-23T17:15:19Z

@eddumelendez Hey, could you please take a look? I've updated PR according to your suggestions.

eddumelendez

@mkorolyov we are close. I left one more comment related to the truststore given it is optional.

eddumelendez · 2025-01-30T20:39:44Z

modules/scylladb/src/main/java/org/testcontainers/scylladb/ScyllaDBContainer.java

+        return this;
+    }
+
+    public ScyllaDBContainer withSsl(MountableFile certificate, MountableFile keyfile, MountableFile truststore) {


truststore is optional. See https://opensource.docs.scylladb.com/stable/operating-scylla/security/client-node-encryption.html#validate-the-clients. So, we should provide another method
withSsl(MountableFile certificate, MountableFile keyfile)

nice catch! thanks, fixed this one

I have reverted this because truststore must be at the system truststore and in order to do that we need to copy the file too. Sorry about that.

modules/scylladb/src/test/java/org/testcontainers/scylladb/ScyllaDBContainerTest.java

eddumelendez · 2025-01-31T20:13:01Z

Thanks for your contribution, @mkorolyov !

mkorolyov requested a review from a team as a code owner December 26, 2023 10:32

github-actions bot added the type/docs label Dec 26, 2023

mkorolyov force-pushed the add-scylladb branch from ca6c8ff to ff3b42c Compare January 2, 2024 07:23

add scylladb module

86ed2cb

mkorolyov force-pushed the add-scylladb branch from ff3b42c to 86ed2cb Compare January 2, 2024 07:43

mykaul reviewed Jan 14, 2024

View reviewed changes

docs/modules/databases/scylladb.md Outdated Show resolved Hide resolved

mykaul reviewed Jan 14, 2024

View reviewed changes

modules/scylladb/build.gradle Outdated Show resolved Hide resolved

mkorolyov and others added 2 commits January 16, 2024 09:25

Update docs/modules/databases/scylladb.md

bc7eb8b

fix typos in docs Co-authored-by: Yaniv Kaul <yaniv.kaul@scylladb.com>

remove com.datastax.cassandra:cassandra-driver-core from deps

35ea1c2

Merge branch 'main' into add-scylladb

8590397

fruch reviewed Jan 24, 2024

View reviewed changes

eddumelendez added the type/new module label Feb 8, 2024

eddumelendez added 2 commits December 17, 2024 11:10

Merge remote-tracking branch 'origin/main' into add-scylladb

f10e620

Polish

1805c16

Add docs

b5fa599

add custom config file. add ssl test and document it

af00b41

Fix format

7badf3d

eddumelendez reviewed Jan 6, 2025

View reviewed changes

mkorolyov added 2 commits January 7, 2025 16:37

remove temporary ssl test

60f7dd4

add ssl cqlsh test and docs

ccd7e41

mkorolyov requested a review from eddumelendez January 14, 2025 07:37

mkorolyov added 2 commits January 14, 2025 08:38

Merge branch 'main' into add-scylladb

8ac1c15

Merge branch 'main' into add-scylladb

7a356f2

add withSsl method to ScyllaDBContainer

181472e

eddumelendez reviewed Jan 30, 2025

View reviewed changes

modules/scylladb/src/test/java/org/testcontainers/scylladb/ScyllaDBContainerTest.java Outdated Show resolved Hide resolved

mkorolyov and others added 3 commits January 31, 2025 00:03

add additional withSsl(certificate, keyfile)

19ae30a

Merge branch 'main' into add-scylladb

3b95f64

Polish

6ab9c7e

eddumelendez changed the title ~~Add scylladb~~ Add scylladb module Jan 31, 2025

eddumelendez added this to the next milestone Jan 31, 2025

eddumelendez merged commit 46fe670 into testcontainers:main Jan 31, 2025
108 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add scylladb module #8002

Add scylladb module #8002

mkorolyov commented Dec 26, 2023

guy9 commented Jan 7, 2024

mykaul commented Jan 14, 2024

mkorolyov commented Jan 16, 2024

guy9 commented Jan 21, 2024

mykaul commented Jan 23, 2024

fruch Jan 24, 2024

fruch Jan 24, 2024

vbazhmin commented Jun 7, 2024

eddumelendez commented Aug 1, 2024

fruch commented Aug 1, 2024

eddumelendez commented Aug 14, 2024

eddumelendez commented Oct 16, 2024

eddumelendez commented Dec 17, 2024

mkorolyov commented Jan 3, 2025

mkorolyov commented Jan 3, 2025

mykaul commented Jan 5, 2025

mkorolyov commented Jan 5, 2025

eddumelendez Jan 6, 2025

mkorolyov Jan 7, 2025

eddumelendez Jan 6, 2025

mkorolyov Jan 7, 2025

eddumelendez Jan 7, 2025

mkorolyov Jan 9, 2025

eddumelendez Jan 24, 2025

mkorolyov Jan 26, 2025

mkorolyov commented Jan 23, 2025

eddumelendez left a comment

eddumelendez Jan 30, 2025

mkorolyov Jan 30, 2025

eddumelendez Jan 31, 2025

eddumelendez commented Jan 31, 2025

Add scylladb module #8002

Add scylladb module #8002

Conversation

mkorolyov commented Dec 26, 2023

guy9 commented Jan 7, 2024

mykaul commented Jan 14, 2024

mkorolyov commented Jan 16, 2024

guy9 commented Jan 21, 2024

mykaul commented Jan 23, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

vbazhmin commented Jun 7, 2024

eddumelendez commented Aug 1, 2024

fruch commented Aug 1, 2024

eddumelendez commented Aug 14, 2024

eddumelendez commented Oct 16, 2024

eddumelendez commented Dec 17, 2024

mkorolyov commented Jan 3, 2025

mkorolyov commented Jan 3, 2025

mykaul commented Jan 5, 2025

mkorolyov commented Jan 5, 2025

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mkorolyov commented Jan 23, 2025

eddumelendez left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

eddumelendez commented Jan 31, 2025