From dc2c844c4473a7561a65e66f93b770b9176b534e Mon Sep 17 00:00:00 2001 From: Kanthi Date: Tue, 23 Jul 2024 16:33:55 -0400 Subject: [PATCH] Snapshot out of memory documentation Snapshot out of memory documentation --- doc/production_setup.md | 16 +++++++++++++++- 1 file changed, 15 insertions(+), 1 deletion(-) diff --git a/doc/production_setup.md b/doc/production_setup.md index 0c799476c..2b739b263 100644 --- a/doc/production_setup.md +++ b/doc/production_setup.md @@ -41,4 +41,18 @@ in terms of number of elements the queue can hold and the maximum size of the qu Flush time of the buffer in milliseconds. The buffer that is stored in memory before being flushed to ClickHouse. buffer.flush.time.ms: "1000" -``` \ No newline at end of file +``` + +## Snapshots (Out of Memory) + +1. The following parameters might be useful to reduce the memory usage of the connector during the snapshotting phase. + + +**Reduce the number of tables**: Use the `snapshot.include.collection.list` to filter include only the necessary tables +as part of the snapshot which will reduce the data retrieved from source. + +**snapshot.fetch.size**: This defines the maximum number of rows that are fetched as part of the batch from the database. +Use a lower number to reduce memory usage. + +**snapshot.max.threads**: Increase this number from 1 to a higher value to enable parallel snapshotting. +