Auto Compaction cntd.

japila-books · Feb 13, 2024 · 08645d0 · 08645d0
1 parent d6f72cd
commit 08645d0
Show file tree

Hide file tree

Showing 4 changed files with 52 additions and 16 deletions.
diff --git a/docs/auto-compaction/AutoCompactBase.md b/docs/auto-compaction/AutoCompactBase.md
@@ -57,7 +57,7 @@ compactIfNecessary(
 
 When [shouldCompact](AutoCompactRequest.md#shouldCompact) is disabled, `compactIfNecessary` returns no [OptimizeMetrics](../commands/optimize/OptimizeMetrics.md).
 
-Otherwise, with [shouldCompact](AutoCompactRequest.md#shouldCompact) turned on, `compactIfNecessary` [performs auto compaction](AutoCompact.md#compact).
+Otherwise, with [shouldCompact](AutoCompactRequest.md#shouldCompact) turned on, `compactIfNecessary` [performs auto compaction](#compact).
 
 ### getAutoCompactType { #getAutoCompactType }
 
@@ -93,3 +93,22 @@ shouldSkipAutoCompact(
 
 1. The given `autoCompactTypeOpt` is empty (`None`)
 1. [isQualifiedForAutoCompact](AutoCompactUtils.md#isQualifiedForAutoCompact) is disabled
+
+### compact { #compact }
+
+```scala
+compact(
+  spark: SparkSession,
+  deltaLog: DeltaLog,
+  catalogTable: Option[CatalogTable],
+  partitionPredicates: Seq[Expression] = Nil,
+  opType: String = OP_TYPE,
+  maxDeletedRowsRatio: Option[Double] = None): Seq[OptimizeMetrics]
+```
+
+`compact` takes the value of the following configuration properties:
+
+* [spark.databricks.delta.autoCompact.maxFileSize](../configuration-properties/index.md#spark.databricks.delta.autoCompact.maxFileSize)
+* [spark.databricks.delta.autoCompact.minFileSize](../configuration-properties/index.md#spark.databricks.delta.autoCompact.minFileSize)
+
+`compact`...FIXME
diff --git a/docs/auto-compaction/AutoCompactUtils.md b/docs/auto-compaction/AutoCompactUtils.md
@@ -81,6 +81,15 @@ isQualifiedForAutoCompact(
 
 * `AutoCompactBase` is requested to [shouldSkipAutoCompact](AutoCompactBase.md#shouldSkipAutoCompact)
 
+### isNonBlindAppendAutoCompactEnabled { #isNonBlindAppendAutoCompactEnabled }
+
+```scala
+isNonBlindAppendAutoCompactEnabled(
+  spark: SparkSession): Boolean
+```
+
+`isNonBlindAppendAutoCompactEnabled` is the value of [spark.databricks.delta.autoCompact.nonBlindAppend.enabled](../configuration-properties/index.md#spark.databricks.delta.autoCompact.nonBlindAppend.enabled) configuration property (in the given `SparkSession`).
+
 ## isModifiedPartitionsOnlyAutoCompactEnabled { #isModifiedPartitionsOnlyAutoCompactEnabled }
 
 ```scala
@@ -90,25 +99,12 @@ isModifiedPartitionsOnlyAutoCompactEnabled(
 
 `isModifiedPartitionsOnlyAutoCompactEnabled` says whether [Auto Compaction](index.md) should run on modified partitions only.
 
+---
+
 `isModifiedPartitionsOnlyAutoCompactEnabled` is the value of [spark.databricks.delta.autoCompact.modifiedPartitionsOnly.enabled](../configuration-properties/index.md#spark.databricks.delta.autoCompact.modifiedPartitionsOnly.enabled) configuration property (in the given `SparkSession`).
 
 ---
 
 `isModifiedPartitionsOnlyAutoCompactEnabled` is used when:
 
 * `AutoCompactUtils` is requested to [choosePartitionsBasedOnMinNumSmallFiles](#choosePartitionsBasedOnMinNumSmallFiles), [isQualifiedForAutoCompact](#isQualifiedForAutoCompact), [reserveTablePartitions](#reserveTablePartitions)
-
-## isNonBlindAppendAutoCompactEnabled { #isNonBlindAppendAutoCompactEnabled }
-
-```scala
-isNonBlindAppendAutoCompactEnabled(
-  spark: SparkSession): Boolean
-```
-
-`isNonBlindAppendAutoCompactEnabled` is the value of [spark.databricks.delta.autoCompact.nonBlindAppend.enabled](../configuration-properties/index.md#spark.databricks.delta.autoCompact.nonBlindAppend.enabled) configuration property (in the given `SparkSession`).
-
----
-
-`isNonBlindAppendAutoCompactEnabled` is used when:
-
-* `AutoCompactUtils` is requested to [isQualifiedForAutoCompact](#isQualifiedForAutoCompact)
diff --git a/docs/configuration-properties/DeltaSQLConf.md b/docs/configuration-properties/DeltaSQLConf.md
@@ -6,6 +6,14 @@
 
 [spark.databricks.delta.autoCompact.enabled](index.md#autoCompact.enabled)
 
+## autoCompact.maxFileSize { #DELTA_AUTO_COMPACT_MAX_FILE_SIZE }
+
+[spark.databricks.delta.autoCompact.maxFileSize](index.md#autoCompact.maxFileSize)
+
+## autoCompact.minFileSize { #DELTA_AUTO_COMPACT_MIN_FILE_SIZE }
+
+[spark.databricks.delta.autoCompact.minFileSize](index.md#autoCompact.minFileSize)
+
 ## autoCompact.modifiedPartitionsOnly.enabled { #DELTA_AUTO_COMPACT_MODIFIED_PARTITIONS_ONLY_ENABLED }
 
 [spark.databricks.delta.autoCompact.modifiedPartitionsOnly.enabled](index.md#autoCompact.modifiedPartitionsOnly.enabled)

diff --git a/docs/configuration-properties/index.md b/docs/configuration-properties/index.md
@@ -55,6 +55,19 @@ Used when:
 
 * `AutoCompactBase` is requested to [compact](../auto-compaction/AutoCompactBase.md#compact)
 
+### <span id="autoCompact.minFileSize"><span id="DELTA_AUTO_COMPACT_MIN_FILE_SIZE"> autoCompact.minFileSize { #spark.databricks.delta.autoCompact.minFileSize }
+
+**spark.databricks.delta.autoCompact.minFileSize**
+
+**(internal)** Files which are smaller than this threshold (in bytes) will be grouped together and rewritten as larger files in [Auto Compaction](../auto-compaction/index.md)
+
+Default: The half of [spark.databricks.delta.autoCompact.maxFileSize](#autoCompact.maxFileSize)
+
+Used when:
+
+* `OptimisticTransactionImpl` is requested to [createAutoCompactStatsCollector](../OptimisticTransactionImpl.md#createAutoCompactStatsCollector)
+* `AutoCompactBase` is requested to [compact](../auto-compaction/AutoCompactBase.md#compact)
+
 ### <span id="DELTA_AUTO_COMPACT_MODIFIED_PARTITIONS_ONLY_ENABLED"><span id="spark.databricks.delta.autoCompact.modifiedPartitionsOnly.enabled"> autoCompact.modifiedPartitionsOnly.enabled { #autoCompact.modifiedPartitionsOnly.enabled }
 
 **(internal)** **spark.databricks.delta.autoCompact.modifiedPartitionsOnly.enabled**