Skip to content

Commit

Permalink
Auto Compaction
Browse files Browse the repository at this point in the history
  • Loading branch information
jaceklaskowski committed Feb 12, 2024
1 parent a508180 commit d6f72cd
Show file tree
Hide file tree
Showing 2 changed files with 49 additions and 6 deletions.
15 changes: 10 additions & 5 deletions docs/auto-compaction/AutoCompactBase.md
Original file line number Diff line number Diff line change
Expand Up @@ -67,13 +67,18 @@ getAutoCompactType(
metadata: Metadata): Option[AutoCompactType]
```

`getAutoCompactType` is the value of the following (in the order of precedence):
??? note "Return Type"
`Option[AutoCompactType]` is the return type but it's a _fancy_ way to say "enabled" or "not".

1. [spark.databricks.delta.autoCompact.enabled](../configuration-properties/DeltaSQLConf.md#autoCompact.enabled), if configured.
1. [delta.autoOptimize](../table-properties/DeltaConfigs.md#AUTO_OPTIMIZE) table property
When `getAutoCompactType` returns `Some[AutoCompactType]` it means "enabled" while `None` is "disabled".

`getAutoCompactType` is enabled when either is `true` (in the order of precedence):

1. [spark.databricks.delta.autoCompact.enabled](../configuration-properties/index.md#autoCompact.enabled)
1. (deprecated) [delta.autoOptimize](../table-properties/DeltaConfigs.md#AUTO_OPTIMIZE) table property
1. [delta.autoOptimize.autoCompact](../table-properties/DeltaConfigs.md#AUTO_COMPACT) table property

`getAutoCompactType` defaults to `false`.
`getAutoCompactType` defaults to `false` (disabled).

### shouldSkipAutoCompact { #shouldSkipAutoCompact }

Expand All @@ -86,5 +91,5 @@ shouldSkipAutoCompact(

`shouldSkipAutoCompact` is enabled (`true`) for the following:

1. The given `AutoCompactType` is empty (`None`)
1. The given `autoCompactTypeOpt` is empty (`None`)
1. [isQualifiedForAutoCompact](AutoCompactUtils.md#isQualifiedForAutoCompact) is disabled
40 changes: 39 additions & 1 deletion docs/auto-compaction/AutoCompactUtils.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,14 +12,52 @@ prepareAutoCompactRequest(
maxDeletedRowsRatio: Option[Double]): AutoCompactRequest
```

`prepareAutoCompactRequest`...FIXME
`prepareAutoCompactRequest` [reserveTablePartitions](#reserveTablePartitions).

In the end, `prepareAutoCompactRequest` [createPartitionPredicate](#createPartitionPredicate) and creates a [AutoCompactRequest](AutoCompactRequest.md).

---

`prepareAutoCompactRequest` is used when:

* `AutoCompactBase` is requested to [compactIfNecessary](AutoCompactBase.md#compactIfNecessary)

### createPartitionPredicate { #createPartitionPredicate }

```scala
createPartitionPredicate(
postCommitSnapshot: Snapshot,
partitions: PartitionKeySet): Seq[Expression]
```

`createPartitionPredicate`...FIXME

### reserveTablePartitions { #reserveTablePartitions }

```scala
reserveTablePartitions(
spark: SparkSession,
deltaLog: DeltaLog,
postCommitSnapshot: Snapshot,
partitionsAddedToOpt: Option[PartitionKeySet],
opType: String,
maxDeletedRowsRatio: Option[Double]): (Boolean, PartitionKeySet)
```

`reserveTablePartitions`...FIXME

### choosePartitionsBasedOnMinNumSmallFiles { #choosePartitionsBasedOnMinNumSmallFiles }

```scala
choosePartitionsBasedOnMinNumSmallFiles(
spark: SparkSession,
deltaLog: DeltaLog,
postCommitSnapshot: Snapshot,
freePartitionsAddedTo: PartitionKeySet): ChosenPartitionsResult
```

`choosePartitionsBasedOnMinNumSmallFiles`...FIXME

## isQualifiedForAutoCompact { #isQualifiedForAutoCompact }

```scala
Expand Down

0 comments on commit d6f72cd

Please sign in to comment.