Skip to content

Commit

Permalink
Automatically Enabled Table Features
Browse files Browse the repository at this point in the history
  • Loading branch information
jaceklaskowski committed Nov 25, 2023
1 parent ac2ce58 commit 4d346cb
Show file tree
Hide file tree
Showing 7 changed files with 44 additions and 11 deletions.
2 changes: 1 addition & 1 deletion docs/DeltaConfigs.md
Original file line number Diff line number Diff line change
Expand Up @@ -138,7 +138,7 @@ Used when:

### <span id="ENABLE_DELETION_VECTORS_CREATION"> enableDeletionVectors { #enableDeletionVectors }

**delta.enableDeletionVectors**
<h4>delta.enableDeletionVectors</h4>

Enables [Deletion Vectors](deletion-vectors/index.md)

Expand Down
2 changes: 1 addition & 1 deletion docs/OptimisticTransactionImpl.md
Original file line number Diff line number Diff line change
Expand Up @@ -711,7 +711,7 @@ setNewProtocolWithFeaturesEnabledByMetadata(
metadata: Metadata): Unit
```

`setNewProtocolWithFeaturesEnabledByMetadata`...FIXME
`setNewProtocolWithFeaturesEnabledByMetadata` [upgradeProtocolFromMetadataForExistingTable](Protocol.md#upgradeProtocolFromMetadataForExistingTable) (with the given [Metadata](Metadata.md) and the [Protocol](#protocol)) and makes it this transaction's [protocol](#newProtocol).

---

Expand Down
6 changes: 3 additions & 3 deletions docs/Protocol.md
Original file line number Diff line number Diff line change
Expand Up @@ -217,7 +217,7 @@ minProtocolComponentsFromMetadata(
`minProtocolComponentsFromMetadata` is used when:

* `Protocol` is requested to [forNewTable](#forNewTable)
* `CloneTableBase` is requested to [runInternal](commands/clone/CloneTableBase.md#runInternal)
* `CloneTableBase` is requested to [determineTargetProtocol](commands/clone/CloneTableBase.md#determineTargetProtocol)

## upgradeProtocolFromMetadataForExistingTable { #upgradeProtocolFromMetadataForExistingTable }

Expand All @@ -243,9 +243,9 @@ minProtocolComponentsFromAutomaticallyEnabledFeatures(
metadata: Metadata): (Int, Int, Set[TableFeature])
```

`minProtocolComponentsFromAutomaticallyEnabledFeatures`...FIXME
`minProtocolComponentsFromAutomaticallyEnabledFeatures` determines the minimum reader and writer versions based on [automatically enabled table features](#extractAutomaticallyEnabledFeatures).

## <span id="demo"> Demo
## Demo

```scala
import org.apache.spark.sql.delta.actions.{Metadata, Protocol}
Expand Down
10 changes: 7 additions & 3 deletions docs/Snapshot.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@
* <span id="timestamp"> Timestamp
* <span id="checksumOpt"> `VersionChecksum`

While being created, `Snapshot` prints out the following INFO message to the logs and [initialize](#init):
While being created, `Snapshot` prints out the following INFO message to the logs and triggers [initialization](#init).

```text
Created snapshot [this]
Expand All @@ -28,13 +28,17 @@ Created snapshot [this]

* `SnapshotManagement` is requested for a [Snapshot](SnapshotManagement.md#createSnapshot)

### <span id="init"> Initializing
### Initializing { #init }

```scala
init(): Unit
```

`init` requests the [DeltaLog](#deltaLog) for the [protocolRead](DeltaLog.md#protocolRead) for the [Protocol](#protocol).
`init` validates the [protocol](#protocol) and the [metadata](#metadata):

1. Requests the [DeltaLog](#deltaLog) to assert that the [Protocol](#protocol) is [protocolRead](DeltaLog.md#protocolRead)
1. Requests the [DeltaLog](#deltaLog) to [assertTableFeaturesMatchMetadata](DeltaLog.md#assertTableFeaturesMatchMetadata) (with the [Protocol](#protocol) and the [Metadata](#metadata))
1. [Records undefined types (in the Delta protocol)](SchemaUtils.md#recordUndefinedTypes) in the [schema](Metadata.md#schema) (of the [Metadata](#metadata))

## <span id="numIndexedCols"> Maximum Number of Indexed Columns

Expand Down
28 changes: 27 additions & 1 deletion docs/commands/clone/CloneConvertedSource.md
Original file line number Diff line number Diff line change
@@ -1,3 +1,29 @@
# CloneConvertedSource

`CloneConvertedSource` is...FIXME
`CloneConvertedSource` is an [extension](#contract) of the [CloneSource](CloneSource.md) abstraction for [convertible non-delta table sources](#implementations) to clone from.

## Contract

### convertTargetTable { #convertTargetTable }

```scala
convertTargetTable: ConvertTargetTable
```

Used when:

* `CloneConvertedSource` is requested for the [data path](#dataPath), [schema](#schema), [metadata](#metadata), [allFiles](#allFiles), [close](#close)

## Implementations

* `CloneIcebergSource`
* [CloneParquetSource](CloneParquetSource.md)

## Creating Instance

`CloneConvertedSource` takes the following to be created:

* <span id="spark"> `SparkSession`

!!! note "Abstract Class"
`CloneConvertedSource` is an abstract class and cannot be created directly. It is created indirectly for the [concrete CloneConvertedSources](#implementations).
2 changes: 1 addition & 1 deletion docs/deletion-vectors/DeletionVectorsTableFeature.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,4 +30,4 @@ With [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) tab

`metadataRequiresFeatureToBeEnabled` is part of the [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md#metadataRequiresFeatureToBeEnabled) abstraction.

`metadataRequiresFeatureToBeEnabled` is the value of [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) table property (from the [Metadata](../DeltaConfig.md#fromMetaData)).
`metadataRequiresFeatureToBeEnabled` is the value of [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) table property (in the [Metadata](../DeltaConfig.md#fromMetaData)).
5 changes: 4 additions & 1 deletion docs/table-features/FeatureAutomaticallyEnabledByMetadata.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,14 @@ metadataRequiresFeatureToBeEnabled(
spark: SparkSession): Boolean
```

`metadataRequiresFeatureToBeEnabled` is enabled (`true`) for automatically enabled features (based on [metadata](../Metadata.md) configuration)
Controls whether this [TableFeature](TableFeature.md) should be supported and enabled because its metadata requirements are satisfied

Enabled (`true`) for automatically enabled features (based on [metadata](../Metadata.md) configuration)

See:

* [AppendOnlyTableFeature](../append-only-tables/AppendOnlyTableFeature.md#metadataRequiresFeatureToBeEnabled)
* [DeletionVectorsTableFeature](../deletion-vectors/DeletionVectorsTableFeature.md#metadataRequiresFeatureToBeEnabled)
* [RowTrackingFeature](../row-tracking/RowTrackingFeature.md#metadataRequiresFeatureToBeEnabled)

Used when:
Expand Down

0 comments on commit 4d346cb

Please sign in to comment.