diff --git a/docs/DeltaConfigs.md b/docs/DeltaConfigs.md index bfc6edd05d..54a99524b8 100644 --- a/docs/DeltaConfigs.md +++ b/docs/DeltaConfigs.md @@ -138,7 +138,7 @@ Used when: ### enableDeletionVectors { #enableDeletionVectors } -**delta.enableDeletionVectors** +

delta.enableDeletionVectors

Enables [Deletion Vectors](deletion-vectors/index.md) diff --git a/docs/OptimisticTransactionImpl.md b/docs/OptimisticTransactionImpl.md index 2de0d6df96..6f1487dd45 100644 --- a/docs/OptimisticTransactionImpl.md +++ b/docs/OptimisticTransactionImpl.md @@ -711,7 +711,7 @@ setNewProtocolWithFeaturesEnabledByMetadata( metadata: Metadata): Unit ``` -`setNewProtocolWithFeaturesEnabledByMetadata`...FIXME +`setNewProtocolWithFeaturesEnabledByMetadata` [upgradeProtocolFromMetadataForExistingTable](Protocol.md#upgradeProtocolFromMetadataForExistingTable) (with the given [Metadata](Metadata.md) and the [Protocol](#protocol)) and makes it this transaction's [protocol](#newProtocol). --- diff --git a/docs/Protocol.md b/docs/Protocol.md index e04018d3fc..c5b60ac3f7 100644 --- a/docs/Protocol.md +++ b/docs/Protocol.md @@ -217,7 +217,7 @@ minProtocolComponentsFromMetadata( `minProtocolComponentsFromMetadata` is used when: * `Protocol` is requested to [forNewTable](#forNewTable) -* `CloneTableBase` is requested to [runInternal](commands/clone/CloneTableBase.md#runInternal) +* `CloneTableBase` is requested to [determineTargetProtocol](commands/clone/CloneTableBase.md#determineTargetProtocol) ## upgradeProtocolFromMetadataForExistingTable { #upgradeProtocolFromMetadataForExistingTable } @@ -243,9 +243,9 @@ minProtocolComponentsFromAutomaticallyEnabledFeatures( metadata: Metadata): (Int, Int, Set[TableFeature]) ``` -`minProtocolComponentsFromAutomaticallyEnabledFeatures`...FIXME +`minProtocolComponentsFromAutomaticallyEnabledFeatures` determines the minimum reader and writer versions based on [automatically enabled table features](#extractAutomaticallyEnabledFeatures). -## Demo +## Demo ```scala import org.apache.spark.sql.delta.actions.{Metadata, Protocol} diff --git a/docs/Snapshot.md b/docs/Snapshot.md index 447e9f45ab..4d7430ef6f 100644 --- a/docs/Snapshot.md +++ b/docs/Snapshot.md @@ -18,7 +18,7 @@ * Timestamp * `VersionChecksum` -While being created, `Snapshot` prints out the following INFO message to the logs and [initialize](#init): +While being created, `Snapshot` prints out the following INFO message to the logs and triggers [initialization](#init). ```text Created snapshot [this] @@ -28,13 +28,17 @@ Created snapshot [this] * `SnapshotManagement` is requested for a [Snapshot](SnapshotManagement.md#createSnapshot) -### Initializing +### Initializing { #init } ```scala init(): Unit ``` -`init` requests the [DeltaLog](#deltaLog) for the [protocolRead](DeltaLog.md#protocolRead) for the [Protocol](#protocol). +`init` validates the [protocol](#protocol) and the [metadata](#metadata): + +1. Requests the [DeltaLog](#deltaLog) to assert that the [Protocol](#protocol) is [protocolRead](DeltaLog.md#protocolRead) +1. Requests the [DeltaLog](#deltaLog) to [assertTableFeaturesMatchMetadata](DeltaLog.md#assertTableFeaturesMatchMetadata) (with the [Protocol](#protocol) and the [Metadata](#metadata)) +1. [Records undefined types (in the Delta protocol)](SchemaUtils.md#recordUndefinedTypes) in the [schema](Metadata.md#schema) (of the [Metadata](#metadata)) ## Maximum Number of Indexed Columns diff --git a/docs/commands/clone/CloneConvertedSource.md b/docs/commands/clone/CloneConvertedSource.md index 1b5f0dd821..fe08057ebe 100644 --- a/docs/commands/clone/CloneConvertedSource.md +++ b/docs/commands/clone/CloneConvertedSource.md @@ -1,3 +1,29 @@ # CloneConvertedSource -`CloneConvertedSource` is...FIXME +`CloneConvertedSource` is an [extension](#contract) of the [CloneSource](CloneSource.md) abstraction for [convertible non-delta table sources](#implementations) to clone from. + +## Contract + +### convertTargetTable { #convertTargetTable } + +```scala +convertTargetTable: ConvertTargetTable +``` + +Used when: + +* `CloneConvertedSource` is requested for the [data path](#dataPath), [schema](#schema), [metadata](#metadata), [allFiles](#allFiles), [close](#close) + +## Implementations + +* `CloneIcebergSource` +* [CloneParquetSource](CloneParquetSource.md) + +## Creating Instance + +`CloneConvertedSource` takes the following to be created: + +* `SparkSession` + +!!! note "Abstract Class" + `CloneConvertedSource` is an abstract class and cannot be created directly. It is created indirectly for the [concrete CloneConvertedSources](#implementations). diff --git a/docs/deletion-vectors/DeletionVectorsTableFeature.md b/docs/deletion-vectors/DeletionVectorsTableFeature.md index c835873d0a..8f150e0892 100644 --- a/docs/deletion-vectors/DeletionVectorsTableFeature.md +++ b/docs/deletion-vectors/DeletionVectorsTableFeature.md @@ -30,4 +30,4 @@ With [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) tab `metadataRequiresFeatureToBeEnabled` is part of the [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md#metadataRequiresFeatureToBeEnabled) abstraction. -`metadataRequiresFeatureToBeEnabled` is the value of [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) table property (from the [Metadata](../DeltaConfig.md#fromMetaData)). +`metadataRequiresFeatureToBeEnabled` is the value of [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) table property (in the [Metadata](../DeltaConfig.md#fromMetaData)). diff --git a/docs/table-features/FeatureAutomaticallyEnabledByMetadata.md b/docs/table-features/FeatureAutomaticallyEnabledByMetadata.md index c69147265d..675ebd3708 100644 --- a/docs/table-features/FeatureAutomaticallyEnabledByMetadata.md +++ b/docs/table-features/FeatureAutomaticallyEnabledByMetadata.md @@ -12,11 +12,14 @@ metadataRequiresFeatureToBeEnabled( spark: SparkSession): Boolean ``` -`metadataRequiresFeatureToBeEnabled` is enabled (`true`) for automatically enabled features (based on [metadata](../Metadata.md) configuration) +Controls whether this [TableFeature](TableFeature.md) should be supported and enabled because its metadata requirements are satisfied + +Enabled (`true`) for automatically enabled features (based on [metadata](../Metadata.md) configuration) See: * [AppendOnlyTableFeature](../append-only-tables/AppendOnlyTableFeature.md#metadataRequiresFeatureToBeEnabled) +* [DeletionVectorsTableFeature](../deletion-vectors/DeletionVectorsTableFeature.md#metadataRequiresFeatureToBeEnabled) * [RowTrackingFeature](../row-tracking/RowTrackingFeature.md#metadataRequiresFeatureToBeEnabled) Used when: