diff --git a/docs/DeltaErrorsBase.md b/docs/DeltaErrorsBase.md index 936d7f8f28..664e2fc9fb 100644 --- a/docs/DeltaErrorsBase.md +++ b/docs/DeltaErrorsBase.md @@ -12,7 +12,7 @@ convertToDeltaRowTrackingEnabledWithoutStatsCollection: Throwable * `messageParameters`: * [spark.databricks.delta.stats.collect](configuration-properties/index.md#DELTA_COLLECT_STATS) * The [default session config key](table-features/TableFeatureProtocolUtils.md#defaultPropertyKey) of [RowTrackingFeature](row-tracking/RowTrackingFeature.md) - * The [default table property key](DeltaConfig.md#defaultTablePropertyKey) of [delta.enableRowTracking](DeltaConfigs.md#ROW_TRACKING_ENABLED) + * The [default table property key](table-properties/DeltaConfig.md#defaultTablePropertyKey) of [delta.enableRowTracking](table-properties/DeltaConfigs.md#ROW_TRACKING_ENABLED) --- @@ -32,7 +32,7 @@ modifyAppendOnlyTableException( * `errorClass`: `DELTA_CANNOT_MODIFY_APPEND_ONLY` * `messageParameters`: * The given `tableName` - * [delta.appendOnly](DeltaConfigs.md#IS_APPEND_ONLY) + * [delta.appendOnly](table-properties/DeltaConfigs.md#IS_APPEND_ONLY) --- diff --git a/docs/DeltaLog.md b/docs/DeltaLog.md index 7bc7bacf5b..dae8f2b02c 100644 --- a/docs/DeltaLog.md +++ b/docs/DeltaLog.md @@ -380,7 +380,7 @@ getSnapshotAt( checkpointInterval: Int ``` -`checkpointInterval` is the current value of [checkpointInterval](DeltaConfigs.md#CHECKPOINT_INTERVAL) table property ([from](DeltaConfigs.md#fromMetaData) the [Metadata](#metadata)). +`checkpointInterval` is the current value of [checkpointInterval](table-properties/DeltaConfigs.md#CHECKPOINT_INTERVAL) table property ([from](table-properties/DeltaConfigs.md#fromMetaData) the [Metadata](#metadata)). `checkpointInterval` is used when: @@ -573,7 +573,7 @@ minFileRetentionTimestamp: Long tombstoneRetentionMillis: Long ``` -`tombstoneRetentionMillis` gives the value of [deletedFileRetentionDuration](DeltaConfigs.md#TOMBSTONE_RETENTION) table property ([from](DeltaConfigs.md#fromMetaData) the [Metadata](#metadata)). +`tombstoneRetentionMillis` gives the value of [deletedFileRetentionDuration](table-properties/DeltaConfigs.md#TOMBSTONE_RETENTION) table property ([from](table-properties/DeltaConfigs.md#fromMetaData) the [Metadata](#metadata)). `tombstoneRetentionMillis` is used when: @@ -707,7 +707,7 @@ assertRemovable(): Unit ??? warning "Procedure" `assertRemovable` is a procedure (returns `Unit`) so _what happens inside stays inside_ (paraphrasing the [former advertising slogan of Las Vegas, Nevada](https://idioms.thefreedictionary.com/what+happens+in+Vegas+stays+in+Vegas)). -With [delta.appendOnly](DeltaConfigs.md#IS_APPEND_ONLY) table property enabled, `assertRemovable` throws a [DeltaUnsupportedOperationException](DeltaErrors.md#modifyAppendOnlyTableException). +With [delta.appendOnly](table-properties/DeltaConfigs.md#IS_APPEND_ONLY) table property enabled, `assertRemovable` throws a [DeltaUnsupportedOperationException](DeltaErrors.md#modifyAppendOnlyTableException). --- diff --git a/docs/FileAction.md b/docs/FileAction.md index bcf8bd3b23..ef708d0603 100644 --- a/docs/FileAction.md +++ b/docs/FileAction.md @@ -17,7 +17,7 @@ Isolation Level | Description [SnapshotIsolation](IsolationLevel.md#SnapshotIsolation) | No data changes (`dataChange` is `false` for all `FileAction`s to be committed) [Serializable](IsolationLevel.md#Serializable) |   -There can be no [RemoveFile](RemoveFile.md)s with `dataChange` enabled for [appendOnly](DeltaConfigs.md#appendOnly) unmodifiable tables (or an [UnsupportedOperationException is thrown](DeltaLog.md#assertRemovable)). +There can be no [RemoveFile](RemoveFile.md)s with `dataChange` enabled for [appendOnly](table-properties/DeltaConfigs.md#appendOnly) unmodifiable tables (or an [UnsupportedOperationException is thrown](DeltaLog.md#assertRemovable)). dataChange Value | When -----------------|--------- diff --git a/docs/InitialSnapshot.md b/docs/InitialSnapshot.md index c30ed8d413..087c2a60f6 100644 --- a/docs/InitialSnapshot.md +++ b/docs/InitialSnapshot.md @@ -33,7 +33,7 @@ Snapshot | Value Metadata | Value ---------|------ - [configuration](#configuration) | [mergeGlobalConfigs](DeltaConfigs.md#mergeGlobalConfigs) + [configuration](#configuration) | [mergeGlobalConfigs](table-properties/DeltaConfigs.md#mergeGlobalConfigs) [createdTime](#createdTime) | Current time (in ms) ## computedState diff --git a/docs/Metadata.md b/docs/Metadata.md index f11fd0beee..8652b92529 100644 --- a/docs/Metadata.md +++ b/docs/Metadata.md @@ -65,7 +65,7 @@ deltaLog.snapshot.metadata.id columnMappingMode: DeltaColumnMappingMode ``` -`columnMappingMode` is the value of [columnMapping.mode](DeltaConfigs.md#COLUMN_MAPPING_MODE) table property ([from this Metadata](DeltaConfig.md#fromMetaData)). +`columnMappingMode` is the value of [columnMapping.mode](table-properties/DeltaConfigs.md#COLUMN_MAPPING_MODE) table property ([from this Metadata](table-properties/DeltaConfig.md#fromMetaData)). `columnMappingMode` is used when: diff --git a/docs/MetadataCleanup.md b/docs/MetadataCleanup.md index 35facea032..12dd6b95fe 100644 --- a/docs/MetadataCleanup.md +++ b/docs/MetadataCleanup.md @@ -13,11 +13,11 @@ ### enableExpiredLogCleanup -`MetadataCleanup` uses [enableExpiredLogCleanup](DeltaConfigs.md#ENABLE_EXPIRED_LOG_CLEANUP) table configuration to enable [log cleanup](#doLogCleanup). +`MetadataCleanup` uses [enableExpiredLogCleanup](table-properties/DeltaConfigs.md#ENABLE_EXPIRED_LOG_CLEANUP) table configuration to enable [log cleanup](#doLogCleanup). ### logRetentionDuration -`MetadataCleanup` uses [logRetentionDuration](DeltaConfigs.md#LOG_RETENTION) table configuration for [cleanUpExpiredLogs](#cleanUpExpiredLogs) (to determine `fileCutOffTime`). +`MetadataCleanup` uses [logRetentionDuration](table-properties/DeltaConfigs.md#LOG_RETENTION) table configuration for [cleanUpExpiredLogs](#cleanUpExpiredLogs) (to determine `fileCutOffTime`). ## Cleaning Up Expired Logs diff --git a/docs/OptimisticTransactionImpl.md b/docs/OptimisticTransactionImpl.md index b5ec4e3cec..c711f1a1c3 100644 --- a/docs/OptimisticTransactionImpl.md +++ b/docs/OptimisticTransactionImpl.md @@ -117,7 +117,7 @@ With all [action](FileAction.md)s with [dataChange](FileAction.md#dataChange) fl ### Registering Post-Commit Hook -`commit` [registers](#registerPostCommitHook) the [GenerateSymlinkManifest](post-commit-hooks/GenerateSymlinkManifest.md) post-commit hook when there is a [FileAction](FileAction.md) among the actions and the [compatibility.symlinkFormatManifest.enabled](DeltaConfigs.md#SYMLINK_FORMAT_MANIFEST_ENABLED) table property is enabled. +`commit` [registers](#registerPostCommitHook) the [GenerateSymlinkManifest](post-commit-hooks/GenerateSymlinkManifest.md) post-commit hook when there is a [FileAction](FileAction.md) among the actions and the [compatibility.symlinkFormatManifest.enabled](table-properties/DeltaConfigs.md#SYMLINK_FORMAT_MANIFEST_ENABLED) table property is enabled. ### doCommitRetryIteratively @@ -393,7 +393,7 @@ Attempting to commit version [attemptVersion] with [n] actions with [isolationLe A commit triggers checkpointing when the following all hold: 1. The committed version is any version greater than `0` -1. The committed version is a multiple of [delta.checkpointInterval](DeltaConfigs.md#CHECKPOINT_INTERVAL) table property +1. The committed version is a multiple of [delta.checkpointInterval](table-properties/DeltaConfigs.md#CHECKPOINT_INTERVAL) table property ### CommitStats diff --git a/docs/Protocol.md b/docs/Protocol.md index c5b60ac3f7..2fdd57cce7 100644 --- a/docs/Protocol.md +++ b/docs/Protocol.md @@ -114,7 +114,7 @@ Protocol(0, 2) ### Append-Only Table -`requiredMinimumProtocol` reads [appendOnly](DeltaConfigs.md#IS_APPEND_ONLY) table property (from the [table configuration](Metadata.md#configuration) of the given [Metadata](Metadata.md)). +`requiredMinimumProtocol` reads [appendOnly](table-properties/DeltaConfigs.md#IS_APPEND_ONLY) table property (from the [table configuration](Metadata.md#configuration) of the given [Metadata](Metadata.md)). If set, `requiredMinimumProtocol` creates a new [Protocol](#creating-instance) with the [minWriterVersion](#minWriterVersion) to be `3`. @@ -144,7 +144,7 @@ Protocol(0, 4) ### Change Data Feed -`requiredMinimumProtocol` checks whether [delta.enableChangeDataFeed](DeltaConfigs.md#CHANGE_DATA_FEED) table property is enabled (in the given [Metadata](Metadata.md)). +`requiredMinimumProtocol` checks whether [delta.enableChangeDataFeed](table-properties/DeltaConfigs.md#CHANGE_DATA_FEED) table property is enabled (in the given [Metadata](Metadata.md)). If enabled, `requiredMinimumProtocol` creates a new [Protocol](#creating-instance) with the [minWriterVersion](#minWriterVersion) to be `4`. diff --git a/docs/Snapshot.md b/docs/Snapshot.md index a16d477473..de638cf288 100644 --- a/docs/Snapshot.md +++ b/docs/Snapshot.md @@ -46,7 +46,7 @@ init(): Unit numIndexedCols: Int ``` -`numIndexedCols` is the value of [dataSkippingNumIndexedCols](DeltaConfigs.md#DATA_SKIPPING_NUM_INDEXED_COLS) table property. +`numIndexedCols` is the value of [dataSkippingNumIndexedCols](table-properties/DeltaConfigs.md#DATA_SKIPPING_NUM_INDEXED_COLS) table property. ??? note "Lazy Value" `numIndexedCols` is a Scala **lazy value** to guarantee that the code to initialize it is executed once only (when accessed for the first time) and the computed value never changes afterwards. diff --git a/docs/TransactionalWrite.md b/docs/TransactionalWrite.md index 369cb10ed5..1ae69959c9 100644 --- a/docs/TransactionalWrite.md +++ b/docs/TransactionalWrite.md @@ -222,7 +222,7 @@ getOptionalStatsTrackerAndStatsCollection( `getOptionalStatsTrackerAndStatsCollection` [getStatsSchema](#getStatsSchema) (for the given `output` and `partitionSchema`). -`getOptionalStatsTrackerAndStatsCollection` reads the value of [delta.dataSkippingNumIndexedCols](DeltaConfigs.md#DATA_SKIPPING_NUM_INDEXED_COLS) table property (from the [Metadata](OptimisticTransactionImpl.md#metadata)). +`getOptionalStatsTrackerAndStatsCollection` reads the value of [delta.dataSkippingNumIndexedCols](table-properties/DeltaConfigs.md#DATA_SKIPPING_NUM_INDEXED_COLS) table property (from the [Metadata](OptimisticTransactionImpl.md#metadata)). `getOptionalStatsTrackerAndStatsCollection` creates a [StatisticsCollection](StatisticsCollection.md) (with the [tableDataSchema](StatisticsCollection.md#tableDataSchema) based on [spark.databricks.delta.stats.collect.using.tableSchema](configuration-properties/DeltaSQLConf.md#DELTA_COLLECT_STATS_USING_TABLE_SCHEMA) configuration property). diff --git a/docs/append-only-tables/AppendOnlyTableFeature.md b/docs/append-only-tables/AppendOnlyTableFeature.md index d3423d94bc..3b446efc18 100644 --- a/docs/append-only-tables/AppendOnlyTableFeature.md +++ b/docs/append-only-tables/AppendOnlyTableFeature.md @@ -7,7 +7,7 @@ Property | Value [Name](../table-features/LegacyWriterFeature.md#name) | `appendOnly` [Minimum writer protocol version](../table-features/LegacyWriterFeature.md#minWriterVersion) | `2` -`AppendOnlyTableFeature` is a [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md) that uses [delta.appendOnly](../DeltaConfigs.md#appendOnly) table property to control [Append-Only Tables](index.md) feature. +`AppendOnlyTableFeature` is a [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md) that uses [delta.appendOnly](../table-properties/DeltaConfigs.md#appendOnly) table property to control [Append-Only Tables](index.md) feature. ## metadataRequiresFeatureToBeEnabled { #metadataRequiresFeatureToBeEnabled } @@ -21,4 +21,4 @@ Property | Value `metadataRequiresFeatureToBeEnabled` is part of the [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md#metadataRequiresFeatureToBeEnabled) abstraction. -`metadataRequiresFeatureToBeEnabled` is the value of [delta.appendOnly](../DeltaConfigs.md#IS_APPEND_ONLY) table property (from the [Metadata](../DeltaConfig.md#fromMetaData)). +`metadataRequiresFeatureToBeEnabled` is the value of [delta.appendOnly](../table-properties/DeltaConfigs.md#IS_APPEND_ONLY) table property (from the [Metadata](../table-properties/DeltaConfig.md#fromMetaData)). diff --git a/docs/append-only-tables/index.md b/docs/append-only-tables/index.md index 45652231ee..846a56e73c 100644 --- a/docs/append-only-tables/index.md +++ b/docs/append-only-tables/index.md @@ -11,11 +11,11 @@ hide: * `DeltaSink` to [addBatch](../spark-connector/DeltaSink.md#addBatch) in `Complete` output mode * [RemoveFile](../RemoveFile.md)s with [dataChange](../RemoveFile.md#dataChange) (at [prepareCommit](../OptimisticTransactionImpl.md#prepareCommit)) -Append-Only Tables is enabled on a delta table using [delta.appendOnly](../DeltaConfigs.md#IS_APPEND_ONLY) table property (indirectly, through [AppendOnlyTableFeature](AppendOnlyTableFeature.md) that is a [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md) and uses this table property). +Append-Only Tables is enabled on a delta table using [delta.appendOnly](../table-properties/DeltaConfigs.md#IS_APPEND_ONLY) table property (indirectly, through [AppendOnlyTableFeature](AppendOnlyTableFeature.md) that is a [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md) and uses this table property). ## Demo -Create a delta table with [delta.appendOnly](../DeltaConfigs.md#appendOnly) table property enabled. +Create a delta table with [delta.appendOnly](../table-properties/DeltaConfigs.md#appendOnly) table property enabled. === "SQL" diff --git a/docs/auto-compaction/AutoCompactBase.md b/docs/auto-compaction/AutoCompactBase.md index 8adf4588a2..8960c3a5e6 100644 --- a/docs/auto-compaction/AutoCompactBase.md +++ b/docs/auto-compaction/AutoCompactBase.md @@ -70,8 +70,8 @@ getAutoCompactType( `getAutoCompactType` is the value of the following (in the order of precedence): 1. [spark.databricks.delta.autoCompact.enabled](../configuration-properties/DeltaSQLConf.md#autoCompact.enabled), if configured. -1. [delta.autoOptimize](../DeltaConfigs.md#AUTO_OPTIMIZE) table property -1. [delta.autoOptimize.autoCompact](../DeltaConfigs.md#AUTO_COMPACT) table property +1. [delta.autoOptimize](../table-properties/DeltaConfigs.md#AUTO_OPTIMIZE) table property +1. [delta.autoOptimize.autoCompact](../table-properties/DeltaConfigs.md#AUTO_COMPACT) table property `getAutoCompactType` defaults to `false`. diff --git a/docs/auto-compaction/index.md b/docs/auto-compaction/index.md index db447f4bbb..0da87d9a71 100644 --- a/docs/auto-compaction/index.md +++ b/docs/auto-compaction/index.md @@ -4,9 +4,9 @@ Auto Compaction can be enabled system-wide using [spark.databricks.delta.autoCompact.enabled](../configuration-properties/index.md#spark.databricks.delta.autoCompact.enabled) configuration property. -Auto Compaction can be enabled on a delta table using [delta.autoOptimize.autoCompact](../DeltaConfigs.md#autoOptimize.autoCompact) table property. +Auto Compaction can be enabled on a delta table using [delta.autoOptimize.autoCompact](../table-properties/DeltaConfigs.md#autoOptimize.autoCompact) table property. ??? note "delta.autoOptimize Table Property is Deprecated" - [delta.autoOptimize](../DeltaConfigs.md#delta.autoOptimize) table property is deprecated. + [delta.autoOptimize](../table-properties/DeltaConfigs.md#delta.autoOptimize) table property is deprecated. Auto Compaction uses [AutoCompact](AutoCompact.md) post-commit hook to be [executed](AutoCompactBase.md#run) at a [successful transaction commit](../OptimisticTransactionImpl.md#registerPostCommitHook) if there are files written to a delta table that can leverage compaction after a commit. diff --git a/docs/change-data-feed/ChangeDataFeedTableFeature.md b/docs/change-data-feed/ChangeDataFeedTableFeature.md index e58154b0ae..1b5677d181 100644 --- a/docs/change-data-feed/ChangeDataFeedTableFeature.md +++ b/docs/change-data-feed/ChangeDataFeedTableFeature.md @@ -7,7 +7,7 @@ Property | Value [Name](../table-features/LegacyWriterFeature.md#name) | `changeDataFeed` [Minimum writer protocol version](../table-features/LegacyWriterFeature.md#minWriterVersion) | `4` -`ChangeDataFeedTableFeature` is a [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md) that uses [delta.enableChangeDataFeed](../DeltaConfigs.md#enableChangeDataFeed) table property to control [Change Data Feed](index.md) feature. +`ChangeDataFeedTableFeature` is a [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md) that uses [delta.enableChangeDataFeed](../table-properties/DeltaConfigs.md#enableChangeDataFeed) table property to control [Change Data Feed](index.md) feature. ## metadataRequiresFeatureToBeEnabled { #metadataRequiresFeatureToBeEnabled } @@ -21,4 +21,4 @@ Property | Value `metadataRequiresFeatureToBeEnabled` is part of the [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md#metadataRequiresFeatureToBeEnabled) abstraction. -`metadataRequiresFeatureToBeEnabled` is the value of [delta.enableChangeDataFeed](../DeltaConfigs.md#enableChangeDataFeed) table property in (the [configuration](../Metadata.md#configuration) of) the given [Metadata](../Metadata.md). +`metadataRequiresFeatureToBeEnabled` is the value of [delta.enableChangeDataFeed](../table-properties/DeltaConfigs.md#enableChangeDataFeed) table property in (the [configuration](../Metadata.md#configuration) of) the given [Metadata](../Metadata.md). diff --git a/docs/change-data-feed/index.md b/docs/change-data-feed/index.md index 4b86aff8b3..07eab373cd 100644 --- a/docs/change-data-feed/index.md +++ b/docs/change-data-feed/index.md @@ -4,6 +4,8 @@ Change Data Feed can be enabled on a delta table using [delta.enableChangeDataFeed](#delta.enableChangeDataFeed) table property. +Change Data Feed can be enabled globally (on all new delta tables) using [spark.databricks.delta.properties.defaults.enableChangeDataFeed](../table-properties/DeltaConfigs.md#spark.databricks.delta.properties.defaults) system-wide configuration property. + CDF data changes are written out (by [DelayedCommitProtocol](../DelayedCommitProtocol.md)) to [_change_data](#_change_data) directory as `cdc-`-prefixed parquet-encoded change data files. With [CDF-Aware Table Scan (CDF Read)](CDCReaderImpl.md#isCDCRead) (based on [readChangeFeed](../spark-connector/options.md#readChangeFeed) read option), [loading a delta table](../spark-connector/DeltaDataSource.md#RelationProvider-createRelation) gives data changes (not the data of a particular version of the delta table). @@ -17,7 +19,7 @@ Change Data Feed was released in Delta Lake 2.0.0 (that was tracked under [Suppo ## delta.enableChangeDataFeed { #delta.enableChangeDataFeed } -Change Data Feed can be enabled on a delta table using [delta.enableChangeDataFeed](../DeltaConfigs.md#enableChangeDataFeed) table property (through [ChangeDataFeedTableFeature](ChangeDataFeedTableFeature.md)). +Change Data Feed can be enabled on a delta table using [delta.enableChangeDataFeed](../table-properties/DeltaConfigs.md#enableChangeDataFeed) table property (through [ChangeDataFeedTableFeature](ChangeDataFeedTableFeature.md)). ```sql ALTER TABLE delta_demo diff --git a/docs/checkpoints/Checkpoints.md b/docs/checkpoints/Checkpoints.md index 2321577b8c..a9a6fa9451 100644 --- a/docs/checkpoints/Checkpoints.md +++ b/docs/checkpoints/Checkpoints.md @@ -93,7 +93,7 @@ In the end, `checkpoint` [cleans up the expired logs](../MetadataCleanup.md#doLo `checkpoint` is used when: -* `OptimisticTransactionImpl` is requested to [postCommit](../OptimisticTransactionImpl.md#postCommit) (based on [checkpoint interval](../DeltaConfigs.md#CHECKPOINT_INTERVAL) table property) +* `OptimisticTransactionImpl` is requested to [postCommit](../OptimisticTransactionImpl.md#postCommit) (based on [checkpoint interval](../table-properties/DeltaConfigs.md#CHECKPOINT_INTERVAL) table property) * `DeltaCommand` is requested to [updateAndCheckpoint](../commands/DeltaCommand.md#updateAndCheckpoint) ### checkpointAndCleanUpDeltaLog { #checkpointAndCleanUpDeltaLog } diff --git a/docs/checkpoints/index.md b/docs/checkpoints/index.md index 9cb66aa777..3adfa91e23 100644 --- a/docs/checkpoints/index.md +++ b/docs/checkpoints/index.md @@ -2,7 +2,7 @@ [Delta Table Checkpoint](Checkpoints.md#checkpoint) is a process of writing out a [Snapshot](../Snapshot.md) of a delta table into one or more checkpoint files for faster state reconstruction (_future replays of the log_). -Delta Table Checkpoint happens regularly at a [transaction commit](../OptimisticTransactionImpl.md#doCommit) every [checkpoint interval](../DeltaConfigs.md#CHECKPOINT_INTERVAL) or once at a [transaction commit](../OptimisticTransactionImpl.md#updateAndCheckpoint) for the following commands: +Delta Table Checkpoint happens regularly at a [transaction commit](../OptimisticTransactionImpl.md#doCommit) every [checkpoint interval](../table-properties/DeltaConfigs.md#CHECKPOINT_INTERVAL) or once at a [transaction commit](../OptimisticTransactionImpl.md#updateAndCheckpoint) for the following commands: * [CloneTableBase](../commands/clone/CloneTableBase.md) * [ConvertToDeltaCommand](../commands/convert/ConvertToDeltaCommand.md) diff --git a/docs/column-mapping/DeltaColumnMappingBase.md b/docs/column-mapping/DeltaColumnMappingBase.md index 0c6714c212..9dee3e9bc4 100644 --- a/docs/column-mapping/DeltaColumnMappingBase.md +++ b/docs/column-mapping/DeltaColumnMappingBase.md @@ -11,8 +11,8 @@ `DeltaColumnMappingBase` defines a [Protocol](../Protocol.md) (with [MIN_READER_VERSION](#MIN_READER_VERSION) and [MIN_WRITER_VERSION](#MIN_WRITER_VERSION)) as the minimum protocol version for the readers and writers to delta tables with [column mapping](index.md). * `Protocol` utility is used for [requiredMinimumProtocol](../Protocol.md#requiredMinimumProtocol) -* [delta.columnMapping.mode](../DeltaConfigs.md#COLUMN_MAPPING_MODE) configuration property -* [delta.columnMapping.maxColumnId](../DeltaConfigs.md#COLUMN_MAPPING_MAX_ID) configuration property +* [delta.columnMapping.mode](../table-properties/DeltaConfigs.md#COLUMN_MAPPING_MODE) configuration property +* [delta.columnMapping.maxColumnId](../table-properties/DeltaConfigs.md#COLUMN_MAPPING_MAX_ID) configuration property * `DeltaErrors` is requested to [changeColumnMappingModeOnOldProtocol](../DeltaErrors.md#changeColumnMappingModeOnOldProtocol) (for error reporting) ### Minimum Reader Version @@ -145,7 +145,7 @@ tryFixMetadata( isChangingModeOnExistingTable: Boolean): Metadata ``` -`tryFixMetadata` reads [columnMapping.mode](../DeltaConfigs.md#columnMapping.mode) table property from the given `newMetadata` [table metadata](../DeltaConfig.md#fromMetaData). +`tryFixMetadata` reads [columnMapping.mode](../table-properties/DeltaConfigs.md#columnMapping.mode) table property from the given `newMetadata` [table metadata](../table-properties/DeltaConfig.md#fromMetaData). If the [DeltaColumnMappingMode](DeltaColumnMappingMode.md) is [IdMapping](DeltaColumnMappingMode.md#IdMapping) or [NameMapping](DeltaColumnMappingMode.md#NameMapping), `tryFixMetadata` [assignColumnIdAndPhysicalName](#assignColumnIdAndPhysicalName) with the given `newMetadata` and `oldMetadata` metadata and `isChangingModeOnExistingTable` flag. diff --git a/docs/column-mapping/DeltaColumnMappingMode.md b/docs/column-mapping/DeltaColumnMappingMode.md index 133e2ea014..23b0407c7b 100644 --- a/docs/column-mapping/DeltaColumnMappingMode.md +++ b/docs/column-mapping/DeltaColumnMappingMode.md @@ -74,4 +74,4 @@ Please upgrade if you want to use this mode. `apply` is used when: -* `DeltaConfigsBase` is requested to build [delta.columnMapping.mode](../DeltaConfigs.md#COLUMN_MAPPING_MODE) configuration property +* `DeltaConfigsBase` is requested to build [delta.columnMapping.mode](../table-properties/DeltaConfigs.md#COLUMN_MAPPING_MODE) configuration property diff --git a/docs/column-mapping/index.md b/docs/column-mapping/index.md index dd5eb7fb2b..d4a83f2152 100644 --- a/docs/column-mapping/index.md +++ b/docs/column-mapping/index.md @@ -2,7 +2,7 @@ **Column Mapping** allows mapping physical columns names to their logical ones for reading and writing parquet data files. -Column Mapping is enabled for a delta table using [delta.columnMapping.mode](../DeltaConfigs.md#COLUMN_MAPPING_MODE) table property. +Column Mapping is enabled for a delta table using [delta.columnMapping.mode](../table-properties/DeltaConfigs.md#COLUMN_MAPPING_MODE) table property. Column Mapping turns [ALTER TABLE RENAME COLUMN](../commands/alter/AlterTableChangeColumnDeltaCommand.md) and [ALTER TABLE CHANGE COLUMN](../commands/alter/AlterTableChangeColumnDeltaCommand.md) commands into logical changes at metadata level (leading to no changes to the physical column names in parquet files and a mere [transactional metadata update](../commands/alter/AlterTableChangeColumnDeltaCommand.md#run-update) in a transaction log). diff --git a/docs/column-statistics/index.md b/docs/column-statistics/index.md index 6d28859922..5d216b43c4 100644 --- a/docs/column-statistics/index.md +++ b/docs/column-statistics/index.md @@ -9,4 +9,4 @@ ## Table Properties -* [delta.dataSkippingNumIndexedCols](../DeltaConfigs.md#dataSkippingNumIndexedCols) +* [delta.dataSkippingNumIndexedCols](../table-properties/DeltaConfigs.md#dataSkippingNumIndexedCols) diff --git a/docs/commands/delete/DeleteCommand.md b/docs/commands/delete/DeleteCommand.md index e9fa68c92a..fe69631897 100644 --- a/docs/commands/delete/DeleteCommand.md +++ b/docs/commands/delete/DeleteCommand.md @@ -97,7 +97,7 @@ rewriteFiles( numFilesToRewrite: Long): Seq[FileAction] ``` -`rewriteFiles` reads the [delta.enableChangeDataFeed](../../DeltaConfigs.md#CHANGE_DATA_FEED) table property of the delta table (from the [Metadata](../../OptimisticTransactionImpl.md#metadata) of the given [OptimisticTransaction](../../OptimisticTransaction.md)). +`rewriteFiles` reads the [delta.enableChangeDataFeed](../../table-properties/DeltaConfigs.md#CHANGE_DATA_FEED) table property of the delta table (from the [Metadata](../../OptimisticTransactionImpl.md#metadata) of the given [OptimisticTransaction](../../OptimisticTransaction.md)). `rewriteFiles` creates a `numTouchedRows` metric and a `numTouchedRowsUdf` UDF to count the number of rows that have been _touched_. diff --git a/docs/commands/merge/DeduplicateCDFDeletes.md b/docs/commands/merge/DeduplicateCDFDeletes.md index 5ba03bc39a..1431b9700a 100644 --- a/docs/commands/merge/DeduplicateCDFDeletes.md +++ b/docs/commands/merge/DeduplicateCDFDeletes.md @@ -23,7 +23,7 @@ More precisely, `DeduplicateCDFDeletes` is an input argument of [MergeOutputGene `DeduplicateCDFDeletes` is given an `enabled` flag when [created](#creating-instance) based on the following: * Whether there are multiple matches (`hasMultipleMatches`) -* [enableChangeDataFeed](../../DeltaConfigs.md#enableChangeDataFeed) table property +* [enableChangeDataFeed](../../table-properties/DeltaConfigs.md#enableChangeDataFeed) table property ### includesInserts Flag { #includesInserts } diff --git a/docs/commands/merge/MergeIntoCommandBase.md b/docs/commands/merge/MergeIntoCommandBase.md index 937a2f4805..335389e6c6 100644 --- a/docs/commands/merge/MergeIntoCommandBase.md +++ b/docs/commands/merge/MergeIntoCommandBase.md @@ -505,7 +505,7 @@ isCdcEnabled( deltaTxn: OptimisticTransaction): Boolean ``` -`isCdcEnabled` is the value of the [enableChangeDataFeed](../../DeltaConfigs.md#CHANGE_DATA_FEED) table property ([from](../../DeltaConfig.md#fromMetaData) the metadata of a delta table). +`isCdcEnabled` is the value of the [enableChangeDataFeed](../../table-properties/DeltaConfigs.md#CHANGE_DATA_FEED) table property ([from](../../table-properties/DeltaConfig.md#fromMetaData) the metadata of a delta table). --- diff --git a/docs/data-skipping/DataSkippingReaderBase.md b/docs/data-skipping/DataSkippingReaderBase.md index 33c5e9b47b..82e81a1e50 100644 --- a/docs/data-skipping/DataSkippingReaderBase.md +++ b/docs/data-skipping/DataSkippingReaderBase.md @@ -267,7 +267,7 @@ Name | Expression columnMappingMode: DeltaColumnMappingMode ``` -`columnMappingMode` is the value of [columnMapping.mode](../DeltaConfigs.md#COLUMN_MAPPING_MODE) table property (in the [Metadata](#metadata)). +`columnMappingMode` is the value of [columnMapping.mode](../table-properties/DeltaConfigs.md#COLUMN_MAPPING_MODE) table property (in the [Metadata](#metadata)). ## getStatsColumnOpt diff --git a/docs/deletion-vectors/DeletionVectorUtils.md b/docs/deletion-vectors/DeletionVectorUtils.md index a7008c1761..ba6c009824 100644 --- a/docs/deletion-vectors/DeletionVectorUtils.md +++ b/docs/deletion-vectors/DeletionVectorUtils.md @@ -42,7 +42,7 @@ deletionVectorsWritable( `deletionVectorsWritable` is enabled (`true`) when the following all hold: 1. [DeletionVectorsTableFeature](DeletionVectorsTableFeature.md) is [supported](../table-features/TableFeatureSupport.md#isFeatureSupported) by the given [Protocol](../Protocol.md) -1. [delta.enableDeletionVectors](../DeltaConfigs.md#ENABLE_DELETION_VECTORS_CREATION) is enabled in the table metadata (in the given [Metadata](../Metadata.md)) +1. [delta.enableDeletionVectors](../table-properties/DeltaConfigs.md#ENABLE_DELETION_VECTORS_CREATION) is enabled in the table metadata (in the given [Metadata](../Metadata.md)) --- @@ -70,5 +70,5 @@ isTableDVFree( `isTableDVFree` is used when: -* `Protocol` is requested to [assertTablePropertyConstraintsSatisfied](../Protocol.md#assertTablePropertyConstraintsSatisfied) (with [compatibility.symlinkFormatManifest.enabled](../DeltaConfigs.md#compatibility.symlinkFormatManifest.enabled) enabled) +* `Protocol` is requested to [assertTablePropertyConstraintsSatisfied](../Protocol.md#assertTablePropertyConstraintsSatisfied) (with [compatibility.symlinkFormatManifest.enabled](../table-properties/DeltaConfigs.md#compatibility.symlinkFormatManifest.enabled) enabled) * `GenerateSymlinkManifestImpl` is requested to `generateFullManifest` diff --git a/docs/deletion-vectors/DeletionVectorsTableFeature.md b/docs/deletion-vectors/DeletionVectorsTableFeature.md index 8f150e0892..a9568659c3 100644 --- a/docs/deletion-vectors/DeletionVectorsTableFeature.md +++ b/docs/deletion-vectors/DeletionVectorsTableFeature.md @@ -2,7 +2,7 @@ `DeletionVectorsTableFeature` is a [ReaderWriterFeature](../table-features/ReaderWriterFeature.md) known by the name of [deletionVectors](../table-features/TableFeature.md#name). -With [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) table property enabled, `DeletionVectorsTableFeature` enables [Deletion Vectors](index.md). +With [delta.enableDeletionVectors](../table-properties/DeltaConfigs.md#enableDeletionVectors) table property enabled, `DeletionVectorsTableFeature` enables [Deletion Vectors](index.md). `DeletionVectorsTableFeature` is a [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md). @@ -30,4 +30,4 @@ With [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) tab `metadataRequiresFeatureToBeEnabled` is part of the [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md#metadataRequiresFeatureToBeEnabled) abstraction. -`metadataRequiresFeatureToBeEnabled` is the value of [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) table property (in the [Metadata](../DeltaConfig.md#fromMetaData)). +`metadataRequiresFeatureToBeEnabled` is the value of [delta.enableDeletionVectors](../table-properties/DeltaConfigs.md#enableDeletionVectors) table property (in the [Metadata](../table-properties/DeltaConfig.md#fromMetaData)). diff --git a/docs/deletion-vectors/index.md b/docs/deletion-vectors/index.md index 54eccd7cc2..76a5f75a2c 100644 --- a/docs/deletion-vectors/index.md +++ b/docs/deletion-vectors/index.md @@ -9,12 +9,12 @@ hide: It is said that Deletion Vectors feature soft-deletes data. -Deletion Vectors can be enabled on a delta table using [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) table property. +Deletion Vectors can be enabled on a delta table using [delta.enableDeletionVectors](../table-properties/DeltaConfigs.md#enableDeletionVectors) table property. Deletion Vectors is used on a delta table when all of the following hold: 1. [spark.databricks.delta.delete.deletionVectors.persistent](../configuration-properties/DeltaSQLConf.md#DELETE_USE_PERSISTENT_DELETION_VECTORS) system-wide configuration property is enabled -1. [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) table property is enabled +1. [delta.enableDeletionVectors](../table-properties/DeltaConfigs.md#enableDeletionVectors) table property is enabled 1. [DeletionVectorsTableFeature](DeletionVectorsTableFeature.md) is [supported](../table-features/TableFeatureSupport.md#isFeatureSupported) by the [Protocol](../Protocol.md) ## REORG TABLE Command @@ -27,7 +27,7 @@ Deletion Vectors is used on a delta table when all of the following hold: ## Demo -Create a delta table with [delta.enableDeletionVectors](../DeltaConfigs.md#enableDeletionVectors) table property enabled. +Create a delta table with [delta.enableDeletionVectors](../table-properties/DeltaConfigs.md#enableDeletionVectors) table property enabled. === "SQL" diff --git a/docs/demo/vacuum.md b/docs/demo/vacuum.md index 144a6e517e..5d57bbc4f8 100644 --- a/docs/demo/vacuum.md +++ b/docs/demo/vacuum.md @@ -159,7 +159,7 @@ If you are not sure, please use a value not less than "168 hours". Attempting to vacuum the delta table (even with `DRY RUN`) gives an `IllegalArgumentException` because of the default values of the following: * [spark.databricks.delta.retentionDurationCheck.enabled](../configuration-properties/DeltaSQLConf.md#DELTA_VACUUM_RETENTION_CHECK_ENABLED) configuration property -* [deletedFileRetentionDuration](../DeltaConfigs.md#TOMBSTONE_RETENTION) table property +* [deletedFileRetentionDuration](../table-properties/DeltaConfigs.md#TOMBSTONE_RETENTION) table property ## Vacuum DRY RUN @@ -215,7 +215,7 @@ q.show(truncate = false) ### deletedFileRetentionDuration Table Property -Let's [DESCRIBE DETAIL](../commands/describe-detail/index.md) to review the current table properties (incl. [deletedFileRetentionDuration](../DeltaConfigs.md#TOMBSTONE_RETENTION)). +Let's [DESCRIBE DETAIL](../commands/describe-detail/index.md) to review the current table properties (incl. [deletedFileRetentionDuration](../table-properties/DeltaConfigs.md#TOMBSTONE_RETENTION)). ```scala val tid = s"delta.`$path`" diff --git a/docs/post-commit-hooks/GenerateSymlinkManifest.md b/docs/post-commit-hooks/GenerateSymlinkManifest.md index 983efada1f..df6a38da5a 100644 --- a/docs/post-commit-hooks/GenerateSymlinkManifest.md +++ b/docs/post-commit-hooks/GenerateSymlinkManifest.md @@ -6,7 +6,7 @@ title: GenerateSymlinkManifest `GenerateSymlinkManifest` is a [post-commit hook](PostCommitHook.md) to generate [incremental](#generateIncrementalManifest) and [full](#generateFullManifest) Hive-style manifests for delta tables. -`GenerateSymlinkManifest` is registered when `OptimisticTransactionImpl` is requested to [commit](../OptimisticTransaction.md#commit) (with [delta.compatibility.symlinkFormatManifest.enabled](../DeltaConfigs.md#SYMLINK_FORMAT_MANIFEST_ENABLED) table property enabled). +`GenerateSymlinkManifest` is registered when `OptimisticTransactionImpl` is requested to [commit](../OptimisticTransaction.md#commit) (with [delta.compatibility.symlinkFormatManifest.enabled](../table-properties/DeltaConfigs.md#SYMLINK_FORMAT_MANIFEST_ENABLED) table property enabled). ## Executing Post-Commit Hook diff --git a/docs/row-tracking/RowId.md b/docs/row-tracking/RowId.md index 9aec0cbc15..e6534dcb50 100644 --- a/docs/row-tracking/RowId.md +++ b/docs/row-tracking/RowId.md @@ -42,7 +42,7 @@ isEnabled( `isEnabled` holds `true` when the following are all met: -1. [delta.enableRowTracking](../DeltaConfigs.md#enableRowTracking) table property is enabled ([in the given Metadata](../DeltaConfig.md#fromMetaData)) +1. [delta.enableRowTracking](../table-properties/DeltaConfigs.md#enableRowTracking) table property is enabled ([in the given Metadata](../table-properties/DeltaConfig.md#fromMetaData)) 1. Row Tracking feature [is supported](#isSupported) !!! note "RowTracking" diff --git a/docs/row-tracking/RowTracking.md b/docs/row-tracking/RowTracking.md index b1dde4e20f..399271a12a 100644 --- a/docs/row-tracking/RowTracking.md +++ b/docs/row-tracking/RowTracking.md @@ -28,7 +28,7 @@ isEnabled( `isEnabled` is enabled (`true`) when the following are all met: -1. [delta.enableRowTracking](../DeltaConfigs.md#ROW_TRACKING_ENABLED) table property is enabled ([in the given Metadata](../DeltaConfig.md#fromMetaData)) +1. [delta.enableRowTracking](../table-properties/DeltaConfigs.md#ROW_TRACKING_ENABLED) table property is enabled ([in the given Metadata](../table-properties/DeltaConfig.md#fromMetaData)) 1. Row Tracking feature [is supported](#isSupported) !!! danger "Not Used" diff --git a/docs/row-tracking/RowTrackingFeature.md b/docs/row-tracking/RowTrackingFeature.md index 676807c143..64b39f9b90 100644 --- a/docs/row-tracking/RowTrackingFeature.md +++ b/docs/row-tracking/RowTrackingFeature.md @@ -2,7 +2,7 @@ `RowTrackingFeature` is a [WriterFeature](../table-features/WriterFeature.md) known by the name of [rowTracking](../table-features/TableFeature.md#name). -With [delta.enableRowTracking](../DeltaConfigs.md#ROW_TRACKING_ENABLED) table property enabled, `RowTrackingFeature` enables [Row Tracking](index.md). +With [delta.enableRowTracking](../table-properties/DeltaConfigs.md#ROW_TRACKING_ENABLED) table property enabled, `RowTrackingFeature` enables [Row Tracking](index.md). `RowTrackingFeature` is a [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md). @@ -30,7 +30,7 @@ With [delta.enableRowTracking](../DeltaConfigs.md#ROW_TRACKING_ENABLED) table pr `metadataRequiresFeatureToBeEnabled` is part of the [FeatureAutomaticallyEnabledByMetadata](../table-features/FeatureAutomaticallyEnabledByMetadata.md#metadataRequiresFeatureToBeEnabled) abstraction. -`metadataRequiresFeatureToBeEnabled` is the value of [delta.enableRowTracking](../DeltaConfigs.md#ROW_TRACKING_ENABLED) table property (from the [Metadata](../DeltaConfig.md#fromMetaData)). +`metadataRequiresFeatureToBeEnabled` is the value of [delta.enableRowTracking](../table-properties/DeltaConfigs.md#ROW_TRACKING_ENABLED) table property (from the [Metadata](../table-properties/DeltaConfig.md#fromMetaData)). ## Required Features { #requiredFeatures } diff --git a/docs/row-tracking/index.md b/docs/row-tracking/index.md index a0b70659b0..62f66e3641 100644 --- a/docs/row-tracking/index.md +++ b/docs/row-tracking/index.md @@ -8,7 +8,7 @@ hide: !!! danger "Under Development" Row Tracking is under development and only available in testing. -**Row Tracking** is enabled on a delta table using [delta.enableRowTracking](../DeltaConfigs.md#ROW_TRACKING_ENABLED) table property. +**Row Tracking** is enabled on a delta table using [delta.enableRowTracking](../table-properties/DeltaConfigs.md#ROW_TRACKING_ENABLED) table property. === "SQL" diff --git a/docs/spark-connector/DeltaDataSource.md b/docs/spark-connector/DeltaDataSource.md index f090d763b8..da3f535678 100644 --- a/docs/spark-connector/DeltaDataSource.md +++ b/docs/spark-connector/DeltaDataSource.md @@ -73,7 +73,7 @@ createRelation( `createSource` creates a [DeltaOptions](DeltaOptions.md) (with the given `parameters` and the current `SQLConf`). -`createSource` [validateConfigurations](../DeltaConfigs.md#validateConfigurations) (with `delta.`-prefixed keys in the given`parameters`). +`createSource` [validateConfigurations](../table-properties/DeltaConfigs.md#validateConfigurations) (with `delta.`-prefixed keys in the given`parameters`). `createRelation` [creates and executes a WriteIntoDelta command](../commands/WriteIntoDelta.md) with the given `data`. diff --git a/docs/table-features/TableFeatureProtocolUtils.md b/docs/table-features/TableFeatureProtocolUtils.md index 9f4f07fd35..4b1e4f72fd 100644 --- a/docs/table-features/TableFeatureProtocolUtils.md +++ b/docs/table-features/TableFeatureProtocolUtils.md @@ -10,7 +10,7 @@ delta.feature. The prefix is used when: -* `DeltaConfigsBase` is requested to [validateConfigurations](../DeltaConfigs.md#validateConfigurations), [mergeGlobalConfigs](../DeltaConfigs.md#mergeGlobalConfigs), [normalizeConfigKeys](../DeltaConfigs.md#normalizeConfigKeys), [normalizeConfigKey](../DeltaConfigs.md#normalizeConfigKey) +* `DeltaConfigsBase` is requested to [validateConfigurations](../table-properties/DeltaConfigs.md#validateConfigurations), [mergeGlobalConfigs](../table-properties/DeltaConfigs.md#mergeGlobalConfigs), [normalizeConfigKeys](../table-properties/DeltaConfigs.md#normalizeConfigKeys), [normalizeConfigKey](../table-properties/DeltaConfigs.md#normalizeConfigKey) * `Snapshot` is requested to [getProperties](../Snapshot.md#getProperties) * `Protocol` is requested to [assertMetadataContainsNoProtocolProps](../Protocol.md#assertMetadataContainsNoProtocolProps) * `TableFeatureProtocolUtils` is requested to [propertyKey](#propertyKey), [getSupportedFeaturesFromTableConfigs](#getSupportedFeaturesFromTableConfigs), [isTableProtocolProperty](#isTableProtocolProperty) @@ -25,7 +25,7 @@ spark.databricks.delta.properties.defaults.feature. The prefix is used when: -* `DeltaConfigsBase` is requested to [mergeGlobalConfigs](../DeltaConfigs.md#mergeGlobalConfigs) +* `DeltaConfigsBase` is requested to [mergeGlobalConfigs](../table-properties/DeltaConfigs.md#mergeGlobalConfigs) * `TableFeatureProtocolUtils` is requested for the [default session property key of a table feature](#defaultPropertyKey) ## isTableProtocolProperty { #isTableProtocolProperty } @@ -39,14 +39,14 @@ isTableProtocolProperty( * [delta.minReaderVersion](../Protocol.md#MIN_READER_VERSION_PROP) * [delta.minWriterVersion](../Protocol.md#MIN_WRITER_VERSION_PROP) -* [delta.ignoreProtocolDefaults](../DeltaConfigs.md#CREATE_TABLE_IGNORE_PROTOCOL_DEFAULTS) +* [delta.ignoreProtocolDefaults](../table-properties/DeltaConfigs.md#CREATE_TABLE_IGNORE_PROTOCOL_DEFAULTS) * The `key` starts with the [delta.feature.](#FEATURE_PROP_PREFIX) prefix --- `isTableProtocolProperty` is used when: -* `DeltaConfigsBase` is requested to [mergeGlobalConfigs](../DeltaConfigs.md#mergeGlobalConfigs) +* `DeltaConfigsBase` is requested to [mergeGlobalConfigs](../table-properties/DeltaConfigs.md#mergeGlobalConfigs) * `OptimisticTransactionImpl` is requested to [updateMetadataInternal](../OptimisticTransactionImpl.md#updateMetadataInternal) * `Protocol` is requested to [forNewTable](../Protocol.md#forNewTable) diff --git a/docs/DeltaConfig.md b/docs/table-properties/DeltaConfig.md similarity index 59% rename from docs/DeltaConfig.md rename to docs/table-properties/DeltaConfig.md index 2d58cb6979..498f425278 100644 --- a/docs/DeltaConfig.md +++ b/docs/table-properties/DeltaConfig.md @@ -11,31 +11,31 @@ * Conversion function (from text representation of the `DeltaConfig` to the `T` type, i.e. `String => T`) * Validation function (that guards from incorrect values, i.e. `T => Boolean`) * Help message -* (optional) Minimum version of [protocol](Protocol.md) supported (default: undefined) +* (optional) Minimum version of [protocol](../Protocol.md) supported (default: undefined) `DeltaConfig` is created when: * `DeltaConfigs` utility is used to [build a DeltaConfig](DeltaConfigs.md#buildConfig) -## Reading Configuration Property From Metadata +## Reading Configuration Property From Metadata { #fromMetaData } ```scala fromMetaData( metadata: Metadata): T ``` -`fromMetaData` looks up the [key](#key) in the [configuration](Metadata.md#configuration) of the given [Metadata](Metadata.md). If not found, `fromMetaData` gives the [default value](#defaultValue). +`fromMetaData` looks up the [key](#key) in the [configuration](../Metadata.md#configuration) of the given [Metadata](../Metadata.md). If not found, `fromMetaData` gives the [default value](#defaultValue). In the end, `fromMetaData` converts the text representation to the proper type using [fromString](#fromString) conversion function. `fromMetaData` is used when: -* `Checkpoints` utility is used to [buildCheckpoint](checkpoints/Checkpoints.md#buildCheckpoint) -* `DeltaErrors` utility is used to [logFileNotFoundException](DeltaErrors.md#logFileNotFoundException) -* `DeltaLog` is requested for [checkpointInterval](DeltaLog.md#checkpointInterval) and [deletedFileRetentionDuration](DeltaLog.md#tombstoneRetentionMillis) table properties, and to [assert a table is not read-only](DeltaLog.md#assertRemovable) -* `MetadataCleanup` is requested for the [enableExpiredLogCleanup](MetadataCleanup.md#enableExpiredLogCleanup) and the [deltaRetentionMillis](MetadataCleanup.md#deltaRetentionMillis) -* `OptimisticTransactionImpl` is requested to [commit](OptimisticTransactionImpl.md#commit) -* `Snapshot` is requested for the [numIndexedCols](Snapshot.md#numIndexedCols) +* `Checkpoints` utility is used to [buildCheckpoint](../checkpoints/Checkpoints.md#buildCheckpoint) +* `DeltaErrors` utility is used to [logFileNotFoundException](../DeltaErrors.md#logFileNotFoundException) +* `DeltaLog` is requested for [checkpointInterval](../DeltaLog.md#checkpointInterval) and [deletedFileRetentionDuration](../DeltaLog.md#tombstoneRetentionMillis) table properties, and to [assert a table is not read-only](../DeltaLog.md#assertRemovable) +* `MetadataCleanup` is requested for the [enableExpiredLogCleanup](../MetadataCleanup.md#enableExpiredLogCleanup) and the [deltaRetentionMillis](../MetadataCleanup.md#deltaRetentionMillis) +* `OptimisticTransactionImpl` is requested to [commit](../OptimisticTransactionImpl.md#commit) +* `Snapshot` is requested for the [numIndexedCols](../Snapshot.md#numIndexedCols) ## Demo diff --git a/docs/DeltaConfigs.md b/docs/table-properties/DeltaConfigs.md similarity index 68% rename from docs/DeltaConfigs.md rename to docs/table-properties/DeltaConfigs.md index 67165df420..c837abee3c 100644 --- a/docs/DeltaConfigs.md +++ b/docs/table-properties/DeltaConfigs.md @@ -1,6 +1,10 @@ +--- +title: DeltaConfigs +--- + # DeltaConfigs (DeltaConfigsBase) -`DeltaConfigs` holds the supported [table properties](table-properties/index.md) in Delta Lake. +`DeltaConfigs` holds the supported [table properties](index.md) in Delta Lake. ## Accessing DeltaConfigs @@ -19,6 +23,12 @@ import org.apache.spark.sql.delta.DeltaConfigs DeltaConfigs.CHANGE_DATA_FEED.fromMetaData(metadata) ``` +## System-Wide Defaults { #spark.databricks.delta.properties.defaults } + +**spark.databricks.delta.properties.defaults** prefix is used for [global table properties](#mergeGlobalConfigs). + +For every [table property](#table-properties) (without the `delta.` prefix) there is the corresponding system-wide (*global*) configuration property with `spark.databricks.delta.properties.defaults` prefix for the default values of the table properties for all delta tables. + ## Table Properties All table properties start with `delta.` prefix. @@ -27,7 +37,7 @@ All table properties start with `delta.` prefix. **delta.appendOnly** -Turns a table into [append-only](append-only-tables/index.md) +Turns a table into [append-only](../append-only-tables/index.md) When enabled, a table allows appends only and no updates or deletes. @@ -35,8 +45,8 @@ Default: `false` Used when: -* `DeltaLog` is requested to [assertRemovable](DeltaLog.md#assertRemovable) (that in turn uses `DeltaErrors` utility to [modifyAppendOnlyTableException](DeltaErrors.md#modifyAppendOnlyTableException)) -* `AppendOnlyTableFeature` is requested to [metadataRequiresFeatureToBeEnabled](append-only-tables/AppendOnlyTableFeature.md#metadataRequiresFeatureToBeEnabled) +* `DeltaLog` is requested to [assertRemovable](../DeltaLog.md#assertRemovable) (that in turn uses `DeltaErrors` utility to [modifyAppendOnlyTableException](../DeltaErrors.md#modifyAppendOnlyTableException)) +* `AppendOnlyTableFeature` is requested to [metadataRequiresFeatureToBeEnabled](../append-only-tables/AppendOnlyTableFeature.md#metadataRequiresFeatureToBeEnabled) ### autoOptimize { #autoOptimize } @@ -53,7 +63,7 @@ Default: `false` **delta.autoOptimize.autoCompact** -Enables [Auto Compaction](auto-compaction/index.md) +Enables [Auto Compaction](../auto-compaction/index.md) Default: `false` @@ -62,11 +72,11 @@ Default: `false` Used when: -* `AutoCompactBase` is requested for the [type of Auto Compaction](auto-compaction/AutoCompactBase.md#getAutoCompactType) +* `AutoCompactBase` is requested for the [type of Auto Compaction](../auto-compaction/AutoCompactBase.md#getAutoCompactType) ### checkpointInterval -How often to [checkpoint](checkpoints/Checkpoints.md#checkpoint) the state of a delta table (at the end of [transaction commit](OptimisticTransactionImpl.md#postCommit)) +How often to [checkpoint](../checkpoints/Checkpoints.md#checkpoint) the state of a delta table (at the end of [transaction commit](../OptimisticTransactionImpl.md#postCommit)) Default: `10` @@ -100,7 +110,7 @@ Default: `undefined` (`Option[Boolean]`) **delta.columnMapping.maxColumnId** -Maximum columnId used in the schema so far for [column mapping](column-mapping/index.md) +Maximum columnId used in the schema so far for [column mapping](../column-mapping/index.md) Cannot be set @@ -110,26 +120,26 @@ Default: `0` **delta.columnMapping.mode** -[DeltaColumnMappingMode](column-mapping/DeltaColumnMappingMode.md) to read and write parquet data files +[DeltaColumnMappingMode](../column-mapping/DeltaColumnMappingMode.md) to read and write parquet data files Name | Description --------|------------ `none` | **(default)** A display name is the only valid identifier of a column `id` | A column ID is the identifier of a column. This mode is used for tables converted from Iceberg and parquet files in this mode will also have corresponding field Ids for each column in their file schema. - `name` | The physical column name is the identifier of a column. Stored as part of `StructField` metadata in the schema. Used for reading statistics and partition values in the [DeltaLog](DeltaLog.md). + `name` | The physical column name is the identifier of a column. Stored as part of `StructField` metadata in the schema. Used for reading statistics and partition values in the [DeltaLog](../DeltaLog.md). Used when: -* `DeltaColumnMappingBase` is requested to [tryFixMetadata](column-mapping/DeltaColumnMappingBase.md#tryFixMetadata) (while `OptimisticTransactionImpl` is requested to [update the metadata](OptimisticTransactionImpl.md#updateMetadata)) -* `DeltaErrors` utility is used to [create a DeltaColumnMappingUnsupportedException](DeltaErrors.md#changeColumnMappingModeOnOldProtocol) (while `OptimisticTransactionImpl` is requested to [update the metadata](OptimisticTransactionImpl.md#updateMetadata)) -* `DeltaErrors` utility is used to [create a DeltaColumnMappingUnsupportedException](DeltaErrors.md#convertToDeltaWithColumnMappingNotSupported) (while [ConvertToDeltaCommand](commands/convert/ConvertToDeltaCommand.md) is executed) -* `Metadata` is requested for the [column mapping mode](Metadata.md#columnMappingMode) (while `DeltaFileFormat` is requested for the [FileFormat](DeltaFileFormat.md#fileFormat)) +* `DeltaColumnMappingBase` is requested to [tryFixMetadata](../column-mapping/DeltaColumnMappingBase.md#tryFixMetadata) (while `OptimisticTransactionImpl` is requested to [update the metadata](../OptimisticTransactionImpl.md#updateMetadata)) +* `DeltaErrors` utility is used to [create a DeltaColumnMappingUnsupportedException](../DeltaErrors.md#changeColumnMappingModeOnOldProtocol) (while `OptimisticTransactionImpl` is requested to [update the metadata](../OptimisticTransactionImpl.md#updateMetadata)) +* `DeltaErrors` utility is used to [create a DeltaColumnMappingUnsupportedException](../DeltaErrors.md#convertToDeltaWithColumnMappingNotSupported) (while [ConvertToDeltaCommand](../commands/convert/ConvertToDeltaCommand.md) is executed) +* `Metadata` is requested for the [column mapping mode](../Metadata.md#columnMappingMode) (while `DeltaFileFormat` is requested for the [FileFormat](../DeltaFileFormat.md#fileFormat)) ### compatibility.symlinkFormatManifest.enabled { #compatibility.symlinkFormatManifest.enabled } **delta.compatibility.symlinkFormatManifest.enabled** -Whether to register the [GenerateSymlinkManifest](post-commit-hooks/GenerateSymlinkManifest.md) post-commit hook while [committing a transaction](OptimisticTransactionImpl.md#commit) or not +Whether to register the [GenerateSymlinkManifest](../post-commit-hooks/GenerateSymlinkManifest.md) post-commit hook while [committing a transaction](../OptimisticTransactionImpl.md#commit) or not Default: `false` @@ -137,7 +147,7 @@ Default: `false` **delta.dataSkippingNumIndexedCols** -The number of columns to collect stats on for [data skipping](data-skipping/index.md). `-1` means collecting stats for all columns. +The number of columns to collect stats on for [data skipping](../data-skipping/index.md). `-1` means collecting stats for all columns. Default: `32` @@ -145,8 +155,8 @@ Must be larger than or equal to `-1`. Used when: -* `Snapshot` is requested for the [maximum number of indexed columns](Snapshot.md#numIndexedCols) -* `TransactionalWrite` is requested to [write data out](TransactionalWrite.md#writeFiles) +* `Snapshot` is requested for the [maximum number of indexed columns](../Snapshot.md#numIndexedCols) +* `TransactionalWrite` is requested to [write data out](../TransactionalWrite.md#writeFiles) ### deletedFileRetentionDuration { #deletedFileRetentionDuration } @@ -160,7 +170,7 @@ Default: `interval 1 week` **delta.enableChangeDataFeed** -Enables [Change Data Feed](change-data-feed/index.md) +Enables [Change Data Feed](../change-data-feed/index.md) Default: `false` @@ -168,25 +178,25 @@ Legacy configuration: `enableChangeDataCapture` Used when: -* `Protocol` is requested for the [requiredMinimumProtocol](Protocol.md#requiredMinimumProtocol) -* `DeleteCommand` is requested to [rewriteFiles](commands/delete/DeleteCommand.md#rewriteFiles) -* `MergeIntoCommand` is requested to [writeAllChanges](commands/merge/MergeIntoCommand.md#writeAllChanges) -* `UpdateCommand` is requested to [shouldOutputCdc](commands/update/UpdateCommand.md#shouldOutputCdc) -* `CDCReader` is requested to [isCDCEnabledOnTable](change-data-feed/CDCReader.md#isCDCEnabledOnTable) +* `Protocol` is requested for the [requiredMinimumProtocol](../Protocol.md#requiredMinimumProtocol) +* `DeleteCommand` is requested to [rewriteFiles](../commands/delete/DeleteCommand.md#rewriteFiles) +* `MergeIntoCommand` is requested to [writeAllChanges](../commands/merge/MergeIntoCommand.md#writeAllChanges) +* `UpdateCommand` is requested to [shouldOutputCdc](../commands/update/UpdateCommand.md#shouldOutputCdc) +* `CDCReader` is requested to [isCDCEnabledOnTable](../change-data-feed/CDCReader.md#isCDCEnabledOnTable) ### enableDeletionVectors { #enableDeletionVectors } **delta.enableDeletionVectors** -Enables [Deletion Vectors](deletion-vectors/index.md) +Enables [Deletion Vectors](../deletion-vectors/index.md) Default: `false` Used when: -* `DeletionVectorsTableFeature` is requested to [metadataRequiresFeatureToBeEnabled](deletion-vectors/DeletionVectorsTableFeature.md#metadataRequiresFeatureToBeEnabled) -* `Protocol` is requested to [assertTablePropertyConstraintsSatisfied](Protocol.md#assertTablePropertyConstraintsSatisfied) -* `DeletionVectorUtils` is requested to [deletionVectorsWritable](deletion-vectors/DeletionVectorUtils.md#deletionVectorsWritable) +* `DeletionVectorsTableFeature` is requested to [metadataRequiresFeatureToBeEnabled](../deletion-vectors/DeletionVectorsTableFeature.md#metadataRequiresFeatureToBeEnabled) +* `Protocol` is requested to [assertTablePropertyConstraintsSatisfied](../Protocol.md#assertTablePropertyConstraintsSatisfied) +* `DeletionVectorUtils` is requested to [deletionVectorsWritable](../deletion-vectors/DeletionVectorUtils.md#deletionVectorsWritable) ### enableExpiredLogCleanup { #enableExpiredLogCleanup } @@ -212,10 +222,10 @@ Default: `false` Used when: -* `DeltaErrorsBase` is requested to [convertToDeltaRowTrackingEnabledWithoutStatsCollection](DeltaErrorsBase.md#convertToDeltaRowTrackingEnabledWithoutStatsCollection) -* `RowId` is requested to [isEnabled](row-tracking/RowId.md#isEnabled) -* `RowTracking` is requested to [isEnabled](row-tracking/RowTracking.md#isEnabled) -* `RowTrackingFeature` is requested to [metadataRequiresFeatureToBeEnabled](row-tracking/RowTrackingFeature.md#metadataRequiresFeatureToBeEnabled) +* `DeltaErrorsBase` is requested to [convertToDeltaRowTrackingEnabledWithoutStatsCollection](../DeltaErrorsBase.md#convertToDeltaRowTrackingEnabledWithoutStatsCollection) +* `RowId` is requested to [isEnabled](../row-tracking/RowId.md#isEnabled) +* `RowTracking` is requested to [isEnabled](../row-tracking/RowTracking.md#isEnabled) +* `RowTrackingFeature` is requested to [metadataRequiresFeatureToBeEnabled](../row-tracking/RowTrackingFeature.md#metadataRequiresFeatureToBeEnabled) ### logRetentionDuration { #logRetentionDuration } @@ -229,7 +239,7 @@ Examples: `2 weeks`, `365 days` (`months` and `years` are not accepted) Used when: -* `MetadataCleanup` is requested for the [deltaRetentionMillis](MetadataCleanup.md#deltaRetentionMillis) +* `MetadataCleanup` is requested for the [deltaRetentionMillis](../MetadataCleanup.md#deltaRetentionMillis) ### minReaderVersion { #minReaderVersion } @@ -316,13 +326,13 @@ mergeGlobalConfigs( protocol: Protocol): Map[String, String] ``` -`mergeGlobalConfigs` finds all [spark.databricks.delta.properties.defaults](#sqlConfPrefix)-prefixed configuration properties among the [entries](#entries). +`mergeGlobalConfigs` finds all [spark.databricks.delta.properties.defaults](#sqlConfPrefix)-prefixed table properties among the [entries](#entries). --- `mergeGlobalConfigs` is used when: -* `OptimisticTransactionImpl` is requested to [withGlobalConfigDefaults](OptimisticTransactionImpl.md#withGlobalConfigDefaults) +* `OptimisticTransactionImpl` is requested to [withGlobalConfigDefaults](../OptimisticTransactionImpl.md#withGlobalConfigDefaults) * `InitialSnapshot` is created ## validateConfigurations { #validateConfigurations } @@ -338,9 +348,9 @@ validateConfigurations( `validateConfigurations` is used when: -* `DeltaCatalog` is requested to [verifyTableAndSolidify](DeltaCatalog.md#verifyTableAndSolidify), [alterTable](DeltaCatalog.md#alterTable) -* `CloneTableBase` is requested to [runInternal](commands/clone/CloneTableBase.md#runInternal) -* `DeltaDataSource` is requested to [create a BaseRelation](spark-connector/DeltaDataSource.md#createRelation) +* `DeltaCatalog` is requested to [verifyTableAndSolidify](../DeltaCatalog.md#verifyTableAndSolidify), [alterTable](../DeltaCatalog.md#alterTable) +* `CloneTableBase` is requested to [runInternal](../commands/clone/CloneTableBase.md#runInternal) +* `DeltaDataSource` is requested to [create a BaseRelation](../spark-connector/DeltaDataSource.md#createRelation) ## normalizeConfigKeys { #normalizeConfigKeys } @@ -355,8 +365,4 @@ normalizeConfigKeys( `normalizeConfigKeys` is used when: -* [AlterTableUnsetPropertiesDeltaCommand](commands/alter/AlterTableUnsetPropertiesDeltaCommand.md) is executed - -## spark.databricks.delta.properties.defaults Prefix { #sqlConfPrefix } - -**spark.databricks.delta.properties.defaults** prefix is used for [global configuration properties](#mergeGlobalConfigs). +* [AlterTableUnsetPropertiesDeltaCommand](../commands/alter/AlterTableUnsetPropertiesDeltaCommand.md) is executed diff --git a/docs/table-properties/index.md b/docs/table-properties/index.md index e24786d1b5..31d126428e 100644 --- a/docs/table-properties/index.md +++ b/docs/table-properties/index.md @@ -2,7 +2,7 @@ **Table Properties** are configuration properties that are used on table basis to customize behaviour of delta tables. -[DeltaConfigs](../DeltaConfigs.md) holds the supported table properties. +[DeltaConfigs](../table-properties/DeltaConfigs.md) holds the supported table properties. Table Properties use `delta.` prefix. diff --git a/mkdocs.yml b/mkdocs.yml index eb767f3e23..343f054055 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -190,8 +190,6 @@ nav: - SetTransaction: SetTransaction.md - SingleAction: SingleAction.md - DelayedCommitProtocol.md - - DeltaConfig: DeltaConfig.md - - DeltaConfigs: DeltaConfigs.md - DeltaErrorsBase.md - DeltaTableIdentifier: DeltaTableIdentifier.md - FileFormat: