From 7235275dc3879526ea6e194a9555320f8f5fc3c5 Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 13 Apr 2023 22:01:10 +0800 Subject: [PATCH 01/42] add v7.1.0 release notes Signed-off-by: Aolin --- releases/release-7.1.0.md | 358 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 358 insertions(+) create mode 100644 releases/release-7.1.0.md diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md new file mode 100644 index 0000000000000..5ea016bc34053 --- /dev/null +++ b/releases/release-7.1.0.md @@ -0,0 +1,358 @@ +--- +title: TiDB 7.1.0 Release Notes +summary: Learn about the new features, compatibility changes, improvements, and bug fixes in TiDB 7.1.0. +--- + +# TiDB 7.1.0 Release Notes + +TiDB version: 7.1.0-[DMR](/releases/versioning.md#development-milestone-releases) + +Quick access: [Quick start](https://docs.pingcap.com/tidb/v7.0/quick-start-with-tidb) | [Installation package](https://www.pingcap.com/download/?version=v7.0.0#version-list) + +In v7.1.0-LTS, the key new features and improvements are as follows: + +## Feature details + +### Scalability + +* TiFlash 支持存算分离和对象存储 (GA) [#6882](https://github.com/pingcap/tiflash/issues/6882) @[flowbehappy](https://github.com/flowbehappy) @[JaySon-Huang](https://github.com/JaySon-Huang) @[breezewish](https://github.com/breezewish) @[JinheLin](https://github.com/JinheLin) @[lidezhu](https://github.com/lidezhu) @[CalvinNeo](https://github.com/CalvinNeo) **tw:qiancai** + + 在 v7.0.0 版本中，TiFlash 在已有的存算一体架构之外，新增存算分离架构。在此架构下，TiFlash 节点分为 Compute Node（计算节点）和 Write Node（写入节点）两种类型，并支持兼容 S3 API 的对象存储。这两种节点都可以单独扩缩容，独立调整计算或数据存储能力。 + + 从 v7.1.0 版本开始，TiFlash 存算分离架构正式 GA。TiFlash 的存算分离架构和存算一体架构不能混合使用、相互转换，需要在部署 TiFlash 时进行相应的配置指定使用其中的一种架构。 + + 更多信息，请参考[用户文档](/tiflash/tiflash-disaggregated-and-s3.md)。 + +### Performance + +* 下一代 [`Partitioned Raft KV`](/partitioned-raft-kv.md) 存储引擎 GA [#issue号](链接) @[busyjay](https://github.com/busyjay) @[tonyxuqqi](https://github.com/tonyxuqqi) @[tabokie](https://github.com/tabokie) @[bufferflies](https://github.com/bufferflies) @[5kbpers](https://github.com/5kbpers) @[SpadeA-Tang](https://github.com/SpadeA-Tang) @[nolouch](https://github.com/nolouch) **tw:Oreoxmt** + + TiDB v6.6.0 引入的全新的 TiKV 存储引擎 [`Partitioned Raft KV`](/partitioned-raft-kv.md) 在 TiDB v7.1.0 版本正式 GA。该引擎使用多个 RocksDB 实例存储 TiKV 的 Region 数据，为每个 Region 提供独立的 RocksDB 实例。此外，该引擎能够更好地管理 RocksDB 实例的文件数和层级，实现 Region 间的数据操作物理隔离，并支持更多数据的平滑扩展。与原 TiKV 存储引擎相比，使用该引擎在相同硬件条件和读写混合场景下，可实现约 2 倍的写入吞吐、3 倍的读取吞吐，并缩短约 4/5 的弹性伸缩时间。该引擎与 TiFlash 引擎兼容，支持 Lightning / BR / TiCDC 等周边工具。该引擎目前仅支持在新集群中使用，暂不支持从原 TiKV 存储引擎直接升级到该引擎。 + + 更多信息，请参考[用户文档](/partitioned-raft-kv.md)。 + +* TiFlash 查询支持延迟物化功能 (GA) [#5829](https://github.com/pingcap/tiflash/issues/5829) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) **tw:qiancai** + + 在 v7.0.0 中，TiFlash 引入了延迟物化实验特性，用于优化查询性能。该特性默认关闭（系统变量 [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-从-v700-版本开始引入) 默认为 `OFF`）。当 `SELECT` 语句中包含过滤条件（`WHERE` 子句）时，TiFlash 默认会先读取该查询所需列的全部数据，然后再根据查询条件对数据进行过滤、聚合等计算任务。开启该特性后，TiFlash 支持下推部分过滤条件到 TableScan 算子，即先扫描过滤条件相关的列数据，过滤得到符合条件的行后，再扫描这些行的其他列数据，继续后续计算，从而减少 IO 扫描和数据处理的计算量。 + + 从 v7.1.0 开始，TiFlash 延迟物化功能正式 GA，默认开启（系统变量 [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-从-v700-版本开始引入) 默认为 `ON`），TiDB 优化器会根据统计信息和查询的过滤条件，决定哪些过滤条件会被下推到 TableScan 算子。 + + 更多信息，请参考[用户文档](/tiflash/tiflash-late-materialization.md)。 + +* TiFlash 支持根据网络交换数据量自动选择 MPP 模式的 Join 算法 [#7084](https://github.com/pingcap/tiflash/issues/7084) @[solotzg](https://github.com/solotzg) **tw:qiancai** + + TiFlash MPP 模式有多种 Join 算法。在 v7.1.0 之前的版本中，TiDB 根据变量 [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-从-v50-版本开始引入) 和 [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-从-v50-版本开始引入) 以及实际数据量决定 TiFlash MPP 模式是否使用 Broadcast Hash Join 算法。 + + 在 v7.1.0 中，TiDB 引入变量 [`tidb_prefer_broadcast_join_by_exchange_data_size`](/system-variables.md#tidb_prefer_broadcast_join_by_exchange_data_size-从-v710-版本开始引入)，控制是否基于最小网络数据交换策略选择 MPP Join 算法。该变量默认关闭，表示默认保持 v7.1.0 之前的算法选择策略。如需开启，请设置该变量为 `ON`，TiDB 会自动估算并考虑多种 Join 算法所需进行网络交换的数据量，然后选择综合开销较小的算法，从而减少网络流量，提升 MPP 查询性能。开启后，[`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-从-v50-版本开始引入) 和 [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-从-v50-版本开始引入) 将不再生效。 + + 更多信息，请参考[用户文档](/tiflash/use-tiflash-mpp-mode.md#mpp-模式的算法支持)。 + +* 自适应副本读来缓解读热点 [#14151](https://github.com/tikv/tikv/issues/14151) @[sticnarf](https://github.com/sticnarf) @[you06](https://github.com/you06) **tw:Oreoxmt** + + 发生读热点场景，其他 TiKV 节点可能仍存在闲置资源，与其在数据主节点持续排队等待，转而从其他节点读取副本可能带来更低的延迟。TiDB 在新版本开始支持负载自适应副本读，通过 [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-从-v700-版本开始引入) 参数来设置排队时间的临界值，当估算的排队时间超过设定时，TiDB 会尝试从副本节点读取数据。在读热点的情况下，相比于不打散有 70%-200% 的读取吞吐量提升。 + + 更多信息，请参考[用户文档](/troubleshoot-hot-spot-issues.md#打散读热点)。 + +* 非 Prepare 语句的执行计划缓存 GA [#36598](https://github.com/pingcap/tidb/issues/36598) @[qw4990](https://github.com/qw4990) **tw:Oreoxmt** + + TiDB 在 v7.0.0 支持缓存非 Prepare 语句的执行计划，以提升在线交易场景的并发能力。v7.1.0 持续优化了这个能力，支持更多模式的 SQL 进入缓存，并正式将这个能力 GA。 + + GA 之后，非 Prepare 与 Prepare SQL 的缓存池合并，以提升内存利用率，缓存大小通过变量 [`tidb_session_plan_cache_size`](/system-variables.md#tidb_session_plan_cache_size) 设置。原有的变量 [`tidb_prepared_plan_cache_size`](/system-variables.md#tidb_prepared_plan_cache_size) 和 [`tidb_non_prepared_plan_cache_size`](/system-variables.md#tidb_non_prepared_plan_cache_size) 将被废弃。 + + 为了保持设置向前兼容，对于从低版本升级到 v7.1.0 的客户，缓存池大小 `tidb_session_plan_cache_size` 将会继承 `tidb_prepared_plan_cache_size`的设置，非 Parepare 语句的缓存保持关闭。经过性能测试后，用户可通过 [`tidb_enable_non_prepared_plan_cache`](/system-variables.md#tidb_enable_non_prepared_plan_cache) 打开。对于新部署的客户，非 Parepare 语句的缓存则默认打开。 + + 更多信息，请参考[用户文档](/sql-non-prepared-plan-cache.md)。 + +### Reliability + +* 资源管控 GA [#38825](https://github.com/pingcap/tidb/issues/38825) @[nolouch](https://github.com/nolouch) @[BornChanger](https://github.com/BornChanger) @[glorv](https://github.com/glorv) @[tiancaiamao](https://github.com/tiancaiamao) @[Connor1996](https://github.com/Connor1996) @[JmPotato](https://github.com/JmPotato) @[hnes](https://github.com/hnes) @[CabinfeverB](https://github.com/CabinfeverB) @[HuSharp](https://github.com/HuSharp) **tw:hfxsd** + + TiDB 持续增强资源管控能力，并将这个特性 GA。该特性将会极大地提升 TiDB 集群的资源利用效率和性能表现。资源管控特性的引入对 TiDB 具有里程碑的意义，你可以将一个分布式数据库集群划分成多个逻辑单元，将不同的数据库用户映射到对应的资源组中，并根据需要设置每个资源组的配额。当集群资源紧张时，来自同一个资源组的会话所使用的全部资源将被限制在配额内，避免其中一个资源组过度消耗，从而影响其他资源组中的会话正常运行。 + + 该特性也可以将多个来自不同系统的中小型应用合入一个 TiDB 集群中，个别应用的负载提升，不会影响其他应用的正常运行。而在系统负载较低的时候，繁忙的应用即使超过设定的读写配额，也仍然可以被分配到所需的系统资源，达到资源的最大化利用。此外，合理利用资源管控特性可以减少集群数量，降低运维难度及管理成本。 + + 在 v7.1.0 中，TiDB 增加了基于实际负载来估算系统容量上限的能力，为客户的容量规划提供了更准确的参考，协助客户更好地管理 TiDB 的资源分配，从而满足企业级场景的稳定性需要。 + + 更多信息，请参考[用户文档](/tidb-resource-control.md)。 + +* 支持 [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-从-v630-版本开始引入) 的检查点机制，提升容错性和自动恢复能力 [#issue](https://github.com/pingcap/tidb/issues/issue) @[tangenta](https://github.com/tangenta) **tw:ran-huang** + + TiDB v7.1.0 版本引入 [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-从-v630-版本开始引入) 的检查点机制，可以大幅提升 [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-从-v630-版本开始引入) 的容错性和自动恢复能力。即使在 TiDB DDL Owner 切换的情况下，也能够通过周期性记录并同步 DDL 进度，让新的 TiDB DDL Owner 仍能以 [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-从-v630-版本开始引入) 方式执行切换前的 DDL 语句，无需手动取消和重新执行 DDL 语句，从而让 DDL 执行更加稳定高效。 + + 更多信息，请参考[用户文档](/ddl-introduction.md)。 + +* BR 备份恢复工具支持断点恢复 [#issue](https://github.com/pingcap/tidb/issues/issue) @[Leavrth](https://github.com/Leavrth) **tw:Oreoxmt** + + 如果用户的 TiDB 集群规模较大，之前在进行数据库的快照恢复或日志恢复时，可能会出现一些意外情况导致恢复过程提前结束，例如硬盘空间占满、节点宕机等等。在 TiDB v7.1.0 之前的版本中，这些意外情况会导致之前恢复的进度作废，需要重新进行恢复，给用户带来大量额外成本和麻烦。 + + 为了解决这个问题，TiDB v7.1.0 引入了备份恢复的断点恢复功能。该功能可以在意外中断后保留上一次恢复的大部分进度，使得用户能够尽可能地继续上一次的恢复的进度，避免不必要的成本和麻烦。 + + 更多信息，请参考[用户文档](/br/br-checkpoint-restore.md)。 + +* 统计信息缓存加载策略优化 [#issue](https://github.com/pingcap/tidb/issues/issue) @[xuyifangreeneyes](https://github.com/xuyifangreeneyes) **tw:hfxsd** + + 在开启[统计信息同步加载](/statistics.md#统计信息的加载)的前提下，TiDB 大幅减少了启动时必须载入的统计信息的数量，并且在加载完成前不接受用户连接。一方面提升了启动时统计信息的加载速度，另一方面，避免了在启动初始阶段由于统计信息不全而引起的性能回退。提升了 TiDB 在复杂运行环境下的稳定性，降低了个别 TiDB 节点重启对整体服务的影响。 + + 更多信息，请参考[用户文档]()。 + +* 优化统计信息收集的对象 [#issue](https://github.com/pingcap/tidb/issues/issue) @[xuyifangreeneyes](https://github.com/xuyifangreeneyes) **tw:hfxsd** + + 在 v7.1.0 中，TiDB 在收集优化器统计信息时，将会忽略一部分特殊的组件和列类型，从而加快了统计信息收集的速度，减少统计信息收集所消耗的系统资源。在一些特定场景，避免了统计信息收集造成的系统稳定性问题。 + + 更多信息，请参考[用户文档]()。 + +### Availability + +### SQL + +* 支持通过 `INSERT INTO SELECT` 语句保存 TiFlash 查询结果 (GA) [#37515](https://github.com/pingcap/tidb/issues/37515) @[gengliqi](https://github.com/gengliqi) **tw:qiancai** + + 从 v6.5.0 起，TiDB 支持下推 `INSERT INTO SELECT` 语句中的 `SELECT` 子句（分析查询）到 TiFlash，你可以将 TiFlash 的查询结果方便地保存到 `INSERT INTO` 指定的 TiDB 表中供后续分析使用，起到了结果缓存（即结果物化）的效果。 + + 在 v7.1.0 版本中，该功能正式 GA。当 TiDB 执行 `INSERT INTO SELECT` 语句中的 `SELECT` 子句时，优化器将根据 SQL Mode 及 TiFlash 副本的代价估算自行决定是否将查询下推至 TiFlash。因此，在实验特性阶段引入的系统变量 `tidb_enable_tiflash_read_for_write_stmt` 将被移除。需要注意的是，TiFlash 对于 `INSERT INTO SELECT` 语句的计算规则不满足 `STRICT SQL Mode` 要求，因此只有当前会话的 SQL Mode 是除 `STRICT_TRANS_TABLES`, `STRICT_ALL_TABLES` 之外的值时，TiDB 才允许将 `INSERT INTO SELECT` 语句中的查询下推至 TiFlash。 + + 更多信息，请参考[用户文档](/tiflash/tiflash-results-materialization.md)。 + +* MySQL 兼容的多值索引 (Multi-Valued Index) GA [#39592](https://github.com/pingcap/tidb/issues/39592) @[xiongjiwei](https://github.com/xiongjiwei) @[qw4990](https://github.com/qw4990) @[YangKeao](https://github.com/YangKeao) **tw:ran-huang** + + 过滤 JSON 列中某个数组的值是一个常见的操作，但普通索引起不到加速作用。在数组上创建多值索引能够大幅提升过滤的性能。如果 JSON 列中的某个数组上存在多值索引，那么函数 `MEMBER OF()`、`JSON_CONTAINS()`、`JSON_OVERLAPS()` 的检索条件可以利用多值索引过滤，从而减少大量的 I/O 消耗，提升执行速度。 + + 在 v7.1.0 中，TiDB 多值索引 (Multi-Valued Index) GA，支持更完整的数据类型，并与 TiDB 的工具链兼容。用户可以在生产环境利用“多值索引”加速对 JSON 数组的检索操作。 + + 更多信息，请参考[用户文档](/sql-statements/sql-statement-create-index.md#多值索引) + +* 支持完善的分区管理 [#42728](https://github.com/pingcap/tidb/issues/42728) @[mjonss](https://github.com/mjonss) **tw:qiancai** + + 在 v7.1.0 版本之前，TiDB 支持的分区类型包括 `RANGE`、`LIST`、`HASH`、`KEY` ，其中 `RANGE` 和 `LIST` 分区支持分区管理功能。从 v7.1.0 版本开始，TiDB 新增 `HASH` 和 `KEY` 分区的 `ADD PARTITION` 和 `COALESCE PARTITION` 管理功能，并支持修改分区表类型（包括分区表和非分表之间的相互转换、不同分区类型之间的修改），整体完善了分表表的分区管理能力。你可以根据需要，灵活地对表的分区方式进行调整。 + + 更多信息，请参考[用户文档](/partitioned-table.md#)。 + +* `LOAD DATA` SQL 支持从 S3、GCS 导入数据，支持任务管理等功能 GA [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** + + 以下 `LOAD DATA` 新增的功能在 7.1 版本 GA： + + - 支持从 S3、GCS 导入数据 + - 支持导入 Parquet 文件数据 + - 支持解析源文件中 ascii、latin1、binary、gbk、utf8mbd 字符集 + - 支持设置 FIELDS DEFINED NULL BY 将源文件的指定的值转换为 Null 写入目标表。 + - 支持设置一个 bath_size 即 1 个 batch 插入到目标表的行数，提升写入性能。 + - 支持设置 detached，允许该 job 在后台运行。 + - 支持 show load data jobs, show load data jobid, drop load data jobid 来管理任务。 + + 更多信息，请参考[用户文档](https://github.com/pingcap/docs-cn/pull/13344)。 + +* `LOAD DATA` SQL 集成 Lightning local backend（physical import mode） 的导入功能，提升导入性能（实验特性）[#42930](https://github.com/pingcap/tidb/issues/42930) @[D3Hunter](https://github.com/D3Hunter) **tw:hfxsd** + + 用户通过 `LOAD DATA` SQL 导入数据时，可以指定 import_mode = physical 来实现 Lightning local backend （physical 导入模式）的导入效果，相比 Load data 原先的 logical 导入模式，可成倍提升导入数据的性能。 + + 更多信息，请参考[用户文档](链接)。 + +* `LOAD DATA` SQL 支持并行导入，提升导入性能（实验特性）[#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** + + 原先 load data sql 无法并行导入数据，性能较差。在该版本中支持设置并行导入的参数，通过提升并发，来提升导入的性能。在实验室环境，相比上个版本，测试逻辑导入性能有接近 4 倍的提升。 + + 更多信息，请参考[用户文档](https://github.com/pingcap/docs-cn/pull/13676)。 + +* 生成列 (Generated Columns) GA [#issue号](链接) @[bb7133](https://github.com/bb7133) **tw:ran-huang** + + 生成列 (Generated Columns) 是 MySQL 数据库很有价值的一个功能。在创建表时定义一列的值由表中其他列的值计算而来，而不是由用户显式插入或更新，它可以是虚拟列 (Virtual Column) 或存储列(Stored Column)。TiDB 在早期版本就提供了 MySQL 兼容的生成列，在 v7.1.0 中将这个功能 GA。 + + 生成列提升了 TiDB 对 MySQL 的兼容性，方便用户从 MySQL 平滑迁移，同时也简化数据维护的复杂度、增强数据的一致性、提高查询效率。 + + 更多信息，请参考[用户文档](/generated-columns.md)。 + +### DB operations + +* DDL 任务支持暂停和恢复操作 [#issue号](链接) @[godouxm](https://github.com/godouxm) **tw:ran-huang** + + TiDB v7.1.0 之前的版本中，当 DDL 任务执行期间遇到业务高峰时间点时，为了减少对业务的影响，用户只能手动取消 DDL 任务。为了减轻 DDL 操作对业务负载的影响，TiDB v7.1.0 引入了 DDL 任务的暂停和恢复操作，用户可以在高峰时间点暂停 DDL 任务，等到业务高峰时间结束后再恢复 DDL 任务，从而避免了对业务的影响。 + + 例如，可以通过如下 [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) 或 [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md) 子句暂停或者恢复多个 DDL 任务： + + ```sql + SQL> admin pause ddl jobs 1,2; + + SQL> admin resume ddl jobs 1,2; + ``` + + 更多信息，请参考 [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) 和 [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md)。 + +* 无需取消 DDL 的平滑集群升级功能 [#issue号](链接) @[zimulala](https://github.com/zimulala) @[hawkingrei](https://github.com/hawkingrei) **tw:ran-huang** + + TiDB v7.1.0 之前的版本在升级集群时需要先手动取消正在运行或排队的 DDL 任务，在升级完毕后再手动添加这些任务。为了让用户享受更加平滑的升级体验，TiDB v7.1.0 引入了自动暂停和恢复 DDL 任务的功能。从 v7.1.0 版本开始，用户可以在不需要手动取消 DDL 任务的情况下升级集群。系统会自动暂停正在执行或排队的 DDL 任务，等待整个集群完成滚动升级后再自动恢复这些任务，让用户可以更加轻松地升级 TiDB 集群。 + +### Data migration + +* 增加优化器诊断信息 [#issue号](链接) @[time-and-fate](https://github.com/time-and-fate) **tw:hfxsd** + + 获取充足的信息是 SQL 性能诊断的关键，在 v7.1.0 中，TiDB 持续向各种诊断工具中添加优化器运行信息，可以更好地解释执行计划如何被选择，协助用户和技术支持对 SQL 性能问题进行定位。其中包括： + + * [`PLAN REPLAYER`](/sql-plan-replayer.md#使用-plan-replayer-保存和恢复集群现场信息) 的输出中增加 `debug_trace.json` 文件。 + * [`EXPLAIN`](/explain-walkthrough.md) 的输出中，为 `operator info` 添加部分统计信息详情。 + * 为[`慢日志`](/identify-slow-queries.md)的 `Stats` 字段添加部分统计信息详情。 + + 更多信息，请参考[使用 `PLAN REPLAYER` 保存和恢复集群线程信息](/sql-plan-replayer.md#使用-plan-replayer-保存和恢复集群现场信息)，[使用 `EXPLAIN` 解读执行计划](/explain-walkthrough.md)和[`慢日志查询`](/identify-slow-queries.md)。 + +### 安全 + +* 更换 TiFlash 系统表信息的查询接口 [#6941](https://github.com/pingcap/tiflash/issues/6941) @[flowbehappy](https://github.com/flowbehappy) **tw:qiancai** + + 从 v7.1.0 起，TiFlash 在向 TiDB 提供 [`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information-schema-tiflash-tables.md) 和 [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md) 系统表查询服务的端口时，不再使用 HTTP 端口，而是使用 gRPC 端口，从而避免 HTTP 服务的安全风险。 + +### 数据迁移 + +* TiCDC 支持 E2E 单行数据正确性校验功能 [#issue号](链接) @[3AceShowHand](https://github.com/3AceShowHand) @[zyguan](https://github.com/zyguan) **tw:Oreoxmt** + + 从 v7.1.0 版本开始，TiCDC 新增了单行数据正确性校验功能，该功能基于 Checksum 算法对单行数据的正确性进行校验。通过该功能可以校验一行数据从 TiDB 写入、经由 TiCDC 流出，再写入到 Kafka 集群的过程中是否发生了数据错误。该功能仅支持下游是 Kafka Sink 的 Changefeed，支持 Canal-JSON / Avro / Open-Protocol 等协议。 + + 更多信息，请参考[用户文档](/ticdc/ticdc-integrity-check.md)。 + +* TiCDC 优化 DDL 同步操作 [#8686](https://github.com/pingcap/tiflow/issues/8686) @[nongfushanquan](https://github.com/nongfushanquan) **tw:ran-huang** + + 在 v7.1.0 版本之前，当用户在一个大表上运行需要影响所有行的 DDL 操作，例如添加 / 删除列时，TiCDC 的同步延迟会显著增加。从 v7.1.0 版本开始，TiCDC 对于这种情况进行了优化，将同步延迟降低到不到 10 秒，以减轻 DDL 操作对下游延迟的影响。 + + 更多信息，请参考[用户文档](链接)。 + +* Lightning local backend (physical import mode) 支持在导入数据之前检测冲突的记录，并支持通过 insert ignore 和 replace 解决导入过程中的冲突记录 (实验特性) [#41629](https://github.com/pingcap/tidb/issues/41629) @[gozssky](https://github.com/gozssky) **tw:hfxsd** + + 在之前的版本使用 Lightning local backend (physical import mode) 导入数据时，当遇到冲突的记录时，无法通过 insert ignore 和 replace 来处理导入过程中的 pk、uk 冲突记录，需要用户自行去重。而本次版本，支持在导入数据之前，检查冲突的记录，并通过 replace 和 insert ignore 的语义来处理 pk、uk 冲突的记录。简化用户的操作，提升处理冲突的性能。 + + 更多信息，请参考[用户文档](链接)。 + +## Compatibility changes + +> **Note:** +> +> This section provides compatibility changes you need to know when you upgrade from v7.0.0 to the current version (v7.1.0). If you are upgrading from v6.6.0 or earlier versions to the current version, you might also need to check the compatibility changes introduced in intermediate versions. + +### MySQL compatibility + +### Behavior changes + +* 为了提高安全性，TiFlash 废弃了 HTTP 服务端口（默认 `8123`），采用 gRPC 端口作为替代 **tw:qiancai** + + 如果你已经将 TiFlash 升级到 v7.1.0，那么在升级 TiDB 到 v7.1.0 的过程中，TiFlash 系统表（[`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information-schema-tiflash-tables.md) 和 [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)）不可读。 + +### System variables + +| Variable name | Change type | Description | +|--------|------------------------------|------| +| | | | +| | | | +| | | | +| | | | + +### Configuration file parameters + +| Configuration file | Configuration parameter | Change type | Description | +| -------- | -------- | -------- | -------- | +| TiFlash | `http_port` | 删除 | 废弃 TiFlash HTTP 服务端口（默认 `8123`）。| +| | | | | +| | | | | +| | | | | + +### Others + +## Deprecated feature + +## Improvements + ++ TiDB + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + ++ TiKV + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + ++ PD + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + ++ TiFlash + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + ++ Tools + + + Backup & Restore (BR) + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + + + TiCDC + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + + + TiDB Data Migration (DM) + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + + + TiDB Lightning + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + + + TiUP + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + +## Bug fixes + ++ TiDB + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + ++ TiKV + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + ++ PD + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + ++ TiFlash + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + ++ Tools + + + Backup & Restore (BR) + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + + + TiCDC + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + + + TiDB Data Migration (DM) + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + + + TiDB Lightning + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + + + TiUP + + - note [#issue](链接) @[贡献者 GitHub ID](链接) + - note [#issue](链接) @[贡献者 GitHub ID](链接) + +## Contributors + +We would like to thank the following contributors from the TiDB community: From 41b26a992a017ef496a08d4ddc0f51a9b4baa435 Mon Sep 17 00:00:00 2001 From: Aolin Date: Fri, 14 Apr 2023 17:02:17 +0800 Subject: [PATCH 02/42] remove note --- releases/release-7.1.0.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 5ea016bc34053..fefecb994e78c 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -95,12 +95,6 @@ In v7.1.0-LTS, the key new features and improvements are as follows: 更多信息，请参考[用户文档]()。 -* 优化统计信息收集的对象 [#issue](https://github.com/pingcap/tidb/issues/issue) @[xuyifangreeneyes](https://github.com/xuyifangreeneyes) **tw:hfxsd** - - 在 v7.1.0 中，TiDB 在收集优化器统计信息时，将会忽略一部分特殊的组件和列类型，从而加快了统计信息收集的速度，减少统计信息收集所消耗的系统资源。在一些特定场景，避免了统计信息收集造成的系统稳定性问题。 - - 更多信息，请参考[用户文档]()。 - ### Availability ### SQL From 335a116c8dd85368ac4c53a50e2e188542114922 Mon Sep 17 00:00:00 2001 From: Ran Date: Tue, 18 Apr 2023 15:21:33 +0800 Subject: [PATCH 03/42] Apply suggestions from code review --- releases/release-7.1.0.md | 47 +++++++++++++++++++-------------------- 1 file changed, 23 insertions(+), 24 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index fefecb994e78c..986e7c60698f0 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -75,11 +75,11 @@ In v7.1.0-LTS, the key new features and improvements are as follows: 更多信息，请参考[用户文档](/tidb-resource-control.md)。 -* 支持 [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-从-v630-版本开始引入) 的检查点机制，提升容错性和自动恢复能力 [#issue](https://github.com/pingcap/tidb/issues/issue) @[tangenta](https://github.com/tangenta) **tw:ran-huang** +* Support the checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-introduced-since-v630) to improve fault tolerance and automatic recovery capability [#issue](https://github.com/pingcap/tidb/issues/issue) @[tangenta](https://github.com/tangenta) - TiDB v7.1.0 版本引入 [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-从-v630-版本开始引入) 的检查点机制，可以大幅提升 [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-从-v630-版本开始引入) 的容错性和自动恢复能力。即使在 TiDB DDL Owner 切换的情况下，也能够通过周期性记录并同步 DDL 进度，让新的 TiDB DDL Owner 仍能以 [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-从-v630-版本开始引入) 方式执行切换前的 DDL 语句，无需手动取消和重新执行 DDL 语句，从而让 DDL 执行更加稳定高效。 + TiDB v7.1.0 introduces a checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-introduced-since-v630), whichs significantly improves the fault tolerance and automatic recovery capability of Fast Online DDL. Even in the case of TiDB DDL owner switching, TiDB can still periodically record and synchronize the progress of DDL statements, allowing the new TiDB DDL owner to execute the ongoing DDL statements in Fast Online DDL mode without manually canceling and re-executing the statements. The checkpoint mechanism makes the DDL execution more stable and efficient. - 更多信息，请参考[用户文档](/ddl-introduction.md)。 + For more information, see [documentation](/ddl-introduction.md). * BR 备份恢复工具支持断点恢复 [#issue](https://github.com/pingcap/tidb/issues/issue) @[Leavrth](https://github.com/Leavrth) **tw:Oreoxmt** @@ -107,13 +107,13 @@ In v7.1.0-LTS, the key new features and improvements are as follows: 更多信息，请参考[用户文档](/tiflash/tiflash-results-materialization.md)。 -* MySQL 兼容的多值索引 (Multi-Valued Index) GA [#39592](https://github.com/pingcap/tidb/issues/39592) @[xiongjiwei](https://github.com/xiongjiwei) @[qw4990](https://github.com/qw4990) @[YangKeao](https://github.com/YangKeao) **tw:ran-huang** +* MySQL-compatible multi-valued index becomes generally available (GA) [#39592](https://github.com/pingcap/tidb/issues/39592) @[xiongjiwei](https://github.com/xiongjiwei) @[qw4990](https://github.com/qw4990) @[YangKeao](https://github.com/YangKeao) - 过滤 JSON 列中某个数组的值是一个常见的操作，但普通索引起不到加速作用。在数组上创建多值索引能够大幅提升过滤的性能。如果 JSON 列中的某个数组上存在多值索引，那么函数 `MEMBER OF()`、`JSON_CONTAINS()`、`JSON_OVERLAPS()` 的检索条件可以利用多值索引过滤，从而减少大量的 I/O 消耗，提升执行速度。 + Filtering the values of an array in a JSON column is a common operation, but normal indexes cannot help speed up such an operation. Creating a multi-valued index on an array can greatly improve filtering performance. If an array in the JSON column has a multi-valued index, you can use the multi-value index to filter the retrieval conditions with MEMBER OF(), JSON_CONTAINS(), JSON_OVERLAPS() functions, thereby reducing much I/O consumption and improving operation speed. - 在 v7.1.0 中，TiDB 多值索引 (Multi-Valued Index) GA，支持更完整的数据类型，并与 TiDB 的工具链兼容。用户可以在生产环境利用“多值索引”加速对 JSON 数组的检索操作。 + In v7.1.0, the multi-valued index becomes a generally available (GA) feature. It supports more complete data types and is compatible with TiDB tools. You can use the multi-valued indexes to speed up the search operations on JSON arrays in production environments. - 更多信息，请参考[用户文档](/sql-statements/sql-statement-create-index.md#多值索引) + For more information, see [documentation](/sql-statements/sql-statement-create-index.md#multi-valued-index). * 支持完善的分区管理 [#42728](https://github.com/pingcap/tidb/issues/42728) @[mjonss](https://github.com/mjonss) **tw:qiancai** @@ -147,33 +147,34 @@ In v7.1.0-LTS, the key new features and improvements are as follows: 更多信息，请参考[用户文档](https://github.com/pingcap/docs-cn/pull/13676)。 -* 生成列 (Generated Columns) GA [#issue号](链接) @[bb7133](https://github.com/bb7133) **tw:ran-huang** +* Generated columns become generally available (GA) [#issue号](链接) @[bb7133](https://github.com/bb7133) **tw:ran-huang** - 生成列 (Generated Columns) 是 MySQL 数据库很有价值的一个功能。在创建表时定义一列的值由表中其他列的值计算而来，而不是由用户显式插入或更新，它可以是虚拟列 (Virtual Column) 或存储列(Stored Column)。TiDB 在早期版本就提供了 MySQL 兼容的生成列，在 v7.1.0 中将这个功能 GA。 + Generated columns are a very valuable feature in MySQL. When creating a table, you can define that the value of a column is calculated from the values of other columns in the table, rather than being explicitly inserted or updated by the user. This generated column can be either a virtual column or a stored column. TiDB has provided generated columns compatible with MySQL since earlier versions, and this feature is officially GA in v7.1.0. - 生成列提升了 TiDB 对 MySQL 的兼容性，方便用户从 MySQL 平滑迁移，同时也简化数据维护的复杂度、增强数据的一致性、提高查询效率。 + Using generated columns can improve TiDB's compatibility with MySQL, simplifying the process of migrating from MySQL. It also reduces data maintenance complexity, improves data consistency and query efficiency. - 更多信息，请参考[用户文档](/generated-columns.md)。 + For more information, see [documentation](/generated-columns.md). ### DB operations -* DDL 任务支持暂停和恢复操作 [#issue号](链接) @[godouxm](https://github.com/godouxm) **tw:ran-huang** +* DDL tasks support pause and resume operations [#issue number](link) @[godouxm](https://github.com/godouxm) **tw:ran-huang** - TiDB v7.1.0 之前的版本中，当 DDL 任务执行期间遇到业务高峰时间点时，为了减少对业务的影响，用户只能手动取消 DDL 任务。为了减轻 DDL 操作对业务负载的影响，TiDB v7.1.0 引入了 DDL 任务的暂停和恢复操作，用户可以在高峰时间点暂停 DDL 任务，等到业务高峰时间结束后再恢复 DDL 任务，从而避免了对业务的影响。 + Prior to TiDB v7.1.0, when a DDL task encountered a business peak period during execution, you could only manually cancel the DDL task to reduce its impact on the business. To mitigate the impact of DDL operations on application workloads, in v7.1.0, TiDB introduces pause and resume operations for DDL tasks. These operations let you pause DDL tasks during peak periods and resume them after the peak ends, thus avoiding any impact on your business. - 例如，可以通过如下 [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) 或 [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md) 子句暂停或者恢复多个 DDL 任务： + For example, you can pause and resume multiple DDL tasks using [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) or [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md): ```sql - SQL> admin pause ddl jobs 1,2; - - SQL> admin resume ddl jobs 1,2; + ADMIN PAUSE ddl jobs 1,2; + ADMIN RESUME ddl jobs 1,2; ``` - 更多信息，请参考 [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) 和 [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md)。 + For more information, see [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) and [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md). + +* Support smooth cluster upgrade without canceling DDL operations [#issue number](link) @[zimulala](https://github.com/zimulala) @[hawkingrei](https://github.com/hawkingrei) -* 无需取消 DDL 的平滑集群升级功能 [#issue号](链接) @[zimulala](https://github.com/zimulala) @[hawkingrei](https://github.com/hawkingrei) **tw:ran-huang** + Before TiDB v7.1.0, to upgrade a cluster, you must manually cancel the running or queued DDL tasks and then add them back after the upgrade is completed. - TiDB v7.1.0 之前的版本在升级集群时需要先手动取消正在运行或排队的 DDL 任务，在升级完毕后再手动添加这些任务。为了让用户享受更加平滑的升级体验，TiDB v7.1.0 引入了自动暂停和恢复 DDL 任务的功能。从 v7.1.0 版本开始，用户可以在不需要手动取消 DDL 任务的情况下升级集群。系统会自动暂停正在执行或排队的 DDL 任务，等待整个集群完成滚动升级后再自动恢复这些任务，让用户可以更加轻松地升级 TiDB 集群。 + To offer a smoother upgrade experience, TiDB v7.1.0 introduces automatic pausing and resuming of DDL tasks. Starting from v7.1.0, you can upgrade your clusters without manually canceling DDL tasks. TiDB will automatically pause any running or queued DDL tasks before the upgrade and resume these tasks after the cluster has completed rolling upgrade, making it easier for you to upgrade your TiDB clusters. ### Data migration @@ -201,11 +202,9 @@ In v7.1.0-LTS, the key new features and improvements are as follows: 更多信息，请参考[用户文档](/ticdc/ticdc-integrity-check.md)。 -* TiCDC 优化 DDL 同步操作 [#8686](https://github.com/pingcap/tiflow/issues/8686) @[nongfushanquan](https://github.com/nongfushanquan) **tw:ran-huang** +* TiCDC Optimizes DDL replication operations [#8686](https://github.com/pingcap/tiflow/issues/8686) @[nongfushanquan](https://github.com/nongfushanquan) - 在 v7.1.0 版本之前，当用户在一个大表上运行需要影响所有行的 DDL 操作，例如添加 / 删除列时，TiCDC 的同步延迟会显著增加。从 v7.1.0 版本开始，TiCDC 对于这种情况进行了优化，将同步延迟降低到不到 10 秒，以减轻 DDL 操作对下游延迟的影响。 - - 更多信息，请参考[用户文档](链接)。 + Prior to v7.1.0, when you performed a DDL operation that affected all rows on a large table (such as adding or deleting a column), the replication delay of TiCDC would significantly increase. Starting from v7.1.0, TiCDC optimizes this issue and reduces the replication delay to less than 10 seconds. This optimization mitigates the impact of DDL operations on downstream latency. * Lightning local backend (physical import mode) 支持在导入数据之前检测冲突的记录，并支持通过 insert ignore 和 replace 解决导入过程中的冲突记录 (实验特性) [#41629](https://github.com/pingcap/tidb/issues/41629) @[gozssky](https://github.com/gozssky) **tw:hfxsd** From a6c96ba3fcff5138e522e30542c1992a9a4ba118 Mon Sep 17 00:00:00 2001 From: qiancai Date: Tue, 18 Apr 2023 23:19:55 +0800 Subject: [PATCH 04/42] add translation for TiFlash late materialization --- releases/release-7.1.0.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 986e7c60698f0..e7c13a1f81dab 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -31,13 +31,13 @@ In v7.1.0-LTS, the key new features and improvements are as follows: 更多信息，请参考[用户文档](/partitioned-raft-kv.md)。 -* TiFlash 查询支持延迟物化功能 (GA) [#5829](https://github.com/pingcap/tiflash/issues/5829) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) **tw:qiancai** +* TiFlash supports late materialization (GA) [#5829](https://github.com/pingcap/tiflash/issues/5829) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) **tw:qiancai** - 在 v7.0.0 中，TiFlash 引入了延迟物化实验特性，用于优化查询性能。该特性默认关闭（系统变量 [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-从-v700-版本开始引入) 默认为 `OFF`）。当 `SELECT` 语句中包含过滤条件（`WHERE` 子句）时，TiFlash 默认会先读取该查询所需列的全部数据，然后再根据查询条件对数据进行过滤、聚合等计算任务。开启该特性后，TiFlash 支持下推部分过滤条件到 TableScan 算子，即先扫描过滤条件相关的列数据，过滤得到符合条件的行后，再扫描这些行的其他列数据，继续后续计算，从而减少 IO 扫描和数据处理的计算量。 + In v7.0.0, TiFlash supports late materialization as an experiment feature for optimizing query performance. This feature is disabled by default (the [`tidb_opt_enable_late_materialization`] system variable(/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) defaults to `OFF`). when processing a `SELECT` statement with filter conditions (`WHERE` clause), TiFlash reads all the data from the columns required by the query, and then filters and aggregates the data based on the query conditions. When Late materialization is enabled, TiDB supports pushing down part of the filter conditions to the TableScan operator. That is, TiFlash first scans the column data related to the filter conditions that are pushed down to the TableScan operator, filters the rows that meet the condition, and then scans the other column data of these rows for further calculation, thereby reducing IO scans and computations of data processing. - 从 v7.1.0 开始，TiFlash 延迟物化功能正式 GA，默认开启（系统变量 [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-从-v700-版本开始引入) 默认为 `ON`），TiDB 优化器会根据统计信息和查询的过滤条件，决定哪些过滤条件会被下推到 TableScan 算子。 + Starting from v7.1.0, the TiFlash late materialization feature is generally available and enabled by default (the [`tidb_opt_enable_late_materialization`] (/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `ON`). The TiDB optimizer decides which filters to be pushed down to the TableScan operator based on the statistics and the filter conditions of the query. - 更多信息，请参考[用户文档](/tiflash/tiflash-late-materialization.md)。 + For more information, see [documentation](/tiflash/tiflash-late-materialization.md)。 * TiFlash 支持根据网络交换数据量自动选择 MPP 模式的 Join 算法 [#7084](https://github.com/pingcap/tiflash/issues/7084) @[solotzg](https://github.com/solotzg) **tw:qiancai** From 4a2b4c6753d5f5ccba1511d94b0a0cefb9ae4c5d Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 19 Apr 2023 12:12:56 +0800 Subject: [PATCH 05/42] Remove Automatic Resume DDL Tasks After Rolling Upgrade Cluster --- releases/release-7.1.0.md | 6 ------ 1 file changed, 6 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index e7c13a1f81dab..5a791baefdb45 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -170,12 +170,6 @@ In v7.1.0-LTS, the key new features and improvements are as follows: For more information, see [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) and [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md). -* Support smooth cluster upgrade without canceling DDL operations [#issue number](link) @[zimulala](https://github.com/zimulala) @[hawkingrei](https://github.com/hawkingrei) - - Before TiDB v7.1.0, to upgrade a cluster, you must manually cancel the running or queued DDL tasks and then add them back after the upgrade is completed. - - To offer a smoother upgrade experience, TiDB v7.1.0 introduces automatic pausing and resuming of DDL tasks. Starting from v7.1.0, you can upgrade your clusters without manually canceling DDL tasks. TiDB will automatically pause any running or queued DDL tasks before the upgrade and resume these tasks after the cluster has completed rolling upgrade, making it easier for you to upgrade your TiDB clusters. - ### Data migration * 增加优化器诊断信息 [#issue号](链接) @[time-and-fate](https://github.com/time-and-fate) **tw:hfxsd** From bf1bef580476c45d99c3101f26051522ee99bcd1 Mon Sep 17 00:00:00 2001 From: qiancai Date: Wed, 19 Apr 2023 17:18:28 +0800 Subject: [PATCH 06/42] add translation for TiFlash related features --- releases/release-7.1.0.md | 30 +++++++++++++++--------------- 1 file changed, 15 insertions(+), 15 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 5a791baefdb45..9fcd8bcc5f7f0 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -39,13 +39,13 @@ In v7.1.0-LTS, the key new features and improvements are as follows: For more information, see [documentation](/tiflash/tiflash-late-materialization.md)。 -* TiFlash 支持根据网络交换数据量自动选择 MPP 模式的 Join 算法 [#7084](https://github.com/pingcap/tiflash/issues/7084) @[solotzg](https://github.com/solotzg) **tw:qiancai** +* TiFlash supports automatically choosing MPP Join algorithm according to the overhead of network transmission [#7084](https://github.com/pingcap/tiflash/issues/7084) @[solotzg](https://github.com/solotzg) **tw:qiancai** - TiFlash MPP 模式有多种 Join 算法。在 v7.1.0 之前的版本中，TiDB 根据变量 [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-从-v50-版本开始引入) 和 [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-从-v50-版本开始引入) 以及实际数据量决定 TiFlash MPP 模式是否使用 Broadcast Hash Join 算法。 + The TiFlash MPP mode supports multiple Join algorithms. Before v7.1.0, TiDB determines whether the MPP mode uses the Broadcast Hash Join algorithm based on the [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) and [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) variables and the actual overhead. - 在 v7.1.0 中，TiDB 引入变量 [`tidb_prefer_broadcast_join_by_exchange_data_size`](/system-variables.md#tidb_prefer_broadcast_join_by_exchange_data_size-从-v710-版本开始引入)，控制是否基于最小网络数据交换策略选择 MPP Join 算法。该变量默认关闭，表示默认保持 v7.1.0 之前的算法选择策略。如需开启，请设置该变量为 `ON`，TiDB 会自动估算并考虑多种 Join 算法所需进行网络交换的数据量，然后选择综合开销较小的算法，从而减少网络流量，提升 MPP 查询性能。开启后，[`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-从-v50-版本开始引入) 和 [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-从-v50-版本开始引入) 将不再生效。 + In v7.1.0, TiDB introduces the [`tidb_prefer_broadcast_join_by_exchange_data_size`](/system-variables.md#tidb_prefer_broadcast_join_by_exchange_data_size-new-in-v710) variable, which controls whether to choose the MPP Join algorithm based on the minimum overhead of network transmission. This variable is disabled by default, indicating that the default algorithm selection method remains the same as that before v7.1.0. You can set the variable to `ON` to enable it. When it is enabled, you no longer need to manually adjust the [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-introduced-from-v50-version) and [`tidb_ broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-introduced-from-v50-version) variables (both variables does not take effect at this time), TiDB automatically estimates the threshold of network transmission by different Join algorithms, and then chooses the algorithm with the smaller overhead overall, thus reducing network traffic and improving MPP query performance. - 更多信息，请参考[用户文档](/tiflash/use-tiflash-mpp-mode.md#mpp-模式的算法支持)。 + For more information, see [documentation](/tiflash/use-tiflash-mpp-mode.md#algorithm-support-for-the-mpp-mode). * 自适应副本读来缓解读热点 [#14151](https://github.com/tikv/tikv/issues/14151) @[sticnarf](https://github.com/sticnarf) @[you06](https://github.com/you06) **tw:Oreoxmt** @@ -99,13 +99,13 @@ In v7.1.0-LTS, the key new features and improvements are as follows: ### SQL -* 支持通过 `INSERT INTO SELECT` 语句保存 TiFlash 查询结果 (GA) [#37515](https://github.com/pingcap/tidb/issues/37515) @[gengliqi](https://github.com/gengliqi) **tw:qiancai** +* Support saving TiFlash query results using the `INSERT INTO SELECT` statement (GA) [#37515](https://github.com/pingcap/tidb/issues/37515) @[gengliqi](https://github.com/gengliqi) **tw:qiancai** - 从 v6.5.0 起，TiDB 支持下推 `INSERT INTO SELECT` 语句中的 `SELECT` 子句（分析查询）到 TiFlash，你可以将 TiFlash 的查询结果方便地保存到 `INSERT INTO` 指定的 TiDB 表中供后续分析使用，起到了结果缓存（即结果物化）的效果。 + Starting from v6.5.0, TiDB supports pushing down the `SELECT` clause (analytical query) of the `INSERT INTO SELECT` statement to TiFlash. In this way, you can easily save the TiFlash query result to a TiDB table specified by `INSERT INTO` for further analysis, which takes effect as result caching (that is, result materialization). - 在 v7.1.0 版本中，该功能正式 GA。当 TiDB 执行 `INSERT INTO SELECT` 语句中的 `SELECT` 子句时，优化器将根据 SQL Mode 及 TiFlash 副本的代价估算自行决定是否将查询下推至 TiFlash。因此，在实验特性阶段引入的系统变量 `tidb_enable_tiflash_read_for_write_stmt` 将被移除。需要注意的是，TiFlash 对于 `INSERT INTO SELECT` 语句的计算规则不满足 `STRICT SQL Mode` 要求，因此只有当前会话的 SQL Mode 是除 `STRICT_TRANS_TABLES`, `STRICT_ALL_TABLES` 之外的值时，TiDB 才允许将 `INSERT INTO SELECT` 语句中的查询下推至 TiFlash。 + In v7.1.0, this feature is generally available. During the execution of the `SELECT` clause in the `INSERT INTO SELECT` statement, the optimizer can intelligently decide whether to push a query down to TiFlash based on the [SQL mode](/sql-mode.md) and the cost estimates of the TiFlash replica. Therefore, the `tidb_enable_tiflash_read_for_write_stmt` system variable introduced during the experimental phase is now deprecated. Note that the computation rules of `INSERT INTO SELECT` statements for TiFlash do not meet the `STRICT SQL Mode` requirement, so TiDB allows the `SELECT` clause in the `INSERT INTO SELECT` statement to be pushed down to TiFlash only when the [SQL Mode](/sql-mode.md) of the current session is not strict, which means that the `sql_mode` value does not contain `STRICT_TRANS_TABLES` and `STRICT_ALL_TABLES`. - 更多信息，请参考[用户文档](/tiflash/tiflash-results-materialization.md)。 + For more information, see [documentation](/tiflash/tiflash-results-materialization.md)。 * MySQL-compatible multi-valued index becomes generally available (GA) [#39592](https://github.com/pingcap/tidb/issues/39592) @[xiongjiwei](https://github.com/xiongjiwei) @[qw4990](https://github.com/qw4990) @[YangKeao](https://github.com/YangKeao) @@ -115,11 +115,11 @@ In v7.1.0-LTS, the key new features and improvements are as follows: For more information, see [documentation](/sql-statements/sql-statement-create-index.md#multi-valued-index). -* 支持完善的分区管理 [#42728](https://github.com/pingcap/tidb/issues/42728) @[mjonss](https://github.com/mjonss) **tw:qiancai** +* Improve the partition management for Hash and Key partitioned tables [#42728](https://github.com/pingcap/tidb/issues/42728) @[mjonss](https://github.com/mjonss) **tw:qiancai** - 在 v7.1.0 版本之前，TiDB 支持的分区类型包括 `RANGE`、`LIST`、`HASH`、`KEY` ，其中 `RANGE` 和 `LIST` 分区支持分区管理功能。从 v7.1.0 版本开始，TiDB 新增 `HASH` 和 `KEY` 分区的 `ADD PARTITION` 和 `COALESCE PARTITION` 管理功能，并支持修改分区表类型（包括分区表和非分表之间的相互转换、不同分区类型之间的修改），整体完善了分表表的分区管理能力。你可以根据需要，灵活地对表的分区方式进行调整。 + Before v7.1.0, Hash and Key partitioned tables in TiDB only support the `TRUNCATE PARTITION` partition management statement. Starting from v7.1.0, Hash and Key partitioned tables also support `ADD PARTITION` and `COALESCE PARTITION` partition management statements. Therefore, you can flexibly adjust the number of partitions in Hash and Key partitioned tables as needed. For example, you can increase the number of partitions with the `ADD PARTITION` statement, or decrease the number of partitions with the `COALESCE PARTITION` statement. - 更多信息，请参考[用户文档](/partitioned-table.md#)。 + For more information, see [documentation](/partitioned-table.md#manage-hash-and-key-partitions). * `LOAD DATA` SQL 支持从 S3、GCS 导入数据，支持任务管理等功能 GA [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** @@ -184,9 +184,9 @@ In v7.1.0-LTS, the key new features and improvements are as follows: ### 安全 -* 更换 TiFlash 系统表信息的查询接口 [#6941](https://github.com/pingcap/tiflash/issues/6941) @[flowbehappy](https://github.com/flowbehappy) **tw:qiancai** +* Replace the interface used for querying TiFlash system table information [#6941](https://github.com/pingcap/tiflash/issues/6941) @[flowbehappy](https://github.com/flowbehappy) **tw:qiancai** - 从 v7.1.0 起，TiFlash 在向 TiDB 提供 [`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information-schema-tiflash-tables.md) 和 [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md) 系统表查询服务的端口时，不再使用 HTTP 端口，而是使用 gRPC 端口，从而避免 HTTP 服务的安全风险。 + Starting from v7.1.0, when providing the query service of [`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information-schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md) system tables for TiDB, TiFlash uses the gRPC port instead of the HTTP port, which avoids the security risks of the HTTP service. ### 数据迁移 @@ -216,9 +216,9 @@ In v7.1.0-LTS, the key new features and improvements are as follows: ### Behavior changes -* 为了提高安全性，TiFlash 废弃了 HTTP 服务端口（默认 `8123`），采用 gRPC 端口作为替代 **tw:qiancai** +* To improve security, TiFlash deprecates the HTTP service port (default `8123`) and uses the gRPC port as a replacement **tw:qiancai** - 如果你已经将 TiFlash 升级到 v7.1.0，那么在升级 TiDB 到 v7.1.0 的过程中，TiFlash 系统表（[`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information-schema-tiflash-tables.md) 和 [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)）不可读。 + If you have upgraded TiFlash to v7.1.0, then during the TiDB upgrade to v7.1.0, the TiFlash system tables ([`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information- schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)) are not readable. ### System variables From eb67c17946801983d7f92cea6e6f0f54a7c3308a Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 19 Apr 2023 17:24:39 +0800 Subject: [PATCH 07/42] Apply suggestions from code review --- releases/release-7.1.0.md | 9 +-------- 1 file changed, 1 insertion(+), 8 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 9fcd8bcc5f7f0..45da962dc655f 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -15,13 +15,6 @@ In v7.1.0-LTS, the key new features and improvements are as follows: ### Scalability -* TiFlash 支持存算分离和对象存储 (GA) [#6882](https://github.com/pingcap/tiflash/issues/6882) @[flowbehappy](https://github.com/flowbehappy) @[JaySon-Huang](https://github.com/JaySon-Huang) @[breezewish](https://github.com/breezewish) @[JinheLin](https://github.com/JinheLin) @[lidezhu](https://github.com/lidezhu) @[CalvinNeo](https://github.com/CalvinNeo) **tw:qiancai** - - 在 v7.0.0 版本中，TiFlash 在已有的存算一体架构之外，新增存算分离架构。在此架构下，TiFlash 节点分为 Compute Node（计算节点）和 Write Node（写入节点）两种类型，并支持兼容 S3 API 的对象存储。这两种节点都可以单独扩缩容，独立调整计算或数据存储能力。 - - 从 v7.1.0 版本开始，TiFlash 存算分离架构正式 GA。TiFlash 的存算分离架构和存算一体架构不能混合使用、相互转换，需要在部署 TiFlash 时进行相应的配置指定使用其中的一种架构。 - - 更多信息，请参考[用户文档](/tiflash/tiflash-disaggregated-and-s3.md)。 ### Performance @@ -261,7 +254,7 @@ In v7.1.0-LTS, the key new features and improvements are as follows: + TiFlash - - note [#issue](链接) @[贡献者 GitHub ID](链接) + - Improves TiFlash performance and stability in the disaggregated storage and compute architecture [#6882](https://github.com/pingcap/tiflash/issues/6882) @[JaySon-Huang](https://github.com/JaySon-Huang) @[breezewish](https://github.com/breezewish) @[JinheLin](https://github.com/JinheLin) **tw:qiancai** - note [#issue](链接) @[贡献者 GitHub ID](链接) + Tools From 78d88d85d98f1172c17bf2fbd77e63ae4e418594 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 19 Apr 2023 18:16:40 +0800 Subject: [PATCH 08/42] update links and translate new features Signed-off-by: Aolin --- releases/release-7.1.0.md | 79 ++++++++++++++++----------------------- 1 file changed, 33 insertions(+), 46 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 45da962dc655f..73ddae55431b5 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -5,32 +5,27 @@ summary: Learn about the new features, compatibility changes, improvements, and # TiDB 7.1.0 Release Notes -TiDB version: 7.1.0-[DMR](/releases/versioning.md#development-milestone-releases) - -Quick access: [Quick start](https://docs.pingcap.com/tidb/v7.0/quick-start-with-tidb) | [Installation package](https://www.pingcap.com/download/?version=v7.0.0#version-list) - -In v7.1.0-LTS, the key new features and improvements are as follows: +In v7.1.0, the key new features and improvements are as follows: ## Feature details ### Scalability - ### Performance * 下一代 [`Partitioned Raft KV`](/partitioned-raft-kv.md) 存储引擎 GA [#issue号](链接) @[busyjay](https://github.com/busyjay) @[tonyxuqqi](https://github.com/tonyxuqqi) @[tabokie](https://github.com/tabokie) @[bufferflies](https://github.com/bufferflies) @[5kbpers](https://github.com/5kbpers) @[SpadeA-Tang](https://github.com/SpadeA-Tang) @[nolouch](https://github.com/nolouch) **tw:Oreoxmt** TiDB v6.6.0 引入的全新的 TiKV 存储引擎 [`Partitioned Raft KV`](/partitioned-raft-kv.md) 在 TiDB v7.1.0 版本正式 GA。该引擎使用多个 RocksDB 实例存储 TiKV 的 Region 数据，为每个 Region 提供独立的 RocksDB 实例。此外，该引擎能够更好地管理 RocksDB 实例的文件数和层级，实现 Region 间的数据操作物理隔离，并支持更多数据的平滑扩展。与原 TiKV 存储引擎相比，使用该引擎在相同硬件条件和读写混合场景下，可实现约 2 倍的写入吞吐、3 倍的读取吞吐，并缩短约 4/5 的弹性伸缩时间。该引擎与 TiFlash 引擎兼容，支持 Lightning / BR / TiCDC 等周边工具。该引擎目前仅支持在新集群中使用，暂不支持从原 TiKV 存储引擎直接升级到该引擎。 - 更多信息，请参考[用户文档](/partitioned-raft-kv.md)。 + For more information, see [documentation](/partitioned-raft-kv.md). * TiFlash supports late materialization (GA) [#5829](https://github.com/pingcap/tiflash/issues/5829) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) **tw:qiancai** - In v7.0.0, TiFlash supports late materialization as an experiment feature for optimizing query performance. This feature is disabled by default (the [`tidb_opt_enable_late_materialization`] system variable(/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) defaults to `OFF`). when processing a `SELECT` statement with filter conditions (`WHERE` clause), TiFlash reads all the data from the columns required by the query, and then filters and aggregates the data based on the query conditions. When Late materialization is enabled, TiDB supports pushing down part of the filter conditions to the TableScan operator. That is, TiFlash first scans the column data related to the filter conditions that are pushed down to the TableScan operator, filters the rows that meet the condition, and then scans the other column data of these rows for further calculation, thereby reducing IO scans and computations of data processing. + In v7.0.0, TiFlash supports late materialization as an experiment feature for optimizing query performance. This feature is disabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `OFF`). when processing a `SELECT` statement with filter conditions (`WHERE` clause), TiFlash reads all the data from the columns required by the query, and then filters and aggregates the data based on the query conditions. When Late materialization is enabled, TiDB supports pushing down part of the filter conditions to the TableScan operator. That is, TiFlash first scans the column data related to the filter conditions that are pushed down to the TableScan operator, filters the rows that meet the condition, and then scans the other column data of these rows for further calculation, thereby reducing IO scans and computations of data processing. - Starting from v7.1.0, the TiFlash late materialization feature is generally available and enabled by default (the [`tidb_opt_enable_late_materialization`] (/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `ON`). The TiDB optimizer decides which filters to be pushed down to the TableScan operator based on the statistics and the filter conditions of the query. + Starting from v7.1.0, the TiFlash late materialization feature is generally available and enabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `ON`). The TiDB optimizer decides which filters to be pushed down to the TableScan operator based on the statistics and the filter conditions of the query. - For more information, see [documentation](/tiflash/tiflash-late-materialization.md)。 + For more information, see [documentation](/tiflash/tiflash-late-materialization.md). * TiFlash supports automatically choosing MPP Join algorithm according to the overhead of network transmission [#7084](https://github.com/pingcap/tiflash/issues/7084) @[solotzg](https://github.com/solotzg) **tw:qiancai** @@ -40,21 +35,21 @@ In v7.1.0-LTS, the key new features and improvements are as follows: For more information, see [documentation](/tiflash/use-tiflash-mpp-mode.md#algorithm-support-for-the-mpp-mode). -* 自适应副本读来缓解读热点 [#14151](https://github.com/tikv/tikv/issues/14151) @[sticnarf](https://github.com/sticnarf) @[you06](https://github.com/you06) **tw:Oreoxmt** +* Support load-based replica read to mitigate read hotspots [#14151](https://github.com/tikv/tikv/issues/14151) @[sticnarf](https://github.com/sticnarf) @[you06](https://github.com/you06) **tw:Oreoxmt** - 发生读热点场景，其他 TiKV 节点可能仍存在闲置资源，与其在数据主节点持续排队等待，转而从其他节点读取副本可能带来更低的延迟。TiDB 在新版本开始支持负载自适应副本读，通过 [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-从-v700-版本开始引入) 参数来设置排队时间的临界值，当估算的排队时间超过设定时，TiDB 会尝试从副本节点读取数据。在读热点的情况下，相比于不打散有 70%-200% 的读取吞吐量提升。 + In a read hotspot scenario, the hotspot TiKV node cannot process read requests in time, resulting in the read requests queuing. However, not all TiKV resources are exhausted at this time. To reduce latency, TiDB v7.1.0 introduces the load-based replica read feature, which allows TiDB to read data from other TiKV nodes without queuing on the hotspot TiKV node. You can control the queue length of read requests using the [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-new-in-v700) system variable. When the estimated queue time of the leader node exceeds this threshold, TiDB prioritizes reading data from follower nodes. This feature can improve read throughput by 70% to 200% in a read hotspot scenario compared to not scattering read hotspots. - 更多信息，请参考[用户文档](/troubleshoot-hot-spot-issues.md#打散读热点)。 + For more information, see [documentation](/troubleshoot-hot-spot-issues.md#scatter-read-hotspots). -* 非 Prepare 语句的执行计划缓存 GA [#36598](https://github.com/pingcap/tidb/issues/36598) @[qw4990](https://github.com/qw4990) **tw:Oreoxmt** +* Support caching execution plans for non-prepared statements (GA) [#36598](https://github.com/pingcap/tidb/issues/36598) @[qw4990](https://github.com/qw4990) **tw:Oreoxmt** - TiDB 在 v7.0.0 支持缓存非 Prepare 语句的执行计划，以提升在线交易场景的并发能力。v7.1.0 持续优化了这个能力，支持更多模式的 SQL 进入缓存，并正式将这个能力 GA。 + TiDB v7.0.0 introduces non-prepared plan cache as an experimental feature to improve the load capacity of concurrent OLTP. In v7.1.0, this feature is generally available, enabled by default, and supports caching more SQL statements. - GA 之后，非 Prepare 与 Prepare SQL 的缓存池合并，以提升内存利用率，缓存大小通过变量 [`tidb_session_plan_cache_size`](/system-variables.md#tidb_session_plan_cache_size) 设置。原有的变量 [`tidb_prepared_plan_cache_size`](/system-variables.md#tidb_prepared_plan_cache_size) 和 [`tidb_non_prepared_plan_cache_size`](/system-variables.md#tidb_non_prepared_plan_cache_size) 将被废弃。 + To improve memory utilization, TiDB v7.1.0 merges the cache pools of non-prepared and prepared plan caches. You can control the cache size using the system variable [`tidb_session_plan_cache_size`](/system-variables.md#tidb_session_plan_cache_size-new-in-v710). The [`tidb_prepared_plan_cache_size`](/system-variables.md#tidb_prepared_plan_cache_size-new-in-v610) and [`tidb_non_prepared_plan_cache_size`](/system-variables.md#tidb_non_prepared_plan_cache_size) system variables are deprecated. - 为了保持设置向前兼容，对于从低版本升级到 v7.1.0 的客户，缓存池大小 `tidb_session_plan_cache_size` 将会继承 `tidb_prepared_plan_cache_size`的设置，非 Parepare 语句的缓存保持关闭。经过性能测试后，用户可通过 [`tidb_enable_non_prepared_plan_cache`](/system-variables.md#tidb_enable_non_prepared_plan_cache) 打开。对于新部署的客户，非 Parepare 语句的缓存则默认打开。 + To maintain forward compatibility, when you upgrade from an earlier version to a v7.1.0 or later cluster, the cache size `tidb_session_plan_cache_size` remains the same value as `tidb_prepared_plan_cache_size`, and [`tidb_enable_non_prepared_plan_cache`](/system-variables.md#tidb_enable_non_prepared_plan_cache) remains the setting before the upgrade. After sufficient performance testing, you can enable non-prepared plan cache using `tidb_enable_non_prepared_plan_cache`. For a newly created cluster, non-prepared plan cache is enabled by default. - 更多信息，请参考[用户文档](/sql-non-prepared-plan-cache.md)。 + For more information, see [documentation](/sql-non-prepared-plan-cache.md). ### Reliability @@ -66,27 +61,25 @@ In v7.1.0-LTS, the key new features and improvements are as follows: 在 v7.1.0 中，TiDB 增加了基于实际负载来估算系统容量上限的能力，为客户的容量规划提供了更准确的参考，协助客户更好地管理 TiDB 的资源分配，从而满足企业级场景的稳定性需要。 - 更多信息，请参考[用户文档](/tidb-resource-control.md)。 + For more information, see [documentation](/tidb-resource-control.md). -* Support the checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-introduced-since-v630) to improve fault tolerance and automatic recovery capability [#issue](https://github.com/pingcap/tidb/issues/issue) @[tangenta](https://github.com/tangenta) +* Support the checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630) to improve fault tolerance and automatic recovery capability [#issue](https://github.com/pingcap/tidb/issues/issue) @[tangenta](https://github.com/tangenta) **tw:ran-huang** - TiDB v7.1.0 introduces a checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-introduced-since-v630), whichs significantly improves the fault tolerance and automatic recovery capability of Fast Online DDL. Even in the case of TiDB DDL owner switching, TiDB can still periodically record and synchronize the progress of DDL statements, allowing the new TiDB DDL owner to execute the ongoing DDL statements in Fast Online DDL mode without manually canceling and re-executing the statements. The checkpoint mechanism makes the DDL execution more stable and efficient. + TiDB v7.1.0 introduces a checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630), whichs significantly improves the fault tolerance and automatic recovery capability of Fast Online DDL. Even in the case of TiDB DDL owner switching, TiDB can still periodically record and synchronize the progress of DDL statements, allowing the new TiDB DDL owner to execute the ongoing DDL statements in Fast Online DDL mode without manually canceling and re-executing the statements. The checkpoint mechanism makes the DDL execution more stable and efficient. For more information, see [documentation](/ddl-introduction.md). -* BR 备份恢复工具支持断点恢复 [#issue](https://github.com/pingcap/tidb/issues/issue) @[Leavrth](https://github.com/Leavrth) **tw:Oreoxmt** +* Backup & Restore supports supports checkpoint restore [#issue](https://github.com/pingcap/tidb/issues/issue) @[Leavrth](https://github.com/Leavrth) **tw:Oreoxmt** - 如果用户的 TiDB 集群规模较大，之前在进行数据库的快照恢复或日志恢复时，可能会出现一些意外情况导致恢复过程提前结束，例如硬盘空间占满、节点宕机等等。在 TiDB v7.1.0 之前的版本中，这些意外情况会导致之前恢复的进度作废，需要重新进行恢复，给用户带来大量额外成本和麻烦。 + Snapshot restore or log restore might be interrupted due to recoverable errors, such as disk exhaustion and node crash. Before TiDB v7.1.0, the recovery progress before the interruption would be invalidated even after the error is addressed, and you need to start the restore from scratch. For large clusters, this incurs considerable extra cost. Starting from TiDB v7.1.0, Backup & Restore (BR) introduces the checkpoint restore feature, which enables you to continue an interrupted restore. This feature can retain most recovery progress of the interrupted restore. - 为了解决这个问题，TiDB v7.1.0 引入了备份恢复的断点恢复功能。该功能可以在意外中断后保留上一次恢复的大部分进度，使得用户能够尽可能地继续上一次的恢复的进度，避免不必要的成本和麻烦。 - - 更多信息，请参考[用户文档](/br/br-checkpoint-restore.md)。 + For more information, see [documentation](/br/br-checkpoint-restore.md). * 统计信息缓存加载策略优化 [#issue](https://github.com/pingcap/tidb/issues/issue) @[xuyifangreeneyes](https://github.com/xuyifangreeneyes) **tw:hfxsd** 在开启[统计信息同步加载](/statistics.md#统计信息的加载)的前提下，TiDB 大幅减少了启动时必须载入的统计信息的数量，并且在加载完成前不接受用户连接。一方面提升了启动时统计信息的加载速度，另一方面，避免了在启动初始阶段由于统计信息不全而引起的性能回退。提升了 TiDB 在复杂运行环境下的稳定性，降低了个别 TiDB 节点重启对整体服务的影响。 - 更多信息，请参考[用户文档]()。 + For more information, see [documentation](). ### Availability @@ -98,9 +91,9 @@ In v7.1.0-LTS, the key new features and improvements are as follows: In v7.1.0, this feature is generally available. During the execution of the `SELECT` clause in the `INSERT INTO SELECT` statement, the optimizer can intelligently decide whether to push a query down to TiFlash based on the [SQL mode](/sql-mode.md) and the cost estimates of the TiFlash replica. Therefore, the `tidb_enable_tiflash_read_for_write_stmt` system variable introduced during the experimental phase is now deprecated. Note that the computation rules of `INSERT INTO SELECT` statements for TiFlash do not meet the `STRICT SQL Mode` requirement, so TiDB allows the `SELECT` clause in the `INSERT INTO SELECT` statement to be pushed down to TiFlash only when the [SQL Mode](/sql-mode.md) of the current session is not strict, which means that the `sql_mode` value does not contain `STRICT_TRANS_TABLES` and `STRICT_ALL_TABLES`. - For more information, see [documentation](/tiflash/tiflash-results-materialization.md)。 + For more information, see [documentation](/tiflash/tiflash-results-materialization.md). -* MySQL-compatible multi-valued index becomes generally available (GA) [#39592](https://github.com/pingcap/tidb/issues/39592) @[xiongjiwei](https://github.com/xiongjiwei) @[qw4990](https://github.com/qw4990) @[YangKeao](https://github.com/YangKeao) +* MySQL-compatible multi-valued index becomes generally available (GA) [#39592](https://github.com/pingcap/tidb/issues/39592) @[xiongjiwei](https://github.com/xiongjiwei) @[qw4990](https://github.com/qw4990) @[YangKeao](https://github.com/YangKeao) **tw:ran-huang** Filtering the values of an array in a JSON column is a common operation, but normal indexes cannot help speed up such an operation. Creating a multi-valued index on an array can greatly improve filtering performance. If an array in the JSON column has a multi-valued index, you can use the multi-value index to filter the retrieval conditions with MEMBER OF(), JSON_CONTAINS(), JSON_OVERLAPS() functions, thereby reducing much I/O consumption and improving operation speed. @@ -108,7 +101,7 @@ In v7.1.0-LTS, the key new features and improvements are as follows: For more information, see [documentation](/sql-statements/sql-statement-create-index.md#multi-valued-index). -* Improve the partition management for Hash and Key partitioned tables [#42728](https://github.com/pingcap/tidb/issues/42728) @[mjonss](https://github.com/mjonss) **tw:qiancai** +* Improve the partition management for Hash and Key partitioned tables [#42728](https://github.com/pingcap/tidb/issues/42728) @[mjonss](https://github.com/mjonss) **tw:qiancai** Before v7.1.0, Hash and Key partitioned tables in TiDB only support the `TRUNCATE PARTITION` partition management statement. Starting from v7.1.0, Hash and Key partitioned tables also support `ADD PARTITION` and `COALESCE PARTITION` partition management statements. Therefore, you can flexibly adjust the number of partitions in Hash and Key partitioned tables as needed. For example, you can increase the number of partitions with the `ADD PARTITION` statement, or decrease the number of partitions with the `COALESCE PARTITION` statement. @@ -126,19 +119,19 @@ In v7.1.0-LTS, the key new features and improvements are as follows: - 支持设置 detached，允许该 job 在后台运行。 - 支持 show load data jobs, show load data jobid, drop load data jobid 来管理任务。 - 更多信息，请参考[用户文档](https://github.com/pingcap/docs-cn/pull/13344)。 + For more information, see [documentation](https://github.com/pingcap/docs-cn/pull/13344). * `LOAD DATA` SQL 集成 Lightning local backend（physical import mode） 的导入功能，提升导入性能（实验特性）[#42930](https://github.com/pingcap/tidb/issues/42930) @[D3Hunter](https://github.com/D3Hunter) **tw:hfxsd** 用户通过 `LOAD DATA` SQL 导入数据时，可以指定 import_mode = physical 来实现 Lightning local backend （physical 导入模式）的导入效果，相比 Load data 原先的 logical 导入模式，可成倍提升导入数据的性能。 - 更多信息，请参考[用户文档](链接)。 + For more information, see [documentation](链接). * `LOAD DATA` SQL 支持并行导入，提升导入性能（实验特性）[#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** 原先 load data sql 无法并行导入数据，性能较差。在该版本中支持设置并行导入的参数，通过提升并发，来提升导入的性能。在实验室环境，相比上个版本，测试逻辑导入性能有接近 4 倍的提升。 - 更多信息，请参考[用户文档](https://github.com/pingcap/docs-cn/pull/13676)。 + For more information, see [documentation](https://github.com/pingcap/docs-cn/pull/13676). * Generated columns become generally available (GA) [#issue号](链接) @[bb7133](https://github.com/bb7133) **tw:ran-huang** @@ -163,7 +156,7 @@ In v7.1.0-LTS, the key new features and improvements are as follows: For more information, see [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) and [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md). -### Data migration +### Observability * 增加优化器诊断信息 [#issue号](链接) @[time-and-fate](https://github.com/time-and-fate) **tw:hfxsd** @@ -175,30 +168,24 @@ In v7.1.0-LTS, the key new features and improvements are as follows: 更多信息，请参考[使用 `PLAN REPLAYER` 保存和恢复集群线程信息](/sql-plan-replayer.md#使用-plan-replayer-保存和恢复集群现场信息)，[使用 `EXPLAIN` 解读执行计划](/explain-walkthrough.md)和[`慢日志查询`](/identify-slow-queries.md)。 -### 安全 +### Security * Replace the interface used for querying TiFlash system table information [#6941](https://github.com/pingcap/tiflash/issues/6941) @[flowbehappy](https://github.com/flowbehappy) **tw:qiancai** Starting from v7.1.0, when providing the query service of [`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information-schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md) system tables for TiDB, TiFlash uses the gRPC port instead of the HTTP port, which avoids the security risks of the HTTP service. -### 数据迁移 +### Data migration -* TiCDC 支持 E2E 单行数据正确性校验功能 [#issue号](链接) @[3AceShowHand](https://github.com/3AceShowHand) @[zyguan](https://github.com/zyguan) **tw:Oreoxmt** +* TiCDC supports the integrity validation feature for single-row data [#8718](https://github.com/pingcap/tiflow/issues/8718) [#42747](https://github.com/pingcap/tidb/issues/42747) @[3AceShowHand](https://github.com/3AceShowHand) @[zyguan](https://github.com/zyguan) **tw:Oreoxmt** - 从 v7.1.0 版本开始，TiCDC 新增了单行数据正确性校验功能，该功能基于 Checksum 算法对单行数据的正确性进行校验。通过该功能可以校验一行数据从 TiDB 写入、经由 TiCDC 流出，再写入到 Kafka 集群的过程中是否发生了数据错误。该功能仅支持下游是 Kafka Sink 的 Changefeed，支持 Canal-JSON / Avro / Open-Protocol 等协议。 + Starting from v7.1.0, TiCDC introduces the data integrity validation feature, which uses a checksum algorithm to validate the integrity of single-row data. This feature helps verify whether any error occurs in the process of writing data from TiDB, replicating it through TiCDC, and then writing it to a Kafka cluster. The data integrity validation feature only supports changefeeds that use Kafka as the downstream and currently supports the Avro protocol. - 更多信息，请参考[用户文档](/ticdc/ticdc-integrity-check.md)。 + For more information, see [documentation](/ticdc/ticdc-integrity-check.md). -* TiCDC Optimizes DDL replication operations [#8686](https://github.com/pingcap/tiflow/issues/8686) @[nongfushanquan](https://github.com/nongfushanquan) +* TiCDC Optimizes DDL replication operations [#8686](https://github.com/pingcap/tiflow/issues/8686) @[nongfushanquan](https://github.com/nongfushanquan) **tw:ran-huang** Prior to v7.1.0, when you performed a DDL operation that affected all rows on a large table (such as adding or deleting a column), the replication delay of TiCDC would significantly increase. Starting from v7.1.0, TiCDC optimizes this issue and reduces the replication delay to less than 10 seconds. This optimization mitigates the impact of DDL operations on downstream latency. -* Lightning local backend (physical import mode) 支持在导入数据之前检测冲突的记录，并支持通过 insert ignore 和 replace 解决导入过程中的冲突记录 (实验特性) [#41629](https://github.com/pingcap/tidb/issues/41629) @[gozssky](https://github.com/gozssky) **tw:hfxsd** - - 在之前的版本使用 Lightning local backend (physical import mode) 导入数据时，当遇到冲突的记录时，无法通过 insert ignore 和 replace 来处理导入过程中的 pk、uk 冲突记录，需要用户自行去重。而本次版本，支持在导入数据之前，检查冲突的记录，并通过 replace 和 insert ignore 的语义来处理 pk、uk 冲突的记录。简化用户的操作，提升处理冲突的性能。 - - 更多信息，请参考[用户文档](链接)。 - ## Compatibility changes > **Note:** From 4b0ff26aab07497b8d16f5be9306e0996bdcc1c7 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 20 Apr 2023 09:44:26 +0800 Subject: [PATCH 09/42] Apply suggestions from code review Co-authored-by: Ran --- releases/release-7.1.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 73ddae55431b5..644310a2d7ce4 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -198,7 +198,7 @@ In v7.1.0, the key new features and improvements are as follows: * To improve security, TiFlash deprecates the HTTP service port (default `8123`) and uses the gRPC port as a replacement **tw:qiancai** - If you have upgraded TiFlash to v7.1.0, then during the TiDB upgrade to v7.1.0, the TiFlash system tables ([`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information- schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)) are not readable. + If you have upgraded TiFlash to v7.1.0, then during the TiDB upgrade to v7.1.0, TiDB cannot read the TiFlash system tables ([`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information- schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)). ### System variables From 5cbf888b8422c7575648a86f70d242a596ee91fc Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Thu, 20 Apr 2023 09:46:39 +0800 Subject: [PATCH 10/42] Apply suggestions from code review Co-authored-by: Ran --- releases/release-7.1.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 644310a2d7ce4..0ef9d1eff0090 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -27,11 +27,11 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/tiflash/tiflash-late-materialization.md). -* TiFlash supports automatically choosing MPP Join algorithm according to the overhead of network transmission [#7084](https://github.com/pingcap/tiflash/issues/7084) @[solotzg](https://github.com/solotzg) **tw:qiancai** +* TiFlash supports automatically choosing an MPP Join algorithm according to the overhead of network transmission [#7084](https://github.com/pingcap/tiflash/issues/7084) @[solotzg](https://github.com/solotzg) **tw:qiancai** - The TiFlash MPP mode supports multiple Join algorithms. Before v7.1.0, TiDB determines whether the MPP mode uses the Broadcast Hash Join algorithm based on the [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) and [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) variables and the actual overhead. + The TiFlash MPP mode supports multiple Join algorithms. Before v7.1.0, TiDB determines whether the MPP mode uses the Broadcast Hash Join algorithm based on the [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) and [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) variables and the actual data volume. - In v7.1.0, TiDB introduces the [`tidb_prefer_broadcast_join_by_exchange_data_size`](/system-variables.md#tidb_prefer_broadcast_join_by_exchange_data_size-new-in-v710) variable, which controls whether to choose the MPP Join algorithm based on the minimum overhead of network transmission. This variable is disabled by default, indicating that the default algorithm selection method remains the same as that before v7.1.0. You can set the variable to `ON` to enable it. When it is enabled, you no longer need to manually adjust the [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-introduced-from-v50-version) and [`tidb_ broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-introduced-from-v50-version) variables (both variables does not take effect at this time), TiDB automatically estimates the threshold of network transmission by different Join algorithms, and then chooses the algorithm with the smaller overhead overall, thus reducing network traffic and improving MPP query performance. + In v7.1.0, TiDB introduces the [`tidb_prefer_broadcast_join_by_exchange_data_size`](/system-variables.md#tidb_prefer_broadcast_join_by_exchange_data_size-new-in-v710) variable, which controls whether to choose the MPP Join algorithm based on the minimum overhead of network transmission. This variable is disabled by default, indicating that the default algorithm selection method remains the same as that before v7.1.0. You can set the variable to `ON` to enable it. When it is enabled, you no longer need to manually adjust the [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-introduced-from-v50-version) and [`tidb_ broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-introduced-from-v50-version) variables (both variables does not take effect at this time), TiDB automatically estimates the threshold of network transmission by different Join algorithms, and then chooses the algorithm with the smallest overhead overall, thus reducing network traffic and improving MPP query performance. For more information, see [documentation](/tiflash/use-tiflash-mpp-mode.md#algorithm-support-for-the-mpp-mode). From e4fe9aa8134de0736b34221233bf4794a77f2a97 Mon Sep 17 00:00:00 2001 From: Ran Date: Thu, 20 Apr 2023 12:46:51 +0800 Subject: [PATCH 11/42] Apply suggestions from code review Co-authored-by: Grace Cai --- releases/release-7.1.0.md | 28 ++++++++++++++-------------- 1 file changed, 14 insertions(+), 14 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 0ef9d1eff0090..4d559e6ef5c81 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -63,9 +63,9 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/tidb-resource-control.md). -* Support the checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630) to improve fault tolerance and automatic recovery capability [#issue](https://github.com/pingcap/tidb/issues/issue) @[tangenta](https://github.com/tangenta) **tw:ran-huang** +* Support the checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630) to improve fault tolerance and automatic recovery capability [#42164](https://github.com/pingcap/tidb/issues/42164) @[tangenta](https://github.com/tangenta) **tw:ran-huang** - TiDB v7.1.0 introduces a checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630), whichs significantly improves the fault tolerance and automatic recovery capability of Fast Online DDL. Even in the case of TiDB DDL owner switching, TiDB can still periodically record and synchronize the progress of DDL statements, allowing the new TiDB DDL owner to execute the ongoing DDL statements in Fast Online DDL mode without manually canceling and re-executing the statements. The checkpoint mechanism makes the DDL execution more stable and efficient. + TiDB v7.1.0 introduces a checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630), which significantly improves the fault tolerance and automatic recovery capability of Fast Online DDL. Even in the case of TiDB DDL owner switching, TiDB can still periodically record and synchronize the progress of DDL statements, allowing the new TiDB DDL owner to execute the ongoing DDL statements in Fast Online DDL mode without manually canceling and re-executing the statements. The checkpoint mechanism makes the DDL execution more stable and efficient. For more information, see [documentation](/ddl-introduction.md). @@ -93,11 +93,11 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/tiflash/tiflash-results-materialization.md). -* MySQL-compatible multi-valued index becomes generally available (GA) [#39592](https://github.com/pingcap/tidb/issues/39592) @[xiongjiwei](https://github.com/xiongjiwei) @[qw4990](https://github.com/qw4990) @[YangKeao](https://github.com/YangKeao) **tw:ran-huang** +* MySQL-compatible multi-valued indexes become generally available (GA) [#39592](https://github.com/pingcap/tidb/issues/39592) @[xiongjiwei](https://github.com/xiongjiwei) @[qw4990](https://github.com/qw4990) @[YangKeao](https://github.com/YangKeao) **tw:ran-huang** - Filtering the values of an array in a JSON column is a common operation, but normal indexes cannot help speed up such an operation. Creating a multi-valued index on an array can greatly improve filtering performance. If an array in the JSON column has a multi-valued index, you can use the multi-value index to filter the retrieval conditions with MEMBER OF(), JSON_CONTAINS(), JSON_OVERLAPS() functions, thereby reducing much I/O consumption and improving operation speed. + Filtering the values of an array in a JSON column is a common operation, but normal indexes cannot help speed up such an operation. Creating a multi-valued index on an array can greatly improve filtering performance. If an array in the JSON column has a multi-valued index, you can use the multi-valued index to filter retrieval conditions in `MEMBER OF()`, `JSON_CONTAINS()`, and `JSON_OVERLAPS()` functions, thereby reducing I/O consumption and improving operation speed. - In v7.1.0, the multi-valued index becomes a generally available (GA) feature. It supports more complete data types and is compatible with TiDB tools. You can use the multi-valued indexes to speed up the search operations on JSON arrays in production environments. + In v7.1.0, the multi-valued index feature becomes generally available (GA). It supports more complete data types and is compatible with TiDB tools. You can use multi-valued indexes to speed up the search operations on JSON arrays in production environments. For more information, see [documentation](/sql-statements/sql-statement-create-index.md#multi-valued-index). @@ -133,25 +133,25 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](https://github.com/pingcap/docs-cn/pull/13676). -* Generated columns become generally available (GA) [#issue号](链接) @[bb7133](https://github.com/bb7133) **tw:ran-huang** +* Generated columns become generally available (GA) @[bb7133](https://github.com/bb7133) **tw:ran-huang** - Generated columns are a very valuable feature in MySQL. When creating a table, you can define that the value of a column is calculated from the values of other columns in the table, rather than being explicitly inserted or updated by the user. This generated column can be either a virtual column or a stored column. TiDB has provided generated columns compatible with MySQL since earlier versions, and this feature is officially GA in v7.1.0. + Generated columns are a valuable feature for database. When creating a table, you can define that the value of a column is calculated based on the values of other columns in the table, rather than being explicitly inserted or updated by users. This generated column can be either a virtual column or a stored column. TiDB has supported MySQL-compatible generated columns since earlier versions, and this feature becomes GA in v7.1.0. - Using generated columns can improve TiDB's compatibility with MySQL, simplifying the process of migrating from MySQL. It also reduces data maintenance complexity, improves data consistency and query efficiency. + Using generated columns can improve MySQL compatibility for TiDB, simplifying the process of migrating from MySQL. It also reduces data maintenance complexity and improves data consistency and query efficiency. For more information, see [documentation](/generated-columns.md). ### DB operations -* DDL tasks support pause and resume operations [#issue number](link) @[godouxm](https://github.com/godouxm) **tw:ran-huang** +* DDL tasks support pause and resume operations (experimental) [#18015](https://github.com/pingcap/tidb/issues/18015) @[godouxm](https://github.com/godouxm) **tw:ran-huang** - Prior to TiDB v7.1.0, when a DDL task encountered a business peak period during execution, you could only manually cancel the DDL task to reduce its impact on the business. To mitigate the impact of DDL operations on application workloads, in v7.1.0, TiDB introduces pause and resume operations for DDL tasks. These operations let you pause DDL tasks during peak periods and resume them after the peak ends, thus avoiding any impact on your business. + Before TiDB v7.1.0, when a DDL task encounters a business peak period during execution, you can only manually cancel the DDL task to reduce its impact on the business. In v7.1.0, TiDB introduces pause and resume operations for DDL tasks. These operations let you pause DDL tasks during peak periods and resume them after the peak ends, thus avoiding any impact on your application workloads. For example, you can pause and resume multiple DDL tasks using [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) or [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md): ```sql - ADMIN PAUSE ddl jobs 1,2; - ADMIN RESUME ddl jobs 1,2; + ADMIN PAUSE DDL JOBS 1,2; + ADMIN RESUME DDL JOBS 1,2; ``` For more information, see [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) and [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md). @@ -182,9 +182,9 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/ticdc/ticdc-integrity-check.md). -* TiCDC Optimizes DDL replication operations [#8686](https://github.com/pingcap/tiflow/issues/8686) @[nongfushanquan](https://github.com/nongfushanquan) **tw:ran-huang** +* TiCDC optimizes DDL replication operations [#8686](https://github.com/pingcap/tiflow/issues/8686) @[nongfushanquan](https://github.com/nongfushanquan) **tw:ran-huang** - Prior to v7.1.0, when you performed a DDL operation that affected all rows on a large table (such as adding or deleting a column), the replication delay of TiCDC would significantly increase. Starting from v7.1.0, TiCDC optimizes this issue and reduces the replication delay to less than 10 seconds. This optimization mitigates the impact of DDL operations on downstream latency. + Before v7.1.0, when you perform a DDL operation that affected all rows on a large table (such as adding or deleting a column), the replication latency of TiCDC would significantly increase. Starting from v7.1.0, TiCDC optimizes this replication operation and reduces the replication latency to less than 10 seconds. This optimization mitigates the impact of DDL operations on downstream latency. ## Compatibility changes From 922844ac7b6f4da0dd2ae848ea97fa81bde339a3 Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 20 Apr 2023 17:52:54 +0800 Subject: [PATCH 12/42] remove Partitioned Raft KV and TiCDC E2E --- releases/release-7.1.0.md | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 4d559e6ef5c81..a8472b01b2581 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -13,12 +13,6 @@ In v7.1.0, the key new features and improvements are as follows: ### Performance -* 下一代 [`Partitioned Raft KV`](/partitioned-raft-kv.md) 存储引擎 GA [#issue号](链接) @[busyjay](https://github.com/busyjay) @[tonyxuqqi](https://github.com/tonyxuqqi) @[tabokie](https://github.com/tabokie) @[bufferflies](https://github.com/bufferflies) @[5kbpers](https://github.com/5kbpers) @[SpadeA-Tang](https://github.com/SpadeA-Tang) @[nolouch](https://github.com/nolouch) **tw:Oreoxmt** - - TiDB v6.6.0 引入的全新的 TiKV 存储引擎 [`Partitioned Raft KV`](/partitioned-raft-kv.md) 在 TiDB v7.1.0 版本正式 GA。该引擎使用多个 RocksDB 实例存储 TiKV 的 Region 数据，为每个 Region 提供独立的 RocksDB 实例。此外，该引擎能够更好地管理 RocksDB 实例的文件数和层级，实现 Region 间的数据操作物理隔离，并支持更多数据的平滑扩展。与原 TiKV 存储引擎相比，使用该引擎在相同硬件条件和读写混合场景下，可实现约 2 倍的写入吞吐、3 倍的读取吞吐，并缩短约 4/5 的弹性伸缩时间。该引擎与 TiFlash 引擎兼容，支持 Lightning / BR / TiCDC 等周边工具。该引擎目前仅支持在新集群中使用，暂不支持从原 TiKV 存储引擎直接升级到该引擎。 - - For more information, see [documentation](/partitioned-raft-kv.md). - * TiFlash supports late materialization (GA) [#5829](https://github.com/pingcap/tiflash/issues/5829) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) **tw:qiancai** In v7.0.0, TiFlash supports late materialization as an experiment feature for optimizing query performance. This feature is disabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `OFF`). when processing a `SELECT` statement with filter conditions (`WHERE` clause), TiFlash reads all the data from the columns required by the query, and then filters and aggregates the data based on the query conditions. When Late materialization is enabled, TiDB supports pushing down part of the filter conditions to the TableScan operator. That is, TiFlash first scans the column data related to the filter conditions that are pushed down to the TableScan operator, filters the rows that meet the condition, and then scans the other column data of these rows for further calculation, thereby reducing IO scans and computations of data processing. @@ -176,12 +170,6 @@ In v7.1.0, the key new features and improvements are as follows: ### Data migration -* TiCDC supports the integrity validation feature for single-row data [#8718](https://github.com/pingcap/tiflow/issues/8718) [#42747](https://github.com/pingcap/tidb/issues/42747) @[3AceShowHand](https://github.com/3AceShowHand) @[zyguan](https://github.com/zyguan) **tw:Oreoxmt** - - Starting from v7.1.0, TiCDC introduces the data integrity validation feature, which uses a checksum algorithm to validate the integrity of single-row data. This feature helps verify whether any error occurs in the process of writing data from TiDB, replicating it through TiCDC, and then writing it to a Kafka cluster. The data integrity validation feature only supports changefeeds that use Kafka as the downstream and currently supports the Avro protocol. - - For more information, see [documentation](/ticdc/ticdc-integrity-check.md). - * TiCDC optimizes DDL replication operations [#8686](https://github.com/pingcap/tiflow/issues/8686) @[nongfushanquan](https://github.com/nongfushanquan) **tw:ran-huang** Before v7.1.0, when you perform a DDL operation that affected all rows on a large table (such as adding or deleting a column), the replication latency of TiCDC would significantly increase. Starting from v7.1.0, TiCDC optimizes this replication operation and reduces the replication latency to less than 10 seconds. This optimization mitigates the impact of DDL operations on downstream latency. From f201c3cf74b35e4f241ecb24e30f54137b7b329e Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 21 Apr 2023 10:50:08 +0800 Subject: [PATCH 13/42] Update releases/release-7.1.0.md --- releases/release-7.1.0.md | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index a8472b01b2581..d507c044fe3a2 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -100,7 +100,11 @@ In v7.1.0, the key new features and improvements are as follows: Before v7.1.0, Hash and Key partitioned tables in TiDB only support the `TRUNCATE PARTITION` partition management statement. Starting from v7.1.0, Hash and Key partitioned tables also support `ADD PARTITION` and `COALESCE PARTITION` partition management statements. Therefore, you can flexibly adjust the number of partitions in Hash and Key partitioned tables as needed. For example, you can increase the number of partitions with the `ADD PARTITION` statement, or decrease the number of partitions with the `COALESCE PARTITION` statement. For more information, see [documentation](/partitioned-table.md#manage-hash-and-key-partitions). +* Range INTERVAL partitioning become generally available (GA) [#35683](https://github.com/pingcap/tidb/issues/35683) @[mjonss](https://github.com/mjonss) **tw:qiancai** + In v6.3.0, TiDB provides INTERVAL partitioning as a new way of defining Range partitions. You do not need to enumerate all partitions, which drastically reduces the length of Range partitioning DDL statements. The syntax is equivalent to that of the original Range partitioning. + + In v7.1.0, Range INTERVAL partitioning is now GA. * `LOAD DATA` SQL 支持从 S3、GCS 导入数据，支持任务管理等功能 GA [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** 以下 `LOAD DATA` 新增的功能在 7.1 版本 GA： From 47fba41ff8a953a4876a76c67f1557dc3a1a9cc5 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 21 Apr 2023 11:52:32 +0800 Subject: [PATCH 14/42] polish descriptions of Range INTERVAL partitioning --- releases/release-7.1.0.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index d507c044fe3a2..aa6c1727a289c 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -100,9 +100,10 @@ In v7.1.0, the key new features and improvements are as follows: Before v7.1.0, Hash and Key partitioned tables in TiDB only support the `TRUNCATE PARTITION` partition management statement. Starting from v7.1.0, Hash and Key partitioned tables also support `ADD PARTITION` and `COALESCE PARTITION` partition management statements. Therefore, you can flexibly adjust the number of partitions in Hash and Key partitioned tables as needed. For example, you can increase the number of partitions with the `ADD PARTITION` statement, or decrease the number of partitions with the `COALESCE PARTITION` statement. For more information, see [documentation](/partitioned-table.md#manage-hash-and-key-partitions). -* Range INTERVAL partitioning become generally available (GA) [#35683](https://github.com/pingcap/tidb/issues/35683) @[mjonss](https://github.com/mjonss) **tw:qiancai** - In v6.3.0, TiDB provides INTERVAL partitioning as a new way of defining Range partitions. You do not need to enumerate all partitions, which drastically reduces the length of Range partitioning DDL statements. The syntax is equivalent to that of the original Range partitioning. +* The syntax of Range INTERVAL partitioning becomes generally available (GA) [#35683](https://github.com/pingcap/tidb/issues/35683) @[mjonss](https://github.com/mjonss) **tw:qiancai** + + The syntax of Range INTERVAL partitioning (introduced in v6.3.0) becomes GA. With this syntax, you can define partitioning by a desired interval without enumerating all partitions, which drastically reduces the length of Range partitioning DDL statements. The syntax is equivalent to that of the original Range partitioning. In v7.1.0, Range INTERVAL partitioning is now GA. * `LOAD DATA` SQL 支持从 S3、GCS 导入数据，支持任务管理等功能 GA [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** From c13feb0e1b285dbe6b9b2d51a93e9be8f7567d66 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Fri, 21 Apr 2023 11:53:07 +0800 Subject: [PATCH 15/42] Update releases/release-7.1.0.md --- releases/release-7.1.0.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index aa6c1727a289c..30fa82a4982a0 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -105,7 +105,8 @@ In v7.1.0, the key new features and improvements are as follows: The syntax of Range INTERVAL partitioning (introduced in v6.3.0) becomes GA. With this syntax, you can define partitioning by a desired interval without enumerating all partitions, which drastically reduces the length of Range partitioning DDL statements. The syntax is equivalent to that of the original Range partitioning. - In v7.1.0, Range INTERVAL partitioning is now GA. + For more information, see [documentation](/partitioned-table#range-interval-partitioning). + * `LOAD DATA` SQL 支持从 S3、GCS 导入数据，支持任务管理等功能 GA [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** 以下 `LOAD DATA` 新增的功能在 7.1 版本 GA： From 8977e4893fce0d16ca256f0a37b90955887f4857 Mon Sep 17 00:00:00 2001 From: Aolin Date: Sun, 23 Apr 2023 14:29:59 +0800 Subject: [PATCH 16/42] Apply suggestions from code review Co-authored-by: Ran --- releases/release-7.1.0.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 30fa82a4982a0..d54513f073020 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -198,7 +198,8 @@ In v7.1.0, the key new features and improvements are as follows: | Variable name | Change type | Description | |--------|------------------------------|------| -| | | | +| `tidb_ddl_distribute_reorg` | Deleted | Renamed to [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710). | +| [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710) | Newly added | Controls whether to enable [the distributed execution framework](/tidb-distributed-execution-framework.md). After enabling distributed execution, DDL, Import and other supported backend tasks will be jointly completed by multiple TiDB nodes in the cluster. This variable was renamed from `tidb_ddl_distribute_reorg`. | | | | | | | | | | | | | From 048ada9c1c88b1d7402dd525be4601454ce489cb Mon Sep 17 00:00:00 2001 From: Aolin Date: Sun, 23 Apr 2023 15:57:09 +0800 Subject: [PATCH 17/42] add note and contributors Signed-off-by: Aolin --- releases/release-7.1.0.md | 40 +++++++++++++++++++++++++++++++-------- 1 file changed, 32 insertions(+), 8 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index d54513f073020..f3941fe12e5fd 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -5,6 +5,12 @@ summary: Learn about the new features, compatibility changes, improvements, and # TiDB 7.1.0 Release Notes +TiDB version: 7.1.0 (upcoming) + +> **Note:** +> +> TiDB v7.1.0 is not yet available. This preview of the release notes provides insights into an upcoming release and is subject to change. The features outlined here are not guaranteed to be included in the final release. + In v7.1.0, the key new features and improvements are as follows: ## Feature details @@ -194,24 +200,33 @@ In v7.1.0, the key new features and improvements are as follows: If you have upgraded TiFlash to v7.1.0, then during the TiDB upgrade to v7.1.0, TiDB cannot read the TiFlash system tables ([`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information- schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)). +* [`SHOW LOAD DATA`](/sql-statements/sql-statement-show-load-data.md) 的返回值中废弃了参数 `Loaded_File_Size`，替换为参数 `Imported_Rows` **tw:hfxsd** + ### System variables | Variable name | Change type | Description | |--------|------------------------------|------| -| `tidb_ddl_distribute_reorg` | Deleted | Renamed to [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710). | +| [`tidb_enable_tiflash_read_for_write_stmt`](/system-variables.md#tidb_enable_tiflash_read_for_write_stmt-new-in-v630) | Deprecated | Changes the default value from `OFF` to `ON`. When [`tidb_allow_mpp = ON`](/system-variables.md#tidb_allow_mpp-new-in-v50), the optimizer intelligently decides whether to push a query down to TiFlash based on the [SQL mode](/sql-mode.md) and the cost estimates of the TiFlash replica. | +| [`tidb_non_prepared_plan_cache_size`](/system-variables.md#tidb_non_prepared_plan_cache_size) | Deprecated | Starting from v7.1.0, this system variable is deprecated. You can use [`tidb_session_plan_cache_size`](#tidb_session_plan_cache_size-new-in-v710) to control the maximum number of plans that can be cached. | +| [`tidb_prepared_plan_cache_size`](/system-variables.md#tidb_prepared_plan_cache_size-new-in-v610) | Deprecated | Starting from v7.1.0, this system variable is deprecated. You can use [`tidb_session_plan_cache_size`](#tidb_session_plan_cache_size-new-in-v710) to control the maximum number of plans that can be cached. | +| `tidb_ddl_distribute_reorg` | Deleted | This variable is renamed to [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710). | +| [`tidb_enable_non_prepared_plan_cache`](/system-variables.md#tidb_enable_non_prepared_plan_cache) | Modified | Changes the default value from `OFF` to `ON` after further tests, meaning that non-prepared plan cache is enabled by default. | +| [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-new-in-v700) | Modified | Takes effect starting from v7.1.0 and controls the threshold for triggering load-based replica read. Changes the default value from `"0s"` to `"1s"` after further tests. | +| [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) | Modified | Changes the default value from `OFF` to `ON`, meaning that the TiFlash late materialization feature is enabled by default. | | [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710) | Newly added | Controls whether to enable [the distributed execution framework](/tidb-distributed-execution-framework.md). After enabling distributed execution, DDL, Import and other supported backend tasks will be jointly completed by multiple TiDB nodes in the cluster. This variable was renamed from `tidb_ddl_distribute_reorg`. | -| | | | -| | | | -| | | | +| [`tidb_enable_non_prepared_plan_cache_for_dml`](/system-variables.md#tidb_enable_non_prepared_plan_cache_for_dml-new-in-v710) | Newly added | Controls whether to enable the [Non-prepared plan cache](/sql-non-prepared-plan-cache.md) feature for DML statements. | +| [`tidb_opt_fix_control`](/system-variables.md#tidb_opt_fix_control-new-in-v710) | Newly added | This variable provides a more fine-grained control over the optimizer and helps to prevent performance regression after upgrading caused by behavior changes in the optimizer. | +| [`tidb_prefer_broadcast_join_by_exchange_data_size`](/system-variables.md#tidb_prefer_broadcast_join_by_exchange_data_size-new-in-v710) | Newly added | Controls whether to use the algorithm with the minimum overhead of network transmission. If this variable is enabled, TiDB estimates the size of the data to be exchanged in the network using `Broadcast Hash Join` and `Shuffled Hash Join` respectively, and then chooses the one with the smaller size. [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) and [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_size-new-in-v50) will not take effect after this variable is enabled. | +| [`tidb_session_plan_cache_size`](/system-variables.md#tidb_session_plan_cache_size-new-in-v710) | Newly added | Controls the maximum number of plans that can be cached. Prepared plan cache and non-prepared plan cache share the same cache. | ### Configuration file parameters | Configuration file | Configuration parameter | Change type | Description | | -------- | -------- | -------- | -------- | -| TiFlash | `http_port` | 删除 | 废弃 TiFlash HTTP 服务端口（默认 `8123`）。| -| | | | | -| | | | | -| | | | | +| TiDB | [`lite-init-stats`](/tidb-configuration-file.md#lite-init-stats-new-in-v710) | Newly added | Controls whether to use lightweight statistics initialization during TiDB startup. | +| PD | [`store-limit-version`](/pd-configuration-file.md#store-limit-version-new-in-v710) | Newly added | Controls the mode of store limit. Value options are `"v1"` and `"v2"`. | +| TiFlash | `http_port` | Deleted | Deprecates the HTTP service port (default `8123`). | +| TiCDC | [`sink.enable-partition-separator`](/ticdc/ticdc-changefeed-config.md#cli-and-configuration-parameters-of-ticdc-changefeeds) | Modified | Changes the default value from `false` to `true` after further tests, meaning that partitions in a table are stored in separate directories by default. It is recommended that you keep the value as `true` to avoid the data loss issue. | ### Others @@ -318,3 +333,12 @@ In v7.1.0, the key new features and improvements are as follows: ## Contributors We would like to thank the following contributors from the TiDB community: + +- [ethercflow](https://github.com/ethercflow) +- [hihihuhu](https://github.com/hihihuhu) +- [jiyfhust](https://github.com/jiyfhust) +- [L-maple](https://github.com/L-maple) +- [lqs](https://github.com/lqs) +- [pingandb](https://github.com/pingandb) +- [yorkhellen](https://github.com/yorkhellen) +- [yujiarista](https://github.com/yujiarista) From ac3bdc1d4fb37d6a49e03c498b4f60737b69fa7f Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Sun, 23 Apr 2023 16:55:10 +0800 Subject: [PATCH 18/42] Apply suggestions from code review Co-authored-by: Ran --- releases/release-7.1.0.md | 63 ++++++++++++++++++++------------------- 1 file changed, 32 insertions(+), 31 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index f3941fe12e5fd..6fb01108cab19 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -53,13 +53,13 @@ In v7.1.0, the key new features and improvements are as follows: ### Reliability -* 资源管控 GA [#38825](https://github.com/pingcap/tidb/issues/38825) @[nolouch](https://github.com/nolouch) @[BornChanger](https://github.com/BornChanger) @[glorv](https://github.com/glorv) @[tiancaiamao](https://github.com/tiancaiamao) @[Connor1996](https://github.com/Connor1996) @[JmPotato](https://github.com/JmPotato) @[hnes](https://github.com/hnes) @[CabinfeverB](https://github.com/CabinfeverB) @[HuSharp](https://github.com/HuSharp) **tw:hfxsd** +* Resource Control GA [#38825](https://github.com/pingcap/tidb/issues/38825) @[nolouch](https://github.com/nolouch) @[BornChanger](https://github.com/BornChanger) @[glorv](https://github.com/glorv) @[tiancaiamao](https://github.com/tiancaiamao) @[Connor1996](https://github.com/Connor1996) @[JmPotato](https://github.com/JmPotato) @[hnes](https://github.com/hnes) @[CabinfeverB](https://github.com/CabinfeverB) @[HuSharp](https://github.com/HuSharp) **tw:hfxsd** - TiDB 持续增强资源管控能力，并将这个特性 GA。该特性将会极大地提升 TiDB 集群的资源利用效率和性能表现。资源管控特性的引入对 TiDB 具有里程碑的意义，你可以将一个分布式数据库集群划分成多个逻辑单元，将不同的数据库用户映射到对应的资源组中，并根据需要设置每个资源组的配额。当集群资源紧张时，来自同一个资源组的会话所使用的全部资源将被限制在配额内，避免其中一个资源组过度消耗，从而影响其他资源组中的会话正常运行。 + TiDB enhances the resource control feature based on resource groups. This feature becomes GA in v7.1.0. This feature significantly improves the resource utilization efficiency and performance of TiDB clusters. The introduction of the resource control feature is a milestone for TiDB. You can divide a distributed database cluster into multiple logical units, map different database users to corresponding resource groups, and set the quota for each resource group as needed. When the cluster resources are limited, all resources used by sessions in the same resource group are limited to the quota. In this way, even if a resource group is over-consumed, the sessions in other resource groups are not affected. - 该特性也可以将多个来自不同系统的中小型应用合入一个 TiDB 集群中，个别应用的负载提升，不会影响其他应用的正常运行。而在系统负载较低的时候，繁忙的应用即使超过设定的读写配额，也仍然可以被分配到所需的系统资源，达到资源的最大化利用。此外，合理利用资源管控特性可以减少集群数量，降低运维难度及管理成本。 + With this feature, you can combine multiple small and medium-sized applications from different systems into a single TiDB cluster. When the workload of an application grows larger, it does not affect the normal operation of other applications. When the system workload is low, busy applications can still be allocated the required system resources even if they exceed the set read and write quotas, which can achieve the maximum utilization of resources. In addition, the rational use of the resource control feature can reduce the number of clusters, ease the difficulty of operation and maintenance, and save management costs. - 在 v7.1.0 中，TiDB 增加了基于实际负载来估算系统容量上限的能力，为客户的容量规划提供了更准确的参考，协助客户更好地管理 TiDB 的资源分配，从而满足企业级场景的稳定性需要。 + In TiDB v7.1.0, this feature adds the ability to estimate system capacity caps based on actual workload and hardware deployment. The estimation ability provides you with a more accurate reference for capacity planning and assists you in better managing TiDB resource allocation to meet the stability needs of enterprise-class scenarios. For more information, see [documentation](/tidb-resource-control.md). @@ -75,11 +75,11 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/br/br-checkpoint-restore.md). -* 统计信息缓存加载策略优化 [#issue](https://github.com/pingcap/tidb/issues/issue) @[xuyifangreeneyes](https://github.com/xuyifangreeneyes) **tw:hfxsd** +* Optimize the policy of loading statistical information cache [#42160](https://github.com/pingcap/tidb/issues/42160) @[xuyifangreeneyes](https://github.com/xuyifangreeneyes) **tw:hfxsd** - 在开启[统计信息同步加载](/statistics.md#统计信息的加载)的前提下，TiDB 大幅减少了启动时必须载入的统计信息的数量，并且在加载完成前不接受用户连接。一方面提升了启动时统计信息的加载速度，另一方面，避免了在启动初始阶段由于统计信息不全而引起的性能回退。提升了 TiDB 在复杂运行环境下的稳定性，降低了个别 TiDB 节点重启对整体服务的影响。 + By enabling [synchronous loading of statistics](/statistics.md#load-statistics), TiDB can significantly reduce the number of statistics that must be loaded upon startup and improves the speed of loading statistics. This feature increases the stability of TiDB in complex runtime environments and reduces the impact of individual TiDB node restart on the overall service. - For more information, see [documentation](). + For more information, see [documentation](/statistics.md#load-statistics)。 ### Availability @@ -109,35 +109,35 @@ In v7.1.0, the key new features and improvements are as follows: * The syntax of Range INTERVAL partitioning becomes generally available (GA) [#35683](https://github.com/pingcap/tidb/issues/35683) @[mjonss](https://github.com/mjonss) **tw:qiancai** - The syntax of Range INTERVAL partitioning (introduced in v6.3.0) becomes GA. With this syntax, you can define partitioning by a desired interval without enumerating all partitions, which drastically reduces the length of Range partitioning DDL statements. The syntax is equivalent to that of the original Range partitioning. + The syntax of Range INTERVAL partitioning (introduced in v6.3.0) becomes GA. With this syntax, you can define Range partitioning by a desired interval without enumerating all partitions, which drastically reduces the length of Range partitioning DDL statements. The syntax is equivalent to that of the original Range partitioning. For more information, see [documentation](/partitioned-table#range-interval-partitioning). -* `LOAD DATA` SQL 支持从 S3、GCS 导入数据，支持任务管理等功能 GA [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** +* `LOAD DATA` some features become GA. [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** - 以下 `LOAD DATA` 新增的功能在 7.1 版本 GA： + The following `LOAD DATA` features become GA in TiDB v7.1.0: - - 支持从 S3、GCS 导入数据 - - 支持导入 Parquet 文件数据 - - 支持解析源文件中 ascii、latin1、binary、gbk、utf8mbd 字符集 - - 支持设置 FIELDS DEFINED NULL BY 将源文件的指定的值转换为 Null 写入目标表。 - - 支持设置一个 bath_size 即 1 个 batch 插入到目标表的行数，提升写入性能。 - - 支持设置 detached，允许该 job 在后台运行。 - - 支持 show load data jobs, show load data jobid, drop load data jobid 来管理任务。 + - Support importing data from SCS and GCS. + - Support importing data from Parquet files. + - Support parsing the following character sets in the source file: `ascii`, `latin1`, `binary`, `gbk`, `utf8mbd`. + - Support setting `FIELDS DEFINED NULL BY` to convert the specified value of the source file to `NULL` for writing to the target table. + - Support setting 1 `bath_size` which is the number of rows inserted into the target table by 1 batch to improve write performance. + - Support setting `detached` to allow the job to run in the background. + - Support using `SHOW LOAD DATA` and `DROP LOAD DATA` to manage jobs. - For more information, see [documentation](https://github.com/pingcap/docs-cn/pull/13344). + For more information, see [documentation](/sql-statements/sql-statement-load-data.md). -* `LOAD DATA` SQL 集成 Lightning local backend（physical import mode） 的导入功能，提升导入性能（实验特性）[#42930](https://github.com/pingcap/tidb/issues/42930) @[D3Hunter](https://github.com/D3Hunter) **tw:hfxsd** +* `LOAD DATA` integrates with TiDB Lightning Physical Import Mode to improve import performance (experimental) [#42930](https://github.com/pingcap/tidb/issues/42930) @[D3Hunter](https://github.com/D3Hunter) **tw:hfxsd** - 用户通过 `LOAD DATA` SQL 导入数据时，可以指定 import_mode = physical 来实现 Lightning local backend （physical 导入模式）的导入效果，相比 Load data 原先的 logical 导入模式，可成倍提升导入数据的性能。 + `LOAD DATA` integrates with TiDB Lightning Physical Import Mode. You can enable it by setting `WITH import_mode = 'PHYSICAL'`. This feature can improve the performance of importing data exponentially compared to Logical Import Mode. - For more information, see [documentation](链接). + For more information, see [documentation](/sql-statements/sql-statement-load-data.md). -* `LOAD DATA` SQL 支持并行导入，提升导入性能（实验特性）[#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** +* `LOAD DATA` supports import concurrency to improve import performance (experimental) [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** - 原先 load data sql 无法并行导入数据，性能较差。在该版本中支持设置并行导入的参数，通过提升并发，来提升导入的性能。在实验室环境，相比上个版本，测试逻辑导入性能有接近 4 倍的提升。 + Previously, `LOAD DATA` did not support concurrent data import, so the performance was poor. In TiDB v7.1.0, you can set the parameter `WITH thread=` for concurrent import, which can improve the performance of import by increasing concurrency. In the lab environment, the performance of test logic import is nearly 4 times better than the previous version. - For more information, see [documentation](https://github.com/pingcap/docs-cn/pull/13676). + For more information, see [documentation](/sql-statements/sql-statement-load-data.md). * Generated columns become generally available (GA) @[bb7133](https://github.com/bb7133) **tw:ran-huang** @@ -164,15 +164,16 @@ In v7.1.0, the key new features and improvements are as follows: ### Observability -* 增加优化器诊断信息 [#issue号](链接) @[time-and-fate](https://github.com/time-and-fate) **tw:hfxsd** +* Add optimizer diagnostics information [#43122](https://github.com/pingcap/tidb/issues/43122) @[time-and-fate](https://github.com/time-and-fate) **tw:hfxsd** - 获取充足的信息是 SQL 性能诊断的关键，在 v7.1.0 中，TiDB 持续向各种诊断工具中添加优化器运行信息，可以更好地解释执行计划如何被选择，协助用户和技术支持对 SQL 性能问题进行定位。其中包括： + Having sufficient information is key to SQL performance diagnostics. In v7.1.0, TiDB continues to add optimizer operation information to various diagnostic tools that can better explain how execution plans are selected. Such information can assist in troubleshooting SQL performance issues. The information includes: - * [`PLAN REPLAYER`](/sql-plan-replayer.md#使用-plan-replayer-保存和恢复集群现场信息) 的输出中增加 `debug_trace.json` 文件。 - * [`EXPLAIN`](/explain-walkthrough.md) 的输出中，为 `operator info` 添加部分统计信息详情。 - * 为[`慢日志`](/identify-slow-queries.md)的 `Stats` 字段添加部分统计信息详情。 + * The output of [`PLAN REPLAYER`](/sql-plan-replayer.md) adds `debug_trace.json`. + * The output of [`EXPLAIN`](/explain-walkthrough.md) adds partial details of the statistics for `operator info`. + * Add partial statistics details to the `Stats` field of [`slow-queries`](/identify-slow-queries.md). + + For more information, see [Use PLAN REPLAYER to Save and Restore the On-Site Information of a Cluster](/sql-plan-replayer.md), [EXPLAIN Walkthrough](/explain-walkthrough.md) and [Identify Slow Queries](/identify-slow-queries.md). - 更多信息，请参考[使用 `PLAN REPLAYER` 保存和恢复集群线程信息](/sql-plan-replayer.md#使用-plan-replayer-保存和恢复集群现场信息)，[使用 `EXPLAIN` 解读执行计划](/explain-walkthrough.md)和[`慢日志查询`](/identify-slow-queries.md)。 ### Security @@ -200,7 +201,7 @@ In v7.1.0, the key new features and improvements are as follows: If you have upgraded TiFlash to v7.1.0, then during the TiDB upgrade to v7.1.0, TiDB cannot read the TiFlash system tables ([`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information- schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)). -* [`SHOW LOAD DATA`](/sql-statements/sql-statement-show-load-data.md) 的返回值中废弃了参数 `Loaded_File_Size`，替换为参数 `Imported_Rows` **tw:hfxsd** +* The parameter `Loaded_File_Size` is deprecated in the return value of [`SHOW LOAD DATA`](/sql-statements/sql-statement-show-load-data.md) and replaced with the parameter `Imported_Rows` **tw:hfxsd** ### System variables From e381f59a1f67f5dde678a7ac263d60e65005d13d Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Sun, 23 Apr 2023 23:15:56 +0800 Subject: [PATCH 19/42] Update releases/release-7.1.0.md --- releases/release-7.1.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 6fb01108cab19..397e48081a07a 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -133,7 +133,7 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/sql-statements/sql-statement-load-data.md). -* `LOAD DATA` supports import concurrency to improve import performance (experimental) [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** +* `LOAD DATA` supports concurrent data import to improve import performance (experimental) [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** Previously, `LOAD DATA` did not support concurrent data import, so the performance was poor. In TiDB v7.1.0, you can set the parameter `WITH thread=` for concurrent import, which can improve the performance of import by increasing concurrency. In the lab environment, the performance of test logic import is nearly 4 times better than the previous version. From ee826fb886df3cf02b7ff40bf75006c60ebd83fb Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Mon, 24 Apr 2023 10:02:56 +0800 Subject: [PATCH 20/42] Apply suggestions from code review --- releases/release-7.1.0.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 397e48081a07a..aba66c2062195 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -77,7 +77,7 @@ In v7.1.0, the key new features and improvements are as follows: * Optimize the policy of loading statistical information cache [#42160](https://github.com/pingcap/tidb/issues/42160) @[xuyifangreeneyes](https://github.com/xuyifangreeneyes) **tw:hfxsd** - By enabling [synchronous loading of statistics](/statistics.md#load-statistics), TiDB can significantly reduce the number of statistics that must be loaded upon startup and improves the speed of loading statistics. This feature increases the stability of TiDB in complex runtime environments and reduces the impact of individual TiDB node restart on the overall service. + By enabling [synchronous loading of statistics](/statistics.md#load-statistics), you can significantly reduce the number of statistics that must be loaded upon startup and improve the speed of loading statistics. This feature increases the stability of TiDB in complex runtime environments and reduces the impact of individual TiDB node restart on the overall service. For more information, see [documentation](/statistics.md#load-statistics)。 @@ -113,7 +113,7 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/partitioned-table#range-interval-partitioning). -* `LOAD DATA` some features become GA. [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** +* Some `LOAD DATA` features become GA [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** The following `LOAD DATA` features become GA in TiDB v7.1.0: @@ -135,7 +135,7 @@ In v7.1.0, the key new features and improvements are as follows: * `LOAD DATA` supports concurrent data import to improve import performance (experimental) [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** - Previously, `LOAD DATA` did not support concurrent data import, so the performance was poor. In TiDB v7.1.0, you can set the parameter `WITH thread=` for concurrent import, which can improve the performance of import by increasing concurrency. In the lab environment, the performance of test logic import is nearly 4 times better than the previous version. + Previously, `LOAD DATA` did not support concurrent data import, so the performance was not as good as expected. In TiDB v7.1.0, you can set the parameter `WITH thread=` for concurrent import, which can improve the performance of import by increasing concurrency. In the lab environment, the performance of test logic import is nearly 4 times better than the previous version. For more information, see [documentation](/sql-statements/sql-statement-load-data.md). @@ -168,9 +168,9 @@ In v7.1.0, the key new features and improvements are as follows: Having sufficient information is key to SQL performance diagnostics. In v7.1.0, TiDB continues to add optimizer operation information to various diagnostic tools that can better explain how execution plans are selected. Such information can assist in troubleshooting SQL performance issues. The information includes: - * The output of [`PLAN REPLAYER`](/sql-plan-replayer.md) adds `debug_trace.json`. - * The output of [`EXPLAIN`](/explain-walkthrough.md) adds partial details of the statistics for `operator info`. - * Add partial statistics details to the `Stats` field of [`slow-queries`](/identify-slow-queries.md). + * In the output of [`PLAN REPLAYER`](/sql-plan-replayer.md), add `debug_trace.json`. + * In the output of [`EXPLAIN`](/explain-walkthrough.md), add partial details of the statistics for `operator info`. + * In the `Stats` field of [`slow-queries`](/identify-slow-queries.md), add partial statistics details. For more information, see [Use PLAN REPLAYER to Save and Restore the On-Site Information of a Cluster](/sql-plan-replayer.md), [EXPLAIN Walkthrough](/explain-walkthrough.md) and [Identify Slow Queries](/identify-slow-queries.md). @@ -201,7 +201,7 @@ In v7.1.0, the key new features and improvements are as follows: If you have upgraded TiFlash to v7.1.0, then during the TiDB upgrade to v7.1.0, TiDB cannot read the TiFlash system tables ([`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information- schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)). -* The parameter `Loaded_File_Size` is deprecated in the return value of [`SHOW LOAD DATA`](/sql-statements/sql-statement-show-load-data.md) and replaced with the parameter `Imported_Rows` **tw:hfxsd** +* In the return value of [`SHOW LOAD DATA`](/sql-statements/sql-statement-show-load-data.md), the parameter `Loaded_File_Size` is deprecated and replaced with the parameter `Imported_Rows` **tw:hfxsd** ### System variables From 9c95aaf7ad055226195c16f57a6794fea212dd98 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Mon, 24 Apr 2023 10:52:20 +0800 Subject: [PATCH 21/42] Update releases/release-7.1.0.md --- releases/release-7.1.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index aba66c2062195..4737077e4519d 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -59,7 +59,7 @@ In v7.1.0, the key new features and improvements are as follows: With this feature, you can combine multiple small and medium-sized applications from different systems into a single TiDB cluster. When the workload of an application grows larger, it does not affect the normal operation of other applications. When the system workload is low, busy applications can still be allocated the required system resources even if they exceed the set read and write quotas, which can achieve the maximum utilization of resources. In addition, the rational use of the resource control feature can reduce the number of clusters, ease the difficulty of operation and maintenance, and save management costs. - In TiDB v7.1.0, this feature adds the ability to estimate system capacity caps based on actual workload and hardware deployment. The estimation ability provides you with a more accurate reference for capacity planning and assists you in better managing TiDB resource allocation to meet the stability needs of enterprise-class scenarios. + In TiDB v7.1.0, this feature adds the ability to estimate system capacity based on actual workload and hardware deployment. The estimation ability provides you with a more accurate reference for capacity planning and assists you in better managing TiDB resource allocation to meet the stability needs of enterprise-class scenarios. For more information, see [documentation](/tidb-resource-control.md). From 3a4d9d2ea5b448c81618746607d66e26c2b7a9ad Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Mon, 24 Apr 2023 11:19:11 +0800 Subject: [PATCH 22/42] Update releases/release-7.1.0.md --- releases/release-7.1.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 4737077e4519d..84efb73717f36 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -135,7 +135,7 @@ In v7.1.0, the key new features and improvements are as follows: * `LOAD DATA` supports concurrent data import to improve import performance (experimental) [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** - Previously, `LOAD DATA` did not support concurrent data import, so the performance was not as good as expected. In TiDB v7.1.0, you can set the parameter `WITH thread=` for concurrent import, which can improve the performance of import by increasing concurrency. In the lab environment, the performance of test logic import is nearly 4 times better than the previous version. + Previously, `LOAD DATA` did not support concurrent data import, so the performance was not as good as expected. In TiDB v7.1.0, you can set the parameter `WITH thread=` for concurrent import, which can improve the performance of import by increasing concurrency. In the lab environment, the logical import performance of the test workload is nearly 4 times better than the previous version. For more information, see [documentation](/sql-statements/sql-statement-load-data.md). From 4c8f203f5f0885c2c5d139ab9c7262cce6833def Mon Sep 17 00:00:00 2001 From: Aolin Date: Mon, 24 Apr 2023 12:58:13 +0800 Subject: [PATCH 23/42] Apply suggestions from code review Co-authored-by: xixirangrang --- releases/release-7.1.0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 84efb73717f36..fb153c18f0cd5 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -47,7 +47,7 @@ In v7.1.0, the key new features and improvements are as follows: To improve memory utilization, TiDB v7.1.0 merges the cache pools of non-prepared and prepared plan caches. You can control the cache size using the system variable [`tidb_session_plan_cache_size`](/system-variables.md#tidb_session_plan_cache_size-new-in-v710). The [`tidb_prepared_plan_cache_size`](/system-variables.md#tidb_prepared_plan_cache_size-new-in-v610) and [`tidb_non_prepared_plan_cache_size`](/system-variables.md#tidb_non_prepared_plan_cache_size) system variables are deprecated. - To maintain forward compatibility, when you upgrade from an earlier version to a v7.1.0 or later cluster, the cache size `tidb_session_plan_cache_size` remains the same value as `tidb_prepared_plan_cache_size`, and [`tidb_enable_non_prepared_plan_cache`](/system-variables.md#tidb_enable_non_prepared_plan_cache) remains the setting before the upgrade. After sufficient performance testing, you can enable non-prepared plan cache using `tidb_enable_non_prepared_plan_cache`. For a newly created cluster, non-prepared plan cache is enabled by default. + To maintain forward compatibility, when you upgrade from an earlier version to v7.1.0 or later versions, the cache size `tidb_session_plan_cache_size` remains the same value as `tidb_prepared_plan_cache_size`, and [`tidb_enable_non_prepared_plan_cache`](/system-variables.md#tidb_enable_non_prepared_plan_cache) remains the setting before the upgrade. After sufficient performance testing, you can enable non-prepared plan cache using `tidb_enable_non_prepared_plan_cache`. For a newly created cluster, non-prepared plan cache is enabled by default. For more information, see [documentation](/sql-non-prepared-plan-cache.md). @@ -69,7 +69,7 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/ddl-introduction.md). -* Backup & Restore supports supports checkpoint restore [#issue](https://github.com/pingcap/tidb/issues/issue) @[Leavrth](https://github.com/Leavrth) **tw:Oreoxmt** +* Backup & Restore supports checkpoint restore [#issue](https://github.com/pingcap/tidb/issues/issue) @[Leavrth](https://github.com/Leavrth) **tw:Oreoxmt** Snapshot restore or log restore might be interrupted due to recoverable errors, such as disk exhaustion and node crash. Before TiDB v7.1.0, the recovery progress before the interruption would be invalidated even after the error is addressed, and you need to start the restore from scratch. For large clusters, this incurs considerable extra cost. Starting from TiDB v7.1.0, Backup & Restore (BR) introduces the checkpoint restore feature, which enables you to continue an interrupted restore. This feature can retain most recovery progress of the interrupted restore. From 4931dde05bdf8322242410b8ba5f258e0f80a268 Mon Sep 17 00:00:00 2001 From: xixirangrang Date: Mon, 24 Apr 2023 15:51:11 +0800 Subject: [PATCH 24/42] Apply suggestions from code review Co-authored-by: Aolin --- releases/release-7.1.0.md | 40 +++++++++++++++++++-------------------- 1 file changed, 20 insertions(+), 20 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index fb153c18f0cd5..50f5048a158a4 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -53,13 +53,13 @@ In v7.1.0, the key new features and improvements are as follows: ### Reliability -* Resource Control GA [#38825](https://github.com/pingcap/tidb/issues/38825) @[nolouch](https://github.com/nolouch) @[BornChanger](https://github.com/BornChanger) @[glorv](https://github.com/glorv) @[tiancaiamao](https://github.com/tiancaiamao) @[Connor1996](https://github.com/Connor1996) @[JmPotato](https://github.com/JmPotato) @[hnes](https://github.com/hnes) @[CabinfeverB](https://github.com/CabinfeverB) @[HuSharp](https://github.com/HuSharp) **tw:hfxsd** +* Resource Control becomes generally available (GA) [#38825](https://github.com/pingcap/tidb/issues/38825) @[nolouch](https://github.com/nolouch) @[BornChanger](https://github.com/BornChanger) @[glorv](https://github.com/glorv) @[tiancaiamao](https://github.com/tiancaiamao) @[Connor1996](https://github.com/Connor1996) @[JmPotato](https://github.com/JmPotato) @[hnes](https://github.com/hnes) @[CabinfeverB](https://github.com/CabinfeverB) @[HuSharp](https://github.com/HuSharp) **tw:hfxsd** - TiDB enhances the resource control feature based on resource groups. This feature becomes GA in v7.1.0. This feature significantly improves the resource utilization efficiency and performance of TiDB clusters. The introduction of the resource control feature is a milestone for TiDB. You can divide a distributed database cluster into multiple logical units, map different database users to corresponding resource groups, and set the quota for each resource group as needed. When the cluster resources are limited, all resources used by sessions in the same resource group are limited to the quota. In this way, even if a resource group is over-consumed, the sessions in other resource groups are not affected. + TiDB enhances the resource control feature based on resource groups, which becomes GA in v7.1.0. This feature significantly improves the resource utilization efficiency and performance of TiDB clusters. The introduction of the resource control feature is a milestone for TiDB. You can divide a distributed database cluster into multiple logical units, map different database users to corresponding resource groups, and set the quota for each resource group as needed. When the cluster resources are limited, all resources used by sessions in the same resource group are limited to the quota. In this way, even if a resource group is over-consumed, the sessions in other resource groups are not affected. With this feature, you can combine multiple small and medium-sized applications from different systems into a single TiDB cluster. When the workload of an application grows larger, it does not affect the normal operation of other applications. When the system workload is low, busy applications can still be allocated the required system resources even if they exceed the set read and write quotas, which can achieve the maximum utilization of resources. In addition, the rational use of the resource control feature can reduce the number of clusters, ease the difficulty of operation and maintenance, and save management costs. - In TiDB v7.1.0, this feature adds the ability to estimate system capacity based on actual workload and hardware deployment. The estimation ability provides you with a more accurate reference for capacity planning and assists you in better managing TiDB resource allocation to meet the stability needs of enterprise-class scenarios. + In TiDB v7.1.0, this feature introduces the ability to estimate system capacity based on actual workload or hardware deployment. The estimation ability provides you with a more accurate reference for capacity planning and assists you in better managing TiDB resource allocation to meet the stability needs of enterprise-level scenarios. For more information, see [documentation](/tidb-resource-control.md). @@ -75,11 +75,11 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/br/br-checkpoint-restore.md). -* Optimize the policy of loading statistical information cache [#42160](https://github.com/pingcap/tidb/issues/42160) @[xuyifangreeneyes](https://github.com/xuyifangreeneyes) **tw:hfxsd** +* Optimize the strategy of loading statistics [#42160](https://github.com/pingcap/tidb/issues/42160) @[xuyifangreeneyes](https://github.com/xuyifangreeneyes) **tw:hfxsd** - By enabling [synchronous loading of statistics](/statistics.md#load-statistics), you can significantly reduce the number of statistics that must be loaded upon startup and improve the speed of loading statistics. This feature increases the stability of TiDB in complex runtime environments and reduces the impact of individual TiDB node restart on the overall service. + Enabling synchronous loading of statistics can significantly reduce the number of statistics that must be loaded during startup, thus improving the speed of loading statistics. This feature increases the stability of TiDB in complex runtime environments and reduces the impact of individual TiDB nodes restart on the overall service. - For more information, see [documentation](/statistics.md#load-statistics)。 + For more information, see [documentation](/statistics.md#load-statistics). ### Availability @@ -115,17 +115,17 @@ In v7.1.0, the key new features and improvements are as follows: * Some `LOAD DATA` features become GA [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** - The following `LOAD DATA` features become GA in TiDB v7.1.0: + In TiDB v7.1.0, the following `LOAD DATA` features become GA: - - Support importing data from SCS and GCS. + - Support importing data from S3 or GCS. - Support importing data from Parquet files. - - Support parsing the following character sets in the source file: `ascii`, `latin1`, `binary`, `gbk`, `utf8mbd`. - - Support setting `FIELDS DEFINED NULL BY` to convert the specified value of the source file to `NULL` for writing to the target table. - - Support setting 1 `bath_size` which is the number of rows inserted into the target table by 1 batch to improve write performance. + - Support parsing the following character sets in the source file: `ascii`, `latin1`, `binary`, `gbk`, and `utf8mbd`. + - Support setting `FIELDS DEFINED NULL BY` to convert the specified value in the source file to `NULL` for writing to the target table. + - Support setting `bath_size` which specifies the number of rows inserted into the target table per batch, to improve write performance. - Support setting `detached` to allow the job to run in the background. - Support using `SHOW LOAD DATA` and `DROP LOAD DATA` to manage jobs. - For more information, see [documentation](/sql-statements/sql-statement-load-data.md). + For more information, see [documentation](/sql-statements/sql-statement-load-data.md). * `LOAD DATA` integrates with TiDB Lightning Physical Import Mode to improve import performance (experimental) [#42930](https://github.com/pingcap/tidb/issues/42930) @[D3Hunter](https://github.com/D3Hunter) **tw:hfxsd** @@ -135,7 +135,7 @@ In v7.1.0, the key new features and improvements are as follows: * `LOAD DATA` supports concurrent data import to improve import performance (experimental) [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** - Previously, `LOAD DATA` did not support concurrent data import, so the performance was not as good as expected. In TiDB v7.1.0, you can set the parameter `WITH thread=` for concurrent import, which can improve the performance of import by increasing concurrency. In the lab environment, the logical import performance of the test workload is nearly 4 times better than the previous version. + Before v7.1.0, `LOAD DATA` does not support concurrent data import, which limits its performance. Starting from TiDB v7.1.0, `LOAD DATA` supports concurrent data import. To enhance import performance, you can increase the concurrency using `WITH thread=`. In internal tests, the logical import performance of the test workload is nearly four times better than that in v7.0.0. For more information, see [documentation](/sql-statements/sql-statement-load-data.md). @@ -164,15 +164,15 @@ In v7.1.0, the key new features and improvements are as follows: ### Observability -* Add optimizer diagnostics information [#43122](https://github.com/pingcap/tidb/issues/43122) @[time-and-fate](https://github.com/time-and-fate) **tw:hfxsd** +* Enhance optimizer diagnostic information [#43122](https://github.com/pingcap/tidb/issues/43122) @[time-and-fate](https://github.com/time-and-fate) **tw:hfxsd** - Having sufficient information is key to SQL performance diagnostics. In v7.1.0, TiDB continues to add optimizer operation information to various diagnostic tools that can better explain how execution plans are selected. Such information can assist in troubleshooting SQL performance issues. The information includes: + Obtaining sufficient information is the key to SQL performance diagnostics. In v7.1.0, TiDB continues to add optimizer runtime information to various diagnostic tools, providing better insights into how execution plans are selected and assisting in troubleshooting SQL performance issues. The new information includes: - * In the output of [`PLAN REPLAYER`](/sql-plan-replayer.md), add `debug_trace.json`. - * In the output of [`EXPLAIN`](/explain-walkthrough.md), add partial details of the statistics for `operator info`. - * In the `Stats` field of [`slow-queries`](/identify-slow-queries.md), add partial statistics details. + * `debug_trace.json` in the output of [`PLAN REPLAYER`](/sql-plan-replayer.md). + * Partial statistics details for `operator info` in the output of [`EXPLAIN`](/explain-walkthrough.md). + * Partial statistics details in the `Stats` field of [`slow-queries`](/identify-slow-queries.md). - For more information, see [Use PLAN REPLAYER to Save and Restore the On-Site Information of a Cluster](/sql-plan-replayer.md), [EXPLAIN Walkthrough](/explain-walkthrough.md) and [Identify Slow Queries](/identify-slow-queries.md). + For more information, see [Use `PLAN REPLAYER` to save and restore the on-site information of a cluster](/sql-plan-replayer.md), [`EXPLAIN` walkthrough](/explain-walkthrough.md), and [Identify slow queries](/identify-slow-queries.md). ### Security @@ -201,7 +201,7 @@ In v7.1.0, the key new features and improvements are as follows: If you have upgraded TiFlash to v7.1.0, then during the TiDB upgrade to v7.1.0, TiDB cannot read the TiFlash system tables ([`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information- schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)). -* In the return value of [`SHOW LOAD DATA`](/sql-statements/sql-statement-show-load-data.md), the parameter `Loaded_File_Size` is deprecated and replaced with the parameter `Imported_Rows` **tw:hfxsd** +* In the outputs of [`SHOW LOAD DATA`](/sql-statements/sql-statement-show-load-data.md), the `Loaded_File_Size` parameter is deprecated and replaced with the `Imported_Rows` parameter. **tw:hfxsd** ### System variables From 98687c7bf6d37717e93d2ececb81702277d0e835 Mon Sep 17 00:00:00 2001 From: Aolin Date: Mon, 24 Apr 2023 16:56:14 +0800 Subject: [PATCH 25/42] update improvements > TiFlash --- releases/release-7.1.0.md | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 50f5048a158a4..e7c4915db9581 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -173,8 +173,6 @@ In v7.1.0, the key new features and improvements are as follows: * Partial statistics details in the `Stats` field of [`slow-queries`](/identify-slow-queries.md). For more information, see [Use `PLAN REPLAYER` to save and restore the on-site information of a cluster](/sql-plan-replayer.md), [`EXPLAIN` walkthrough](/explain-walkthrough.md), and [Identify slow queries](/identify-slow-queries.md). - - ### Security * Replace the interface used for querying TiFlash system table information [#6941](https://github.com/pingcap/tiflash/issues/6941) @[flowbehappy](https://github.com/flowbehappy) **tw:qiancai** @@ -252,8 +250,8 @@ In v7.1.0, the key new features and improvements are as follows: + TiFlash - - Improves TiFlash performance and stability in the disaggregated storage and compute architecture [#6882](https://github.com/pingcap/tiflash/issues/6882) @[JaySon-Huang](https://github.com/JaySon-Huang) @[breezewish](https://github.com/breezewish) @[JinheLin](https://github.com/JinheLin) **tw:qiancai** - - note [#issue](链接) @[贡献者 GitHub ID](链接) + - Improve TiFlash performance and stability in the disaggregated storage and compute architecture [#6882](https://github.com/pingcap/tiflash/issues/6882) @[JaySon-Huang](https://github.com/JaySon-Huang) @[breezewish](https://github.com/breezewish) @[JinheLin](https://github.com/JinheLin) **tw:qiancai** + - Support optimizing query performance in Semi Join or Anti Semi Join by selecting the smaller table as the build side of to improve query performance [#7280](https://github.com/pingcap/tiflash/issues/7280) @[yibin87](https://github.com/yibin87) + Tools From 309df5342a9dc163be8ff3e8b3df6f638a87c06c Mon Sep 17 00:00:00 2001 From: Aolin Date: Mon, 24 Apr 2023 17:05:03 +0800 Subject: [PATCH 26/42] update TOC Signed-off-by: Aolin --- TOC.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/TOC.md b/TOC.md index 25904234ae8c4..f0bf04f80e8cc 100644 --- a/TOC.md +++ b/TOC.md @@ -4,7 +4,7 @@ - [Docs Home](https://docs.pingcap.com/) - About TiDB - [TiDB Introduction](/overview.md) - - [TiDB 7.0 Release Notes](/releases/release-7.0.0.md) + - [TiDB 7.1 (upcoming) Release Notes](/releases/release-7.1.0.md) - [Features](/basic-features.md) - [MySQL Compatibility](/mysql-compatibility.md) - [TiDB Limitations](/tidb-limitations.md) @@ -960,6 +960,8 @@ - [Release Timeline](/releases/release-timeline.md) - [TiDB Versioning](/releases/versioning.md) - [TiDB Installation Packages](/binary-package.md) + - v7.1 (upcoming) + - [7.1.0](/releases/release-7.1.0.md) - v7.0 - [7.0.0-DMR](/releases/release-7.0.0.md) - v6.6 From 77d679a7f435677d04f4691a776b34b0289d3d8c Mon Sep 17 00:00:00 2001 From: Aolin Date: Tue, 25 Apr 2023 11:56:38 +0800 Subject: [PATCH 27/42] add highlights Signed-off-by: Aolin --- releases/release-7.1.0.md | 48 ++++++++++++++++++++++++++++----------- 1 file changed, 35 insertions(+), 13 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index e7c4915db9581..d644ddbe74a49 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -13,9 +13,38 @@ TiDB version: 7.1.0 (upcoming) In v7.1.0, the key new features and improvements are as follows: -## Feature details + + + + + + + + + + + + + + + + + + + + + + + + + + + + +
CategoryFeatureDescription
Scalability and PerformanceSession level non-prepared plan cache (GA)Support automatically reusing plan cache at the session level to reduce compilation and shorten the query time for the same SQL patterns without manually setting prepare statements in advance.
Load-based replica readIn a read hotspot scenario, TiDB can redirect read requests from the hotspot TiKV node to its replicas. This feature efficiently scatters read hotspots and optimizes the use of cluster resources. To control the threshold for triggering load-based replica read, you can adjust the system variable tidb_load_based_replica_read_threshold.
SQLMulti-valued index (GA)Support MySQL-compatible multi-valued indexes and enhance the JSON type to improve the compatibility with MySQL 8.0. This feature improves the efficiency of membership checks on multi-valued columns.
Generated columns (GA)The value of a generated column is calculated by the SQL expression in the column definition in real time. This feature pushes some application logic to the database level, thus improving efficiency. +
-### Scalability +## Feature details ### Performance @@ -55,11 +84,11 @@ In v7.1.0, the key new features and improvements are as follows: * Resource Control becomes generally available (GA) [#38825](https://github.com/pingcap/tidb/issues/38825) @[nolouch](https://github.com/nolouch) @[BornChanger](https://github.com/BornChanger) @[glorv](https://github.com/glorv) @[tiancaiamao](https://github.com/tiancaiamao) @[Connor1996](https://github.com/Connor1996) @[JmPotato](https://github.com/JmPotato) @[hnes](https://github.com/hnes) @[CabinfeverB](https://github.com/CabinfeverB) @[HuSharp](https://github.com/HuSharp) **tw:hfxsd** - TiDB enhances the resource control feature based on resource groups, which becomes GA in v7.1.0. This feature significantly improves the resource utilization efficiency and performance of TiDB clusters. The introduction of the resource control feature is a milestone for TiDB. You can divide a distributed database cluster into multiple logical units, map different database users to corresponding resource groups, and set the quota for each resource group as needed. When the cluster resources are limited, all resources used by sessions in the same resource group are limited to the quota. In this way, even if a resource group is over-consumed, the sessions in other resource groups are not affected. + TiDB enhances the resource control feature based on resource groups, which becomes GA in v7.1.0. This feature significantly improves the resource utilization efficiency and performance of TiDB clusters. The introduction of the resource control feature is a milestone for TiDB. You can divide a distributed database cluster into multiple logical units, map different database users to corresponding resource groups, and set the quota for each resource group as needed. When the cluster resources are limited, all resources used by sessions in the same resource group are limited to the quota. In this way, even if a resource group is over-consumed, the sessions in other resource groups are not affected. With this feature, you can combine multiple small and medium-sized applications from different systems into a single TiDB cluster. When the workload of an application grows larger, it does not affect the normal operation of other applications. When the system workload is low, busy applications can still be allocated the required system resources even if they exceed the set read and write quotas, which can achieve the maximum utilization of resources. In addition, the rational use of the resource control feature can reduce the number of clusters, ease the difficulty of operation and maintenance, and save management costs. - In TiDB v7.1.0, this feature introduces the ability to estimate system capacity based on actual workload or hardware deployment. The estimation ability provides you with a more accurate reference for capacity planning and assists you in better managing TiDB resource allocation to meet the stability needs of enterprise-level scenarios. + In TiDB v7.1.0, this feature introduces the ability to estimate system capacity based on actual workload or hardware deployment. The estimation ability provides you with a more accurate reference for capacity planning and assists you in better managing TiDB resource allocation to meet the stability needs of enterprise-level scenarios. For more information, see [documentation](/tidb-resource-control.md). @@ -81,8 +110,6 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/statistics.md#load-statistics). -### Availability - ### SQL * Support saving TiFlash query results using the `INSERT INTO SELECT` statement (GA) [#37515](https://github.com/pingcap/tidb/issues/37515) @[gengliqi](https://github.com/gengliqi) **tw:qiancai** @@ -119,7 +146,7 @@ In v7.1.0, the key new features and improvements are as follows: - Support importing data from S3 or GCS. - Support importing data from Parquet files. - - Support parsing the following character sets in the source file: `ascii`, `latin1`, `binary`, `gbk`, and `utf8mbd`. + - Support parsing the following character sets in the source file: `ascii`, `latin1`, `binary`, `gbk`, and `utf8mbd`. - Support setting `FIELDS DEFINED NULL BY` to convert the specified value in the source file to `NULL` for writing to the target table. - Support setting `bath_size` which specifies the number of rows inserted into the target table per batch, to improve write performance. - Support setting `detached` to allow the job to run in the background. @@ -173,18 +200,13 @@ In v7.1.0, the key new features and improvements are as follows: * Partial statistics details in the `Stats` field of [`slow-queries`](/identify-slow-queries.md). For more information, see [Use `PLAN REPLAYER` to save and restore the on-site information of a cluster](/sql-plan-replayer.md), [`EXPLAIN` walkthrough](/explain-walkthrough.md), and [Identify slow queries](/identify-slow-queries.md). + ### Security * Replace the interface used for querying TiFlash system table information [#6941](https://github.com/pingcap/tiflash/issues/6941) @[flowbehappy](https://github.com/flowbehappy) **tw:qiancai** Starting from v7.1.0, when providing the query service of [`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information-schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md) system tables for TiDB, TiFlash uses the gRPC port instead of the HTTP port, which avoids the security risks of the HTTP service. -### Data migration - -* TiCDC optimizes DDL replication operations [#8686](https://github.com/pingcap/tiflow/issues/8686) @[nongfushanquan](https://github.com/nongfushanquan) **tw:ran-huang** - - Before v7.1.0, when you perform a DDL operation that affected all rows on a large table (such as adding or deleting a column), the replication latency of TiCDC would significantly increase. Starting from v7.1.0, TiCDC optimizes this replication operation and reduces the replication latency to less than 10 seconds. This optimization mitigates the impact of DDL operations on downstream latency. - ## Compatibility changes > **Note:** From 6ea3785e2ffc7dbf647f09f3737d8b33e2aa706f Mon Sep 17 00:00:00 2001 From: Aolin Date: Tue, 25 Apr 2023 15:50:40 +0800 Subject: [PATCH 28/42] Apply suggestions from code review Co-authored-by: Ran --- releases/release-7.1.0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index d644ddbe74a49..77375dfdf54e6 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -219,7 +219,7 @@ In v7.1.0, the key new features and improvements are as follows: * To improve security, TiFlash deprecates the HTTP service port (default `8123`) and uses the gRPC port as a replacement **tw:qiancai** - If you have upgraded TiFlash to v7.1.0, then during the TiDB upgrade to v7.1.0, TiDB cannot read the TiFlash system tables ([`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information- schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)). + If you have upgraded TiFlash to v7.1.0, then during the TiDB upgrade to v7.1.0, TiDB cannot read the TiFlash system tables ([`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information-schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)). * In the outputs of [`SHOW LOAD DATA`](/sql-statements/sql-statement-show-load-data.md), the `Loaded_File_Size` parameter is deprecated and replaced with the `Imported_Rows` parameter. **tw:hfxsd** @@ -273,7 +273,7 @@ In v7.1.0, the key new features and improvements are as follows: + TiFlash - Improve TiFlash performance and stability in the disaggregated storage and compute architecture [#6882](https://github.com/pingcap/tiflash/issues/6882) @[JaySon-Huang](https://github.com/JaySon-Huang) @[breezewish](https://github.com/breezewish) @[JinheLin](https://github.com/JinheLin) **tw:qiancai** - - Support optimizing query performance in Semi Join or Anti Semi Join by selecting the smaller table as the build side of to improve query performance [#7280](https://github.com/pingcap/tiflash/issues/7280) @[yibin87](https://github.com/yibin87) + - Support optimizing query performance in Semi Join or Anti Semi Join by selecting the smaller table as the build side [#7280](https://github.com/pingcap/tiflash/issues/7280) @[yibin87](https://github.com/yibin87) + Tools From 3f11f4a3cd6a8fcbdfa09f0b6d49e2e11062c792 Mon Sep 17 00:00:00 2001 From: Aolin Date: Tue, 25 Apr 2023 16:41:18 +0800 Subject: [PATCH 29/42] Apply suggestions from code review --- releases/release-7.1.0.md | 1 + 1 file changed, 1 insertion(+) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 77375dfdf54e6..cb2695f1b70f7 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -237,6 +237,7 @@ In v7.1.0, the key new features and improvements are as follows: | [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710) | Newly added | Controls whether to enable [the distributed execution framework](/tidb-distributed-execution-framework.md). After enabling distributed execution, DDL, Import and other supported backend tasks will be jointly completed by multiple TiDB nodes in the cluster. This variable was renamed from `tidb_ddl_distribute_reorg`. | | [`tidb_enable_non_prepared_plan_cache_for_dml`](/system-variables.md#tidb_enable_non_prepared_plan_cache_for_dml-new-in-v710) | Newly added | Controls whether to enable the [Non-prepared plan cache](/sql-non-prepared-plan-cache.md) feature for DML statements. | | [`tidb_opt_fix_control`](/system-variables.md#tidb_opt_fix_control-new-in-v710) | Newly added | This variable provides a more fine-grained control over the optimizer and helps to prevent performance regression after upgrading caused by behavior changes in the optimizer. | +| [`tidb_plan_cache_max_plan_size`](/system-variables.md#tidb_plan_cache_max_plan_size-new-in-v710) | Newly added | Controls the maximum size of a plan that can be cached in prepared or non-prepared plan cache. | | [`tidb_prefer_broadcast_join_by_exchange_data_size`](/system-variables.md#tidb_prefer_broadcast_join_by_exchange_data_size-new-in-v710) | Newly added | Controls whether to use the algorithm with the minimum overhead of network transmission. If this variable is enabled, TiDB estimates the size of the data to be exchanged in the network using `Broadcast Hash Join` and `Shuffled Hash Join` respectively, and then chooses the one with the smaller size. [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) and [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_size-new-in-v50) will not take effect after this variable is enabled. | | [`tidb_session_plan_cache_size`](/system-variables.md#tidb_session_plan_cache_size-new-in-v710) | Newly added | Controls the maximum number of plans that can be cached. Prepared plan cache and non-prepared plan cache share the same cache. | From 4ee5d9e5ed2ce876dceaf56ab9c83e52857078d0 Mon Sep 17 00:00:00 2001 From: Grace Cai Date: Wed, 26 Apr 2023 10:34:41 +0800 Subject: [PATCH 30/42] Apply suggestions from code review Co-authored-by: Aolin --- releases/release-7.1.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index cb2695f1b70f7..fe7c2697c5600 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -50,7 +50,7 @@ In v7.1.0, the key new features and improvements are as follows: * TiFlash supports late materialization (GA) [#5829](https://github.com/pingcap/tiflash/issues/5829) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) **tw:qiancai** - In v7.0.0, TiFlash supports late materialization as an experiment feature for optimizing query performance. This feature is disabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `OFF`). when processing a `SELECT` statement with filter conditions (`WHERE` clause), TiFlash reads all the data from the columns required by the query, and then filters and aggregates the data based on the query conditions. When Late materialization is enabled, TiDB supports pushing down part of the filter conditions to the TableScan operator. That is, TiFlash first scans the column data related to the filter conditions that are pushed down to the TableScan operator, filters the rows that meet the condition, and then scans the other column data of these rows for further calculation, thereby reducing IO scans and computations of data processing. + In v7.0.0, TiFlash supports late materialization as an experimental feature for optimizing query performance. This feature is disabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `OFF`). when processing a `SELECT` statement with filter conditions (`WHERE` clause), TiFlash reads all the data from the columns required by the query, and then filters and aggregates the data based on the query conditions. When Late materialization is enabled, TiDB supports pushing down part of the filter conditions to the TableScan operator. That is, TiFlash first scans the column data related to the filter conditions that are pushed down to the TableScan operator, filters the rows that meet the condition, and then scans the other column data of these rows for further calculation, thereby reducing IO scans and computations of data processing. Starting from v7.1.0, the TiFlash late materialization feature is generally available and enabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `ON`). The TiDB optimizer decides which filters to be pushed down to the TableScan operator based on the statistics and the filter conditions of the query. From 70f5bfa97de2031e6a7ca45120ff2e4227c2d083 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 26 Apr 2023 10:41:39 +0800 Subject: [PATCH 31/42] Apply suggestions from code review Co-authored-by: Grace Cai --- releases/release-7.1.0.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index fe7c2697c5600..ccc57bbefda84 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -9,7 +9,7 @@ TiDB version: 7.1.0 (upcoming) > **Note:** > -> TiDB v7.1.0 is not yet available. This preview of the release notes provides insights into an upcoming release and is subject to change. The features outlined here are not guaranteed to be included in the final release. +> TiDB v7.1.0 is not yet available. This release note is a preview version to provide insights into upcoming features and is subject to change. The features outlined here are not guaranteed to be included in the final release. In v7.1.0, the key new features and improvements are as follows: @@ -29,16 +29,16 @@ In v7.1.0, the key new features and improvements are as follows: Load-based replica read - In a read hotspot scenario, TiDB can redirect read requests from the hotspot TiKV node to its replicas. This feature efficiently scatters read hotspots and optimizes the use of cluster resources. To control the threshold for triggering load-based replica read, you can adjust the system variable tidb_load_based_replica_read_threshold. + In a read hotspot scenario, TiDB can redirect read requests for a hotspot TiKV node to its replicas. This feature efficiently scatters read hotspots and optimizes the use of cluster resources. To control the threshold for triggering load-based replica read, you can adjust the system variable tidb_load_based_replica_read_threshold. SQL Multi-valued index (GA) - Support MySQL-compatible multi-valued indexes and enhance the JSON type to improve the compatibility with MySQL 8.0. This feature improves the efficiency of membership checks on multi-valued columns. + Support MySQL-compatible multi-valued indexes and enhance the JSON type to improve compatibility with MySQL 8.0. This feature improves the efficiency of membership checks on multi-valued columns. Generated columns (GA) - The value of a generated column is calculated by the SQL expression in the column definition in real time. This feature pushes some application logic to the database level, thus improving efficiency. + The value of a generated column is calculated by the SQL expression in the column definition in real time. This feature pushes some application logic to the database level, thus improving query efficiency. From abba8598a7d6674c2960f0ec943acdb332f493f8 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 26 Apr 2023 13:46:57 +0800 Subject: [PATCH 32/42] highlights: add links Signed-off-by: Aolin --- releases/release-7.1.0.md | 119 ++++---------------------------------- 1 file changed, 12 insertions(+), 107 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index ccc57bbefda84..d4398913b9b92 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -24,20 +24,20 @@ In v7.1.0, the key new features and improvements are as follows: Scalability and Performance - Session level non-prepared plan cache (GA) + Session level non-prepared plan cache (GA) Support automatically reusing plan cache at the session level to reduce compilation and shorten the query time for the same SQL patterns without manually setting prepare statements in advance. - Load-based replica read + Load-based replica read In a read hotspot scenario, TiDB can redirect read requests for a hotspot TiKV node to its replicas. This feature efficiently scatters read hotspots and optimizes the use of cluster resources. To control the threshold for triggering load-based replica read, you can adjust the system variable tidb_load_based_replica_read_threshold. SQL - Multi-valued index (GA) + Multi-valued index (GA) Support MySQL-compatible multi-valued indexes and enhance the JSON type to improve compatibility with MySQL 8.0. This feature improves the efficiency of membership checks on multi-valued columns. - Generated columns (GA) + Generated columns (GA) The value of a generated column is calculated by the SQL expression in the column definition in real time. This feature pushes some application logic to the database level, thus improving query efficiency. @@ -50,7 +50,7 @@ In v7.1.0, the key new features and improvements are as follows: * TiFlash supports late materialization (GA) [#5829](https://github.com/pingcap/tiflash/issues/5829) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) **tw:qiancai** - In v7.0.0, TiFlash supports late materialization as an experimental feature for optimizing query performance. This feature is disabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `OFF`). when processing a `SELECT` statement with filter conditions (`WHERE` clause), TiFlash reads all the data from the columns required by the query, and then filters and aggregates the data based on the query conditions. When Late materialization is enabled, TiDB supports pushing down part of the filter conditions to the TableScan operator. That is, TiFlash first scans the column data related to the filter conditions that are pushed down to the TableScan operator, filters the rows that meet the condition, and then scans the other column data of these rows for further calculation, thereby reducing IO scans and computations of data processing. + In v7.0.0, TiFlash supports late materialization as an experimental feature for optimizing query performance. This feature is disabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `OFF`). When processing a `SELECT` statement with filter conditions (`WHERE` clause), TiFlash reads all the data from the columns required by the query, and then filters and aggregates the data based on the query conditions. When Late materialization is enabled, TiDB supports pushing down part of the filter conditions to the TableScan operator. That is, TiFlash first scans the column data related to the filter conditions that are pushed down to the TableScan operator, filters the rows that meet the condition, and then scans the other column data of these rows for further calculation, thereby reducing IO scans and computations of data processing. Starting from v7.1.0, the TiFlash late materialization feature is generally available and enabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `ON`). The TiDB optimizer decides which filters to be pushed down to the TableScan operator based on the statistics and the filter conditions of the query. @@ -58,9 +58,9 @@ In v7.1.0, the key new features and improvements are as follows: * TiFlash supports automatically choosing an MPP Join algorithm according to the overhead of network transmission [#7084](https://github.com/pingcap/tiflash/issues/7084) @[solotzg](https://github.com/solotzg) **tw:qiancai** - The TiFlash MPP mode supports multiple Join algorithms. Before v7.1.0, TiDB determines whether the MPP mode uses the Broadcast Hash Join algorithm based on the [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) and [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) variables and the actual data volume. + The TiFlash MPP mode supports multiple Join algorithms. Before v7.1.0, TiDB determines whether the MPP mode uses the Broadcast Hash Join algorithm based on the [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) and [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_size-new-in-v50) variables and the actual data volume. - In v7.1.0, TiDB introduces the [`tidb_prefer_broadcast_join_by_exchange_data_size`](/system-variables.md#tidb_prefer_broadcast_join_by_exchange_data_size-new-in-v710) variable, which controls whether to choose the MPP Join algorithm based on the minimum overhead of network transmission. This variable is disabled by default, indicating that the default algorithm selection method remains the same as that before v7.1.0. You can set the variable to `ON` to enable it. When it is enabled, you no longer need to manually adjust the [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-introduced-from-v50-version) and [`tidb_ broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_count-introduced-from-v50-version) variables (both variables does not take effect at this time), TiDB automatically estimates the threshold of network transmission by different Join algorithms, and then chooses the algorithm with the smallest overhead overall, thus reducing network traffic and improving MPP query performance. + In v7.1.0, TiDB introduces the [`tidb_prefer_broadcast_join_by_exchange_data_size`](/system-variables.md#tidb_prefer_broadcast_join_by_exchange_data_size-new-in-v710) variable, which controls whether to choose the MPP Join algorithm based on the minimum overhead of network transmission. This variable is disabled by default, indicating that the default algorithm selection method remains the same as that before v7.1.0. You can set the variable to `ON` to enable it. When it is enabled, you no longer need to manually adjust the [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) and [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_size-new-in-v50) variables (both variables does not take effect at this time), TiDB automatically estimates the threshold of network transmission by different Join algorithms, and then chooses the algorithm with the smallest overhead overall, thus reducing network traffic and improving MPP query performance. For more information, see [documentation](/tiflash/use-tiflash-mpp-mode.md#algorithm-support-for-the-mpp-mode). @@ -100,7 +100,9 @@ In v7.1.0, the key new features and improvements are as follows: * Backup & Restore supports checkpoint restore [#issue](https://github.com/pingcap/tidb/issues/issue) @[Leavrth](https://github.com/Leavrth) **tw:Oreoxmt** - Snapshot restore or log restore might be interrupted due to recoverable errors, such as disk exhaustion and node crash. Before TiDB v7.1.0, the recovery progress before the interruption would be invalidated even after the error is addressed, and you need to start the restore from scratch. For large clusters, this incurs considerable extra cost. Starting from TiDB v7.1.0, Backup & Restore (BR) introduces the checkpoint restore feature, which enables you to continue an interrupted restore. This feature can retain most recovery progress of the interrupted restore. + Snapshot restore or log restore might be interrupted due to recoverable errors, such as disk exhaustion and node crash. Before TiDB v7.1.0, the recovery progress before the interruption would be invalidated even after the error is addressed, and you need to start the restore from scratch. For large clusters, this incurs considerable extra cost. + + Starting from TiDB v7.1.0, Backup & Restore (BR) introduces the checkpoint restore feature, which enables you to continue an interrupted restore. This feature can retain most recovery progress of the interrupted restore. For more information, see [documentation](/br/br-checkpoint-restore.md). @@ -213,8 +215,6 @@ In v7.1.0, the key new features and improvements are as follows: > > This section provides compatibility changes you need to know when you upgrade from v7.0.0 to the current version (v7.1.0). If you are upgrading from v6.6.0 or earlier versions to the current version, you might also need to check the compatibility changes introduced in intermediate versions. -### MySQL compatibility - ### Behavior changes * To improve security, TiFlash deprecates the HTTP service port (default `8123`) and uses the gRPC port as a replacement **tw:qiancai** @@ -228,8 +228,8 @@ In v7.1.0, the key new features and improvements are as follows: | Variable name | Change type | Description | |--------|------------------------------|------| | [`tidb_enable_tiflash_read_for_write_stmt`](/system-variables.md#tidb_enable_tiflash_read_for_write_stmt-new-in-v630) | Deprecated | Changes the default value from `OFF` to `ON`. When [`tidb_allow_mpp = ON`](/system-variables.md#tidb_allow_mpp-new-in-v50), the optimizer intelligently decides whether to push a query down to TiFlash based on the [SQL mode](/sql-mode.md) and the cost estimates of the TiFlash replica. | -| [`tidb_non_prepared_plan_cache_size`](/system-variables.md#tidb_non_prepared_plan_cache_size) | Deprecated | Starting from v7.1.0, this system variable is deprecated. You can use [`tidb_session_plan_cache_size`](#tidb_session_plan_cache_size-new-in-v710) to control the maximum number of plans that can be cached. | -| [`tidb_prepared_plan_cache_size`](/system-variables.md#tidb_prepared_plan_cache_size-new-in-v610) | Deprecated | Starting from v7.1.0, this system variable is deprecated. You can use [`tidb_session_plan_cache_size`](#tidb_session_plan_cache_size-new-in-v710) to control the maximum number of plans that can be cached. | +| [`tidb_non_prepared_plan_cache_size`](/system-variables.md#tidb_non_prepared_plan_cache_size) | Deprecated | Starting from v7.1.0, this system variable is deprecated. You can use [`tidb_session_plan_cache_size`](/system-variables.md#tidb_session_plan_cache_size-new-in-v710) to control the maximum number of plans that can be cached. | +| [`tidb_prepared_plan_cache_size`](/system-variables.md#tidb_prepared_plan_cache_size-new-in-v610) | Deprecated | Starting from v7.1.0, this system variable is deprecated. You can use [`tidb_session_plan_cache_size`](/system-variables.md#tidb_session_plan_cache_size-new-in-v710) to control the maximum number of plans that can be cached. | | `tidb_ddl_distribute_reorg` | Deleted | This variable is renamed to [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710). | | [`tidb_enable_non_prepared_plan_cache`](/system-variables.md#tidb_enable_non_prepared_plan_cache) | Modified | Changes the default value from `OFF` to `ON` after further tests, meaning that non-prepared plan cache is enabled by default. | | [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-new-in-v700) | Modified | Takes effect starting from v7.1.0 and controls the threshold for triggering load-based replica read. Changes the default value from `"0s"` to `"1s"` after further tests. | @@ -250,108 +250,13 @@ In v7.1.0, the key new features and improvements are as follows: | TiFlash | `http_port` | Deleted | Deprecates the HTTP service port (default `8123`). | | TiCDC | [`sink.enable-partition-separator`](/ticdc/ticdc-changefeed-config.md#cli-and-configuration-parameters-of-ticdc-changefeeds) | Modified | Changes the default value from `false` to `true` after further tests, meaning that partitions in a table are stored in separate directories by default. It is recommended that you keep the value as `true` to avoid the data loss issue. | -### Others - -## Deprecated feature - ## Improvements -+ TiDB - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - -+ TiKV - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - -+ PD - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - + TiFlash - Improve TiFlash performance and stability in the disaggregated storage and compute architecture [#6882](https://github.com/pingcap/tiflash/issues/6882) @[JaySon-Huang](https://github.com/JaySon-Huang) @[breezewish](https://github.com/breezewish) @[JinheLin](https://github.com/JinheLin) **tw:qiancai** - Support optimizing query performance in Semi Join or Anti Semi Join by selecting the smaller table as the build side [#7280](https://github.com/pingcap/tiflash/issues/7280) @[yibin87](https://github.com/yibin87) -+ Tools - - + Backup & Restore (BR) - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - + TiCDC - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - + TiDB Data Migration (DM) - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - + TiDB Lightning - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - + TiUP - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - -## Bug fixes - -+ TiDB - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - -+ TiKV - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - -+ PD - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - -+ TiFlash - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - -+ Tools - - + Backup & Restore (BR) - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - + TiCDC - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - + TiDB Data Migration (DM) - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - + TiDB Lightning - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - + TiUP - - - note [#issue](链接) @[贡献者 GitHub ID](链接) - - note [#issue](链接) @[贡献者 GitHub ID](链接) - ## Contributors We would like to thank the following contributors from the TiDB community: From 3224ee366e97b8e783ef8fa150bc30ef3f6b2a4b Mon Sep 17 00:00:00 2001 From: Ran Date: Wed, 26 Apr 2023 14:08:04 +0800 Subject: [PATCH 33/42] Update releases/release-7.1.0.md Co-authored-by: Aolin --- releases/release-7.1.0.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index d4398913b9b92..e482925ad7cb8 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -182,14 +182,14 @@ In v7.1.0, the key new features and improvements are as follows: Before TiDB v7.1.0, when a DDL task encounters a business peak period during execution, you can only manually cancel the DDL task to reduce its impact on the business. In v7.1.0, TiDB introduces pause and resume operations for DDL tasks. These operations let you pause DDL tasks during peak periods and resume them after the peak ends, thus avoiding any impact on your application workloads. - For example, you can pause and resume multiple DDL tasks using [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) or [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md): + For example, you can pause and resume multiple DDL tasks using `ADMIN PAUSE DDL JOBS` or `ADMIN RESUME DDL JOBS`: ```sql ADMIN PAUSE DDL JOBS 1,2; ADMIN RESUME DDL JOBS 1,2; ``` - For more information, see [`ADMIN PAUSE DDL JOBS`](/sql-statements/sql-statement-admin-pause-ddl.md) and [`ADMIN RESUME DDL JOBS`](/sql-statements/sql-statement-admin-resume-ddl.md). + For more information, see [documentation](/ddl-introduction.md#ddl-related-commands). ### Observability From 9df2513513ce1454f84fa24602dc966181126c8b Mon Sep 17 00:00:00 2001 From: Ran Date: Wed, 26 Apr 2023 14:09:52 +0800 Subject: [PATCH 34/42] Update releases/release-7.1.0.md Co-authored-by: Aolin --- releases/release-7.1.0.md | 1 + 1 file changed, 1 insertion(+) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index e482925ad7cb8..bda9745d33018 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -237,6 +237,7 @@ In v7.1.0, the key new features and improvements are as follows: | [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710) | Newly added | Controls whether to enable [the distributed execution framework](/tidb-distributed-execution-framework.md). After enabling distributed execution, DDL, Import and other supported backend tasks will be jointly completed by multiple TiDB nodes in the cluster. This variable was renamed from `tidb_ddl_distribute_reorg`. | | [`tidb_enable_non_prepared_plan_cache_for_dml`](/system-variables.md#tidb_enable_non_prepared_plan_cache_for_dml-new-in-v710) | Newly added | Controls whether to enable the [Non-prepared plan cache](/sql-non-prepared-plan-cache.md) feature for DML statements. | | [`tidb_opt_fix_control`](/system-variables.md#tidb_opt_fix_control-new-in-v710) | Newly added | This variable provides a more fine-grained control over the optimizer and helps to prevent performance regression after upgrading caused by behavior changes in the optimizer. | +| [`tidb_plan_cache_invalidation_on_fresh_stats `](/system-variables.md#tidb_plan_cache_invalidation_on_fresh_stats-new-in-v710) | Newly added | Controls whether to invalidate the plan cache automatically when statistics on related tables are updated. | | [`tidb_plan_cache_max_plan_size`](/system-variables.md#tidb_plan_cache_max_plan_size-new-in-v710) | Newly added | Controls the maximum size of a plan that can be cached in prepared or non-prepared plan cache. | | [`tidb_prefer_broadcast_join_by_exchange_data_size`](/system-variables.md#tidb_prefer_broadcast_join_by_exchange_data_size-new-in-v710) | Newly added | Controls whether to use the algorithm with the minimum overhead of network transmission. If this variable is enabled, TiDB estimates the size of the data to be exchanged in the network using `Broadcast Hash Join` and `Shuffled Hash Join` respectively, and then chooses the one with the smaller size. [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) and [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_size-new-in-v50) will not take effect after this variable is enabled. | | [`tidb_session_plan_cache_size`](/system-variables.md#tidb_session_plan_cache_size-new-in-v710) | Newly added | Controls the maximum number of plans that can be cached. Prepared plan cache and non-prepared plan cache share the same cache. | From 4c4c57ea6a04298b8479a14239f6a4c7439a3db6 Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 26 Apr 2023 19:52:54 +0800 Subject: [PATCH 35/42] remove tw id and fix format Signed-off-by: Aolin --- releases/release-7.1.0.md | 49 +++++++++++++++++++-------------------- 1 file changed, 24 insertions(+), 25 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index bda9745d33018..13cda2d1674e3 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -38,8 +38,7 @@ In v7.1.0, the key new features and improvements are as follows: Generated columns (GA) - The value of a generated column is calculated by the SQL expression in the column definition in real time. This feature pushes some application logic to the database level, thus improving query efficiency. - + The value of a generated column is calculated by the SQL expression in the column definition in real time. This feature pushes some application logic to the database level, thus improving query efficiency. @@ -48,7 +47,7 @@ In v7.1.0, the key new features and improvements are as follows: ### Performance -* TiFlash supports late materialization (GA) [#5829](https://github.com/pingcap/tiflash/issues/5829) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) **tw:qiancai** +* TiFlash supports late materialization (GA) [#5829](https://github.com/pingcap/tiflash/issues/5829) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) In v7.0.0, TiFlash supports late materialization as an experimental feature for optimizing query performance. This feature is disabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `OFF`). When processing a `SELECT` statement with filter conditions (`WHERE` clause), TiFlash reads all the data from the columns required by the query, and then filters and aggregates the data based on the query conditions. When Late materialization is enabled, TiDB supports pushing down part of the filter conditions to the TableScan operator. That is, TiFlash first scans the column data related to the filter conditions that are pushed down to the TableScan operator, filters the rows that meet the condition, and then scans the other column data of these rows for further calculation, thereby reducing IO scans and computations of data processing. @@ -56,7 +55,7 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/tiflash/tiflash-late-materialization.md). -* TiFlash supports automatically choosing an MPP Join algorithm according to the overhead of network transmission [#7084](https://github.com/pingcap/tiflash/issues/7084) @[solotzg](https://github.com/solotzg) **tw:qiancai** +* TiFlash supports automatically choosing an MPP Join algorithm according to the overhead of network transmission [#7084](https://github.com/pingcap/tiflash/issues/7084) @[solotzg](https://github.com/solotzg) The TiFlash MPP mode supports multiple Join algorithms. Before v7.1.0, TiDB determines whether the MPP mode uses the Broadcast Hash Join algorithm based on the [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) and [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_size-new-in-v50) variables and the actual data volume. @@ -64,13 +63,13 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/tiflash/use-tiflash-mpp-mode.md#algorithm-support-for-the-mpp-mode). -* Support load-based replica read to mitigate read hotspots [#14151](https://github.com/tikv/tikv/issues/14151) @[sticnarf](https://github.com/sticnarf) @[you06](https://github.com/you06) **tw:Oreoxmt** +* Support load-based replica read to mitigate read hotspots [#14151](https://github.com/tikv/tikv/issues/14151) @[sticnarf](https://github.com/sticnarf) @[you06](https://github.com/you06) In a read hotspot scenario, the hotspot TiKV node cannot process read requests in time, resulting in the read requests queuing. However, not all TiKV resources are exhausted at this time. To reduce latency, TiDB v7.1.0 introduces the load-based replica read feature, which allows TiDB to read data from other TiKV nodes without queuing on the hotspot TiKV node. You can control the queue length of read requests using the [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-new-in-v700) system variable. When the estimated queue time of the leader node exceeds this threshold, TiDB prioritizes reading data from follower nodes. This feature can improve read throughput by 70% to 200% in a read hotspot scenario compared to not scattering read hotspots. For more information, see [documentation](/troubleshoot-hot-spot-issues.md#scatter-read-hotspots). -* Support caching execution plans for non-prepared statements (GA) [#36598](https://github.com/pingcap/tidb/issues/36598) @[qw4990](https://github.com/qw4990) **tw:Oreoxmt** +* Support caching execution plans for non-prepared statements (GA) [#36598](https://github.com/pingcap/tidb/issues/36598) @[qw4990](https://github.com/qw4990) TiDB v7.0.0 introduces non-prepared plan cache as an experimental feature to improve the load capacity of concurrent OLTP. In v7.1.0, this feature is generally available, enabled by default, and supports caching more SQL statements. @@ -82,7 +81,7 @@ In v7.1.0, the key new features and improvements are as follows: ### Reliability -* Resource Control becomes generally available (GA) [#38825](https://github.com/pingcap/tidb/issues/38825) @[nolouch](https://github.com/nolouch) @[BornChanger](https://github.com/BornChanger) @[glorv](https://github.com/glorv) @[tiancaiamao](https://github.com/tiancaiamao) @[Connor1996](https://github.com/Connor1996) @[JmPotato](https://github.com/JmPotato) @[hnes](https://github.com/hnes) @[CabinfeverB](https://github.com/CabinfeverB) @[HuSharp](https://github.com/HuSharp) **tw:hfxsd** +* Resource Control becomes generally available (GA) [#38825](https://github.com/pingcap/tidb/issues/38825) @[nolouch](https://github.com/nolouch) @[BornChanger](https://github.com/BornChanger) @[glorv](https://github.com/glorv) @[tiancaiamao](https://github.com/tiancaiamao) @[Connor1996](https://github.com/Connor1996) @[JmPotato](https://github.com/JmPotato) @[hnes](https://github.com/hnes) @[CabinfeverB](https://github.com/CabinfeverB) @[HuSharp](https://github.com/HuSharp) TiDB enhances the resource control feature based on resource groups, which becomes GA in v7.1.0. This feature significantly improves the resource utilization efficiency and performance of TiDB clusters. The introduction of the resource control feature is a milestone for TiDB. You can divide a distributed database cluster into multiple logical units, map different database users to corresponding resource groups, and set the quota for each resource group as needed. When the cluster resources are limited, all resources used by sessions in the same resource group are limited to the quota. In this way, even if a resource group is over-consumed, the sessions in other resource groups are not affected. @@ -92,13 +91,13 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/tidb-resource-control.md). -* Support the checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630) to improve fault tolerance and automatic recovery capability [#42164](https://github.com/pingcap/tidb/issues/42164) @[tangenta](https://github.com/tangenta) **tw:ran-huang** +* Support the checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630) to improve fault tolerance and automatic recovery capability [#42164](https://github.com/pingcap/tidb/issues/42164) @[tangenta](https://github.com/tangenta) TiDB v7.1.0 introduces a checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630), which significantly improves the fault tolerance and automatic recovery capability of Fast Online DDL. Even in the case of TiDB DDL owner switching, TiDB can still periodically record and synchronize the progress of DDL statements, allowing the new TiDB DDL owner to execute the ongoing DDL statements in Fast Online DDL mode without manually canceling and re-executing the statements. The checkpoint mechanism makes the DDL execution more stable and efficient. For more information, see [documentation](/ddl-introduction.md). -* Backup & Restore supports checkpoint restore [#issue](https://github.com/pingcap/tidb/issues/issue) @[Leavrth](https://github.com/Leavrth) **tw:Oreoxmt** +* Backup & Restore supports checkpoint restore [#issue](https://github.com/pingcap/tidb/issues/issue) @[Leavrth](https://github.com/Leavrth) Snapshot restore or log restore might be interrupted due to recoverable errors, such as disk exhaustion and node crash. Before TiDB v7.1.0, the recovery progress before the interruption would be invalidated even after the error is addressed, and you need to start the restore from scratch. For large clusters, this incurs considerable extra cost. @@ -106,7 +105,7 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/br/br-checkpoint-restore.md). -* Optimize the strategy of loading statistics [#42160](https://github.com/pingcap/tidb/issues/42160) @[xuyifangreeneyes](https://github.com/xuyifangreeneyes) **tw:hfxsd** +* Optimize the strategy of loading statistics [#42160](https://github.com/pingcap/tidb/issues/42160) @[xuyifangreeneyes](https://github.com/xuyifangreeneyes) Enabling synchronous loading of statistics can significantly reduce the number of statistics that must be loaded during startup, thus improving the speed of loading statistics. This feature increases the stability of TiDB in complex runtime environments and reduces the impact of individual TiDB nodes restart on the overall service. @@ -114,7 +113,7 @@ In v7.1.0, the key new features and improvements are as follows: ### SQL -* Support saving TiFlash query results using the `INSERT INTO SELECT` statement (GA) [#37515](https://github.com/pingcap/tidb/issues/37515) @[gengliqi](https://github.com/gengliqi) **tw:qiancai** +* Support saving TiFlash query results using the `INSERT INTO SELECT` statement (GA) [#37515](https://github.com/pingcap/tidb/issues/37515) @[gengliqi](https://github.com/gengliqi) Starting from v6.5.0, TiDB supports pushing down the `SELECT` clause (analytical query) of the `INSERT INTO SELECT` statement to TiFlash. In this way, you can easily save the TiFlash query result to a TiDB table specified by `INSERT INTO` for further analysis, which takes effect as result caching (that is, result materialization). @@ -122,7 +121,7 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/tiflash/tiflash-results-materialization.md). -* MySQL-compatible multi-valued indexes become generally available (GA) [#39592](https://github.com/pingcap/tidb/issues/39592) @[xiongjiwei](https://github.com/xiongjiwei) @[qw4990](https://github.com/qw4990) @[YangKeao](https://github.com/YangKeao) **tw:ran-huang** +* MySQL-compatible multi-valued indexes become generally available (GA) [#39592](https://github.com/pingcap/tidb/issues/39592) @[xiongjiwei](https://github.com/xiongjiwei) @[qw4990](https://github.com/qw4990) @[YangKeao](https://github.com/YangKeao) Filtering the values of an array in a JSON column is a common operation, but normal indexes cannot help speed up such an operation. Creating a multi-valued index on an array can greatly improve filtering performance. If an array in the JSON column has a multi-valued index, you can use the multi-valued index to filter retrieval conditions in `MEMBER OF()`, `JSON_CONTAINS()`, and `JSON_OVERLAPS()` functions, thereby reducing I/O consumption and improving operation speed. @@ -130,19 +129,19 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/sql-statements/sql-statement-create-index.md#multi-valued-index). -* Improve the partition management for Hash and Key partitioned tables [#42728](https://github.com/pingcap/tidb/issues/42728) @[mjonss](https://github.com/mjonss) **tw:qiancai** +* Improve the partition management for Hash and Key partitioned tables [#42728](https://github.com/pingcap/tidb/issues/42728) @[mjonss](https://github.com/mjonss) Before v7.1.0, Hash and Key partitioned tables in TiDB only support the `TRUNCATE PARTITION` partition management statement. Starting from v7.1.0, Hash and Key partitioned tables also support `ADD PARTITION` and `COALESCE PARTITION` partition management statements. Therefore, you can flexibly adjust the number of partitions in Hash and Key partitioned tables as needed. For example, you can increase the number of partitions with the `ADD PARTITION` statement, or decrease the number of partitions with the `COALESCE PARTITION` statement. For more information, see [documentation](/partitioned-table.md#manage-hash-and-key-partitions). -* The syntax of Range INTERVAL partitioning becomes generally available (GA) [#35683](https://github.com/pingcap/tidb/issues/35683) @[mjonss](https://github.com/mjonss) **tw:qiancai** +* The syntax of Range INTERVAL partitioning becomes generally available (GA) [#35683](https://github.com/pingcap/tidb/issues/35683) @[mjonss](https://github.com/mjonss) The syntax of Range INTERVAL partitioning (introduced in v6.3.0) becomes GA. With this syntax, you can define Range partitioning by a desired interval without enumerating all partitions, which drastically reduces the length of Range partitioning DDL statements. The syntax is equivalent to that of the original Range partitioning. For more information, see [documentation](/partitioned-table#range-interval-partitioning). -* Some `LOAD DATA` features become GA [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** +* Some `LOAD DATA` features become GA [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) In TiDB v7.1.0, the following `LOAD DATA` features become GA: @@ -156,19 +155,19 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/sql-statements/sql-statement-load-data.md). -* `LOAD DATA` integrates with TiDB Lightning Physical Import Mode to improve import performance (experimental) [#42930](https://github.com/pingcap/tidb/issues/42930) @[D3Hunter](https://github.com/D3Hunter) **tw:hfxsd** +* `LOAD DATA` integrates with TiDB Lightning Physical Import Mode to improve import performance (experimental) [#42930](https://github.com/pingcap/tidb/issues/42930) @[D3Hunter](https://github.com/D3Hunter) `LOAD DATA` integrates with TiDB Lightning Physical Import Mode. You can enable it by setting `WITH import_mode = 'PHYSICAL'`. This feature can improve the performance of importing data exponentially compared to Logical Import Mode. For more information, see [documentation](/sql-statements/sql-statement-load-data.md). -* `LOAD DATA` supports concurrent data import to improve import performance (experimental) [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) **tw:hfxsd** +* `LOAD DATA` supports concurrent data import to improve import performance (experimental) [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) Before v7.1.0, `LOAD DATA` does not support concurrent data import, which limits its performance. Starting from TiDB v7.1.0, `LOAD DATA` supports concurrent data import. To enhance import performance, you can increase the concurrency using `WITH thread=`. In internal tests, the logical import performance of the test workload is nearly four times better than that in v7.0.0. For more information, see [documentation](/sql-statements/sql-statement-load-data.md). -* Generated columns become generally available (GA) @[bb7133](https://github.com/bb7133) **tw:ran-huang** +* Generated columns become generally available (GA) @[bb7133](https://github.com/bb7133) Generated columns are a valuable feature for database. When creating a table, you can define that the value of a column is calculated based on the values of other columns in the table, rather than being explicitly inserted or updated by users. This generated column can be either a virtual column or a stored column. TiDB has supported MySQL-compatible generated columns since earlier versions, and this feature becomes GA in v7.1.0. @@ -178,7 +177,7 @@ In v7.1.0, the key new features and improvements are as follows: ### DB operations -* DDL tasks support pause and resume operations (experimental) [#18015](https://github.com/pingcap/tidb/issues/18015) @[godouxm](https://github.com/godouxm) **tw:ran-huang** +* DDL tasks support pause and resume operations (experimental) [#18015](https://github.com/pingcap/tidb/issues/18015) @[godouxm](https://github.com/godouxm) Before TiDB v7.1.0, when a DDL task encounters a business peak period during execution, you can only manually cancel the DDL task to reduce its impact on the business. In v7.1.0, TiDB introduces pause and resume operations for DDL tasks. These operations let you pause DDL tasks during peak periods and resume them after the peak ends, thus avoiding any impact on your application workloads. @@ -193,7 +192,7 @@ In v7.1.0, the key new features and improvements are as follows: ### Observability -* Enhance optimizer diagnostic information [#43122](https://github.com/pingcap/tidb/issues/43122) @[time-and-fate](https://github.com/time-and-fate) **tw:hfxsd** +* Enhance optimizer diagnostic information [#43122](https://github.com/pingcap/tidb/issues/43122) @[time-and-fate](https://github.com/time-and-fate) Obtaining sufficient information is the key to SQL performance diagnostics. In v7.1.0, TiDB continues to add optimizer runtime information to various diagnostic tools, providing better insights into how execution plans are selected and assisting in troubleshooting SQL performance issues. The new information includes: @@ -205,7 +204,7 @@ In v7.1.0, the key new features and improvements are as follows: ### Security -* Replace the interface used for querying TiFlash system table information [#6941](https://github.com/pingcap/tiflash/issues/6941) @[flowbehappy](https://github.com/flowbehappy) **tw:qiancai** +* Replace the interface used for querying TiFlash system table information [#6941](https://github.com/pingcap/tiflash/issues/6941) @[flowbehappy](https://github.com/flowbehappy) Starting from v7.1.0, when providing the query service of [`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information-schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md) system tables for TiDB, TiFlash uses the gRPC port instead of the HTTP port, which avoids the security risks of the HTTP service. @@ -217,11 +216,11 @@ In v7.1.0, the key new features and improvements are as follows: ### Behavior changes -* To improve security, TiFlash deprecates the HTTP service port (default `8123`) and uses the gRPC port as a replacement **tw:qiancai** +* To improve security, TiFlash deprecates the HTTP service port (default `8123`) and uses the gRPC port as a replacement If you have upgraded TiFlash to v7.1.0, then during the TiDB upgrade to v7.1.0, TiDB cannot read the TiFlash system tables ([`INFORMATION_SCHEMA.TIFLASH_TABLES`](/information-schema/information-schema-tiflash-tables.md) and [`INFORMATION_SCHEMA.TIFLASH_SEGMENTS`](/information-schema/information-schema-tiflash-segments.md)). -* In the outputs of [`SHOW LOAD DATA`](/sql-statements/sql-statement-show-load-data.md), the `Loaded_File_Size` parameter is deprecated and replaced with the `Imported_Rows` parameter. **tw:hfxsd** +* In the outputs of [`SHOW LOAD DATA`](/sql-statements/sql-statement-show-load-data.md), the `Loaded_File_Size` parameter is deprecated and replaced with the `Imported_Rows` parameter. ### System variables @@ -234,7 +233,7 @@ In v7.1.0, the key new features and improvements are as follows: | [`tidb_enable_non_prepared_plan_cache`](/system-variables.md#tidb_enable_non_prepared_plan_cache) | Modified | Changes the default value from `OFF` to `ON` after further tests, meaning that non-prepared plan cache is enabled by default. | | [`tidb_load_based_replica_read_threshold`](/system-variables.md#tidb_load_based_replica_read_threshold-new-in-v700) | Modified | Takes effect starting from v7.1.0 and controls the threshold for triggering load-based replica read. Changes the default value from `"0s"` to `"1s"` after further tests. | | [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) | Modified | Changes the default value from `OFF` to `ON`, meaning that the TiFlash late materialization feature is enabled by default. | -| [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710) | Newly added | Controls whether to enable [the distributed execution framework](/tidb-distributed-execution-framework.md). After enabling distributed execution, DDL, Import and other supported backend tasks will be jointly completed by multiple TiDB nodes in the cluster. This variable was renamed from `tidb_ddl_distribute_reorg`. | +| [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710) | Newly added | Controls whether to enable the distributed execution framework. After enabling distributed execution, DDL, Import and other supported backend tasks will be jointly completed by multiple TiDB nodes in the cluster. This variable was renamed from `tidb_ddl_distribute_reorg`. | | [`tidb_enable_non_prepared_plan_cache_for_dml`](/system-variables.md#tidb_enable_non_prepared_plan_cache_for_dml-new-in-v710) | Newly added | Controls whether to enable the [Non-prepared plan cache](/sql-non-prepared-plan-cache.md) feature for DML statements. | | [`tidb_opt_fix_control`](/system-variables.md#tidb_opt_fix_control-new-in-v710) | Newly added | This variable provides a more fine-grained control over the optimizer and helps to prevent performance regression after upgrading caused by behavior changes in the optimizer. | | [`tidb_plan_cache_invalidation_on_fresh_stats `](/system-variables.md#tidb_plan_cache_invalidation_on_fresh_stats-new-in-v710) | Newly added | Controls whether to invalidate the plan cache automatically when statistics on related tables are updated. | @@ -255,7 +254,7 @@ In v7.1.0, the key new features and improvements are as follows: + TiFlash - - Improve TiFlash performance and stability in the disaggregated storage and compute architecture [#6882](https://github.com/pingcap/tiflash/issues/6882) @[JaySon-Huang](https://github.com/JaySon-Huang) @[breezewish](https://github.com/breezewish) @[JinheLin](https://github.com/JinheLin) **tw:qiancai** + - Improve TiFlash performance and stability in the disaggregated storage and compute architecture [#6882](https://github.com/pingcap/tiflash/issues/6882) @[JaySon-Huang](https://github.com/JaySon-Huang) @[breezewish](https://github.com/breezewish) @[JinheLin](https://github.com/JinheLin) - Support optimizing query performance in Semi Join or Anti Semi Join by selecting the smaller table as the build side [#7280](https://github.com/pingcap/tiflash/issues/7280) @[yibin87](https://github.com/yibin87) ## Contributors From cba2c9236eaa9eb5ae91f18faae54f54ae5bb26f Mon Sep 17 00:00:00 2001 From: Aolin Date: Wed, 26 Apr 2023 19:59:17 +0800 Subject: [PATCH 36/42] make ci happy --- releases/release-7.1.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 13cda2d1674e3..0c60bd28371dc 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -139,7 +139,7 @@ In v7.1.0, the key new features and improvements are as follows: The syntax of Range INTERVAL partitioning (introduced in v6.3.0) becomes GA. With this syntax, you can define Range partitioning by a desired interval without enumerating all partitions, which drastically reduces the length of Range partitioning DDL statements. The syntax is equivalent to that of the original Range partitioning. - For more information, see [documentation](/partitioned-table#range-interval-partitioning). + For more information, see [documentation](/partitioned-table.md#range-interval-partitioning). * Some `LOAD DATA` features become GA [#40499](https://github.com/pingcap/tidb/issues/40499) @[lance6716](https://github.com/lance6716) From edd380abd86a7c5cd9abd38fb73afaa2a21f4057 Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 27 Apr 2023 10:29:59 +0800 Subject: [PATCH 37/42] Apply suggestions from code review Co-authored-by: Grace Cai Co-authored-by: Ran --- releases/release-7.1.0.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 0c60bd28371dc..99906f71ab7ea 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -29,7 +29,7 @@ In v7.1.0, the key new features and improvements are as follows: Load-based replica read - In a read hotspot scenario, TiDB can redirect read requests for a hotspot TiKV node to its replicas. This feature efficiently scatters read hotspots and optimizes the use of cluster resources. To control the threshold for triggering load-based replica read, you can adjust the system variable tidb_load_based_replica_read_threshold. + In a read hotspot scenario, TiDB can redirect read requests for a hotspot TiKV node to its replicas. This feature efficiently scatters read hotspots and optimizes the use of cluster resources. To control the threshold for triggering load-based replica read, you can adjust the system variable tidb_load_based_replica_read_threshold. SQL @@ -93,9 +93,9 @@ In v7.1.0, the key new features and improvements are as follows: * Support the checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630) to improve fault tolerance and automatic recovery capability [#42164](https://github.com/pingcap/tidb/issues/42164) @[tangenta](https://github.com/tangenta) - TiDB v7.1.0 introduces a checkpoint mechanism for [Fast Online DDL](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630), which significantly improves the fault tolerance and automatic recovery capability of Fast Online DDL. Even in the case of TiDB DDL owner switching, TiDB can still periodically record and synchronize the progress of DDL statements, allowing the new TiDB DDL owner to execute the ongoing DDL statements in Fast Online DDL mode without manually canceling and re-executing the statements. The checkpoint mechanism makes the DDL execution more stable and efficient. + TiDB v7.1.0 introduces a checkpoint mechanism for [Fast Online DDL](/ddl-introduction.md), which significantly improves the fault tolerance and automatic recovery capability of Fast Online DDL. Even if the TiDB owner node is restarted or changed due to failures, TiDB can still recover progress from checkpoints that are automatically updated on a regular basis, making the DDL execution more stable and efficient. - For more information, see [documentation](/ddl-introduction.md). + For more information, see [documentation](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630). * Backup & Restore supports checkpoint restore [#issue](https://github.com/pingcap/tidb/issues/issue) @[Leavrth](https://github.com/Leavrth) @@ -236,7 +236,7 @@ In v7.1.0, the key new features and improvements are as follows: | [`tidb_enable_dist_task`](/system-variables.md#tidb_enable_dist_task-new-in-v710) | Newly added | Controls whether to enable the distributed execution framework. After enabling distributed execution, DDL, Import and other supported backend tasks will be jointly completed by multiple TiDB nodes in the cluster. This variable was renamed from `tidb_ddl_distribute_reorg`. | | [`tidb_enable_non_prepared_plan_cache_for_dml`](/system-variables.md#tidb_enable_non_prepared_plan_cache_for_dml-new-in-v710) | Newly added | Controls whether to enable the [Non-prepared plan cache](/sql-non-prepared-plan-cache.md) feature for DML statements. | | [`tidb_opt_fix_control`](/system-variables.md#tidb_opt_fix_control-new-in-v710) | Newly added | This variable provides a more fine-grained control over the optimizer and helps to prevent performance regression after upgrading caused by behavior changes in the optimizer. | -| [`tidb_plan_cache_invalidation_on_fresh_stats `](/system-variables.md#tidb_plan_cache_invalidation_on_fresh_stats-new-in-v710) | Newly added | Controls whether to invalidate the plan cache automatically when statistics on related tables are updated. | +| [`tidb_plan_cache_invalidation_on_fresh_stats`](/system-variables.md#tidb_plan_cache_invalidation_on_fresh_stats-new-in-v710) | Newly added | Controls whether to invalidate the plan cache automatically when statistics on related tables are updated. | | [`tidb_plan_cache_max_plan_size`](/system-variables.md#tidb_plan_cache_max_plan_size-new-in-v710) | Newly added | Controls the maximum size of a plan that can be cached in prepared or non-prepared plan cache. | | [`tidb_prefer_broadcast_join_by_exchange_data_size`](/system-variables.md#tidb_prefer_broadcast_join_by_exchange_data_size-new-in-v710) | Newly added | Controls whether to use the algorithm with the minimum overhead of network transmission. If this variable is enabled, TiDB estimates the size of the data to be exchanged in the network using `Broadcast Hash Join` and `Shuffled Hash Join` respectively, and then chooses the one with the smaller size. [`tidb_broadcast_join_threshold_count`](/system-variables.md#tidb_broadcast_join_threshold_count-new-in-v50) and [`tidb_broadcast_join_threshold_size`](/system-variables.md#tidb_broadcast_join_threshold_size-new-in-v50) will not take effect after this variable is enabled. | | [`tidb_session_plan_cache_size`](/system-variables.md#tidb_session_plan_cache_size-new-in-v710) | Newly added | Controls the maximum number of plans that can be cached. Prepared plan cache and non-prepared plan cache share the same cache. | @@ -248,7 +248,7 @@ In v7.1.0, the key new features and improvements are as follows: | TiDB | [`lite-init-stats`](/tidb-configuration-file.md#lite-init-stats-new-in-v710) | Newly added | Controls whether to use lightweight statistics initialization during TiDB startup. | | PD | [`store-limit-version`](/pd-configuration-file.md#store-limit-version-new-in-v710) | Newly added | Controls the mode of store limit. Value options are `"v1"` and `"v2"`. | | TiFlash | `http_port` | Deleted | Deprecates the HTTP service port (default `8123`). | -| TiCDC | [`sink.enable-partition-separator`](/ticdc/ticdc-changefeed-config.md#cli-and-configuration-parameters-of-ticdc-changefeeds) | Modified | Changes the default value from `false` to `true` after further tests, meaning that partitions in a table are stored in separate directories by default. It is recommended that you keep the value as `true` to avoid the data loss issue. | +| TiCDC | [`sink.enable-partition-separator`](/ticdc/ticdc-changefeed-config.md#cli-and-configuration-parameters-of-ticdc-changefeeds) | Modified | Changes the default value from `false` to `true` after further tests, meaning that partitions in a table are stored in separate directories by default. It is recommended that you keep the value as `true` to avoid the potential issue of data loss during replication of partitioned tables to storage services. | ## Improvements From 499a303094a5a4752614bf54bba2bd5f72f8d39d Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 27 Apr 2023 11:10:05 +0800 Subject: [PATCH 38/42] Apply suggestions from code review Co-authored-by: Sam Dillard --- releases/release-7.1.0.md | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 99906f71ab7ea..34d58a2c983db 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -24,13 +24,18 @@ In v7.1.0, the key new features and improvements are as follows: Scalability and Performance - Session level non-prepared plan cache (GA) - Support automatically reusing plan cache at the session level to reduce compilation and shorten the query time for the same SQL patterns without manually setting prepare statements in advance. + Session-level plan cache for non-prepared plans (GA) + Support automatically reusing plan cache at the session level to remove query planning time, reducing query time for repeat SQL patterns without manually setting prepare statements in advance. Load-based replica read In a read hotspot scenario, TiDB can redirect read requests for a hotspot TiKV node to its replicas. This feature efficiently scatters read hotspots and optimizes the use of cluster resources. To control the threshold for triggering load-based replica read, you can adjust the system variable tidb_load_based_replica_read_threshold. + + Reliability and availability + Resource control by Resource Groups (GA) + Support resource management based on resource groups, which maps database users to the corresponding resource groups and sets quotas for each resource group based on actual needs. + SQL Multi-valued index (GA) @@ -38,7 +43,7 @@ In v7.1.0, the key new features and improvements are as follows: Generated columns (GA) - The value of a generated column is calculated by the SQL expression in the column definition in real time. This feature pushes some application logic to the database level, thus improving query efficiency. + Values in a generated column are calculated by a SQL expression in the column definition in real time. This feature pushes some application logic to the database level, thus improving query efficiency. From b0c8e22b7888a15f95e365dc51cf9b6a6c847890 Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 27 Apr 2023 11:57:05 +0800 Subject: [PATCH 39/42] Apply suggestions from code review Co-authored-by: Grace Cai --- releases/release-7.1.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 34d58a2c983db..ee4d3eed2a114 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -24,7 +24,7 @@ In v7.1.0, the key new features and improvements are as follows: Scalability and Performance - Session-level plan cache for non-prepared plans (GA) + Session-level plan cache for non-prepared statements (GA) Support automatically reusing plan cache at the session level to remove query planning time, reducing query time for repeat SQL patterns without manually setting prepare statements in advance. From 066c722228ffc2a209c3cb0c8a96e99fabe4cca1 Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 27 Apr 2023 12:09:57 +0800 Subject: [PATCH 40/42] Apply suggestions from code review Co-authored-by: Sam Dillard --- releases/release-7.1.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index ee4d3eed2a114..1610aa08f0adb 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -54,7 +54,7 @@ In v7.1.0, the key new features and improvements are as follows: * TiFlash supports late materialization (GA) [#5829](https://github.com/pingcap/tiflash/issues/5829) @[Lloyd-Pottiger](https://github.com/Lloyd-Pottiger) - In v7.0.0, TiFlash supports late materialization as an experimental feature for optimizing query performance. This feature is disabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `OFF`). When processing a `SELECT` statement with filter conditions (`WHERE` clause), TiFlash reads all the data from the columns required by the query, and then filters and aggregates the data based on the query conditions. When Late materialization is enabled, TiDB supports pushing down part of the filter conditions to the TableScan operator. That is, TiFlash first scans the column data related to the filter conditions that are pushed down to the TableScan operator, filters the rows that meet the condition, and then scans the other column data of these rows for further calculation, thereby reducing IO scans and computations of data processing. + In v7.0.0, late materialization was introduced in TiFlash as an experimental feature for optimizing query performance. This feature is disabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `OFF`). When processing a `SELECT` statement with filter conditions (`WHERE` clause), TiFlash reads all the data from the columns required by the query, and then filters and aggregates the data based on the query conditions. When Late materialization is enabled, TiDB supports pushing down part of the filter conditions to the TableScan operator. That is, TiFlash first scans the column data related to the filter conditions that are pushed down to the TableScan operator, filters the rows that meet the condition, and then scans the other column data of these rows for further calculation, thereby reducing IO scans and computations of data processing. Starting from v7.1.0, the TiFlash late materialization feature is generally available and enabled by default (the [`tidb_opt_enable_late_materialization`](/system-variables.md#tidb_opt_enable_late_materialization-new-in-v700) system variable defaults to `ON`). The TiDB optimizer decides which filters to be pushed down to the TableScan operator based on the statistics and the filter conditions of the query. From 5e36267a6e53e37d099be2521b002720e0cd8f8d Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 27 Apr 2023 14:08:36 +0800 Subject: [PATCH 41/42] Apply suggestions from code review --- releases/release-7.1.0.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 1610aa08f0adb..3d6301a37eb16 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -33,7 +33,7 @@ In v7.1.0, the key new features and improvements are as follows: Reliability and availability - Resource control by Resource Groups (GA) + Resource control by resource groups (GA) Support resource management based on resource groups, which maps database users to the corresponding resource groups and sets quotas for each resource group based on actual needs. @@ -122,7 +122,7 @@ In v7.1.0, the key new features and improvements are as follows: Starting from v6.5.0, TiDB supports pushing down the `SELECT` clause (analytical query) of the `INSERT INTO SELECT` statement to TiFlash. In this way, you can easily save the TiFlash query result to a TiDB table specified by `INSERT INTO` for further analysis, which takes effect as result caching (that is, result materialization). - In v7.1.0, this feature is generally available. During the execution of the `SELECT` clause in the `INSERT INTO SELECT` statement, the optimizer can intelligently decide whether to push a query down to TiFlash based on the [SQL mode](/sql-mode.md) and the cost estimates of the TiFlash replica. Therefore, the `tidb_enable_tiflash_read_for_write_stmt` system variable introduced during the experimental phase is now deprecated. Note that the computation rules of `INSERT INTO SELECT` statements for TiFlash do not meet the `STRICT SQL Mode` requirement, so TiDB allows the `SELECT` clause in the `INSERT INTO SELECT` statement to be pushed down to TiFlash only when the [SQL Mode](/sql-mode.md) of the current session is not strict, which means that the `sql_mode` value does not contain `STRICT_TRANS_TABLES` and `STRICT_ALL_TABLES`. + In v7.1.0, this feature is generally available. During the execution of the `SELECT` clause in the `INSERT INTO SELECT` statement, the optimizer can intelligently decide whether to push a query down to TiFlash based on the [SQL mode](/sql-mode.md) and the cost estimates of the TiFlash replica. Therefore, the `tidb_enable_tiflash_read_for_write_stmt` system variable introduced during the experimental phase is now deprecated. Note that the computation rules of `INSERT INTO SELECT` statements for TiFlash do not meet the `STRICT SQL Mode` requirement, so TiDB allows the `SELECT` clause in the `INSERT INTO SELECT` statement to be pushed down to TiFlash only when the [SQL mode](/sql-mode.md) of the current session is not strict, which means that the `sql_mode` value does not contain `STRICT_TRANS_TABLES` and `STRICT_ALL_TABLES`. For more information, see [documentation](/tiflash/tiflash-results-materialization.md). @@ -203,7 +203,7 @@ In v7.1.0, the key new features and improvements are as follows: * `debug_trace.json` in the output of [`PLAN REPLAYER`](/sql-plan-replayer.md). * Partial statistics details for `operator info` in the output of [`EXPLAIN`](/explain-walkthrough.md). - * Partial statistics details in the `Stats` field of [`slow-queries`](/identify-slow-queries.md). + * Partial statistics details in the `Stats` field of [slow queries](/identify-slow-queries.md). For more information, see [Use `PLAN REPLAYER` to save and restore the on-site information of a cluster](/sql-plan-replayer.md), [`EXPLAIN` walkthrough](/explain-walkthrough.md), and [Identify slow queries](/identify-slow-queries.md). From 094c8a8c53dde748780467703e203f04d40e9706 Mon Sep 17 00:00:00 2001 From: Aolin Date: Thu, 27 Apr 2023 14:19:02 +0800 Subject: [PATCH 42/42] Apply suggestions from code review --- releases/release-7.1.0.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/releases/release-7.1.0.md b/releases/release-7.1.0.md index 3d6301a37eb16..00456d0014dea 100644 --- a/releases/release-7.1.0.md +++ b/releases/release-7.1.0.md @@ -102,7 +102,7 @@ In v7.1.0, the key new features and improvements are as follows: For more information, see [documentation](/system-variables.md#tidb_ddl_enable_fast_reorg-new-in-v630). -* Backup & Restore supports checkpoint restore [#issue](https://github.com/pingcap/tidb/issues/issue) @[Leavrth](https://github.com/Leavrth) +* Backup & Restore supports checkpoint restore [#42339](https://github.com/pingcap/tidb/issues/42339) @[Leavrth](https://github.com/Leavrth) Snapshot restore or log restore might be interrupted due to recoverable errors, such as disk exhaustion and node crash. Before TiDB v7.1.0, the recovery progress before the interruption would be invalidated even after the error is addressed, and you need to start the restore from scratch. For large clusters, this incurs considerable extra cost.