24.3.15.5

Version 24.3.15.5

Release date: February 6, 2025

Functionality

• Added the ability to register a database node using a certificate. In the Node Broker the flag AuthorizeByCertificate has been added to enable certificate-based registration.
• Added priorities for authentication ticket through a third-party IAM provider, with the highest priority given to requests from new users. Tickets in the cache update their information with a lower priority.

Performance

• Improved tablet startup time on large clusters: 210 ms → 125 ms (SSD), 260 ms → 165 ms (HDD).

Bug Fixes

• Removed the restriction on writing values greater than 127 to the Uint8 type.
• Fixed an issue where reading small messages from a topic in small chunks significantly increased CPU load, which could lead to delays in reading and writing to the topic.
• Fixed an issue with restoring from a backup stored in S3 with path-style addressing.
• Fixed an issue with restoring from a backup that was created during an automatic table split.
• Fixed an issue with Uuid serialization for CDC.
• Fixed an issue with "frozen" locks, which could be caused by bulk operations (e.g., TTL-based deletions).
• Fixed an issue where reading from a follower of tablets sometimes caused crashes during automatic table splits.
• Fixed an issue where the coordination node successfully registered proxy servers despite a connection loss.
• Fixed an issue that occurred when opening the Embedded UI tab with information about distributed storage groups.
• Fixed an issue where the Health Check did not report time synchronization issues.
• Fixed a rare issue that caused errors during read queries.
• Fixed an uncommitted changes leak and cleaned them up on startup.
• Fixed consistency issues related to caching deleted ranges.

24.3.11.14

Version 24.3.11.14

Release date: January 9, 2025.

Functionality

• Added support for restart without downtime in a minimal fault-tolerant configuration of a cluster that uses the three-node variant of mirror-3-dc.
• Added new UDF Roaring Bitmap functions: AndNotWithBinary, FromUint32List, RunOptimize.

24.3.11.13

Version 24.3.11.13

Release date: December 24, 2024.

Functionality

• Introduced query tracing, a tool that allows you to view the detailed path of a request through a distributed system.
• Added support for asynchronous replication, that allows synchronizing data between YDB databases in near real time. It can also be used for data migration between databases with minimal downtime for applications interacting with these databases.
• Added support for views, which can be enabled by the cluster administrator using the enable_views setting in dynamic configuration.
• Extended federated query capabilities to support new external data sources: MySQL, Microsoft SQL Server, and Greenplum.
• Published documentation on deploying YDB with federated query functionality (manual setup).
• Added a new launch parameter FQ_CONNECTOR_ENDPOINT for YDB Docker containers that specifies an external data source connector address. Added support for TLS encryption for connections to the connector and the ability to expose the connector service port locally on the same host as the dynamic YDB node.
• Added an auto-partitioning mode for topics, where partitions can dynamically split based on load while preserving message read-order and exactly-once guarantees. The mode can be enabled by the cluster administrator using the settings enable_topic_split_merge and enable_pqconfig_transactions_at_scheme_shard in dynamic configuration.
• Added support for transactions involving topics and row-based tables, enabling transactional data transfer between tables and topics, or between topics, ensuring no data loss or duplication. Transactions can be enabled by the cluster administrator using the settings enable_topic_service_tx and enable_pqconfig_transactions_at_scheme_shard in dynamic configuration.
• Implemented Change Data Capture (CDC) for synchronous secondary indexes.
• Added support for changing record retention periods in CDC topics.
• Added support for auto-increment columns as part of a table's primary key.
• Added audit logging for user login events in YDB, session termination events in the user interface, and backup/restore operations.
• Added a system view with information about sessions installed from the database using a query.
• Added support literal default values for row-oriented tables. When inserting a new row in YDB Query default values will be assigned to the column if specified.
• Added support for RETURNING clause in queries.
• Added start/end times and authors in the metadata for backup/restore operations from S3-compatible storage.
• Added support for backup/restore of ACL for tables from/to S3-compatible storage.
• Included paths and decompression methods in query plans for reading from S3.
• Added new parsing options for timestamp/datetime fields when reading data from S3.
• Added support for the Decimal type in partitioning keys.
• Improved diagnostics for storage issues in HealthCheck.
• (Experimental) Added a cost-based optimizer for complex queries, involving column-oriented tables. The cost-based optimizer considers a large number of alternative execution plans for each query and selects the best one based on the cost estimate for each option. Currently, this optimizer only works with plans that contain JOIN operations.
• (Experimental) Initial version of the workload manager was implemented. It allows to create resource pools with CPU, memory and active queries count limits. Resource classifiers were implemented to assign queries to specific resource pool.
• (Experimental) Implemented automatic index selection for queries, which can be enabled via the index_auto_choose_mode setting in table_service_config in dynamic configuration.

YDB UI

• Added support for creating and viewing information on asynchronous replication instances.
• Added an indicator for auto-increment columns.
• Added a tab with information about tablets.
• Added a tab with details about distributed storage groups.
• Added a setting to trace all queries and display tracing results.
• Enhanced the PDisk page with attributes, disk space consumption details, and a button to initiate disk decommissioning.
• Added information about currently running queries.
• Added a row limit setting for query editor output and a notification when results exceed the limit.
• Added a tab to display top CPU-consuming queries over the last hour.
• Added a control to search the history and saved queries pages.
• Added the ability to cancel query execution.
• Added a shortcut to save queries in the editor.
• Separated donor disks from other disks in the UI.
• Added support for InterruptInheritance ACL and improved visualization of active ACLs.
• Added a display of the current UI version.
• Added a tab with information about the status of settings for enabling experimental functionality.

Performance

• Accelerated recovery of tables with secondary indexes from backups up to 20% according to our tests.
• Optimized Interconnect throughput.
• Improved the performance of CDC topics with thousands of partitions.
• Enhanced the Hive tablet balancing algorithm.

Bug fixes

• Fixed an issue that caused databases with a large number of tables or partitions to become non-functional during restoration from a backup. Now, if database size limits are exceeded, the restoration operation will fail, but the database will remain operational.
• Implemented a mechanism to forcibly trigger background compaction when discrepancies between the data schema and stored data are detected in DataShard. This resolves a rare issue with delays in schema changes.
• Resolved duplication of authentication tickets, which led to an increased number of requests to authentication providers.
• Fixed an invariant violation issue during the initial scan of CDC, leading to an abnormal termination of the ydbd server process.
• Prohibited schema changes for backup tables.
• Fixed an issue with an initial scan freezing during CDC when the table is frequently updated.
• Excluded deleted indexes from the count against the maximum index limit.
• Fixed a bug in the display of the scheduled execution time for a set of transactions (planned step).
• Fixed a problem with interruptions in blue–green deployment in large clusters caused by frequent updates to the node list.
• Resolved a rare issue that caused transaction order violations.
• Fixed an issue in the EvWrite API that resulted in incorrect memory deallocation.
• Resolved a problem with volatile transactions hanging after a restart.
• Fixed a bug in the CDC, which in some cases leads to increased CPU consumption, up to a core per CDC partition.
• Eliminated read delays occurring during and after the splitting of certain partitions.
• Fixed issues when reading data from S3.
• Corrected th...

YDB Server 24.2.8

Long domain name (#9108)

YDB Server 24.1.18

Merge stable-24-1-16-analytics into stable-24-1 (#5596)

Co-authored-by: Vitalii Gridnev <gridnevvvit@gmail.com>
Co-authored-by: Andrey Neporada <aneporada@ydb.tech>
Co-authored-by: niksaveliev <nik@saveliev.me>
Co-authored-by: Sergey Veselov <sergeyveselov@ydb.tech>
Co-authored-by: alexnick88 <alexnick@ydb.tech>
Co-authored-by: Ilnaz Nizametdinov <ilnaz@ydb.tech>
Co-authored-by: Iuliia Sidorina <yulia@ydb.tech>
Co-authored-by: kungurtsev <kungasc@ydb.tech>
Co-authored-by: Nikolay Shestakov <tesseract@ydb.tech>
Co-authored-by: azevaykin <145343289+azevaykin@users.noreply.github.com>
Co-authored-by: DimasKovas <34828390+DimasKovas@users.noreply.github.com>
Co-authored-by: ijon <ijon@ydb.tech>
Co-authored-by: vporyadke <zalyalov@ydb.tech>
Co-authored-by: kruall <kruall@ydb.tech>
Co-authored-by: ivanmorozov333 <ivanmorozov@ydb.tech>
Co-authored-by: Alexander Rutkovsky <alexvru@ydb.tech>
Co-authored-by: ildar-khisambeev <ikhis@ydb.tech>
Co-authored-by: nsofya <nsofya@ydb.tech>
Co-authored-by: nsofya <nsofya@yandex.ru>
Co-authored-by: Sofya Novozhilova <nsofya@qavm-4bd1829b.qemu>
Co-authored-by: Олег <150132506+iddqdex@users.noreply.github.com>
Co-authored-by: Ivan Morozov <ivanmorozov@localhost.localdomain>
Co-authored-by: Andrei Rykov <arykov@ydb.tech>
Co-authored-by: qyryq <qyryq@ydb.tech>
Co-authored-by: Daniil Cherednik <dcherednik@ydb.tech>
Co-authored-by: Aleksei Borzenkov <snaury@ydb.tech>

YDB Server 23.4.11

Ydb stable 23-4-11

x-stable-origin-commit: 9cc0616f07997e08c92266a30796bc4ff25aea66

YDB CLI 2.8.0

Released on January 12, 2024. To update to version 2.8.0, select the Downloads section.

Features:

• Added configuration management commands for the cluster ydb admin config and ydb admin volatile-config.

• Added support for loading PostgreSQL-compatible data types by ydb import file csv|tsv|json command. Only for row-oriented tables.

• Added support for directory load from an S3-compatible storage in the ydb import s3 command. Currently only available on Linux and Mac OS.

• Added support for outputting the results of ydb table query execute, ydb yql and ydb scripting yql commands in the Apache Parquet format.

• In the ydb workload commands, the --executer option has been added, which allows to specify which type of queries to use.

• Added a column with median benchmark execution time in the statistics table of the ydb workload clickbench command.

• (Experimental) Added the generic request type to the ydb table query execute command, allowing to perform DDL and DML operations, return with arbitrarily-sized results and support for MVCC. The command uses an experimental API, compatibility is not guaranteed.

• (Experimental) In the ydb table query explain command, the --collect-diagnostics option has been added to collect query diagnostics and save it to a file. The command uses an experimental API, compatibility is not guaranteed.

Bug fixes:

• Fixed an error displaying tables in pretty format with Unicode characters.

• Fixed an error substituting the wrong primary key in the command ydb tools pg-convert.

Full Changelog: CLI_2.7.0...CLI_2.8.0

YDB Server 23.3.17

Bugfixes:

• Fix InFlightTxCount become negative during split/merge
• Allow compatibility between non-stable versions of different applications
• Various stability improvements (segfault, handle balancer and boot queue events in StateInit, data race when reporting issue counters, memory leak in TRopeArena)
• CPU overutilization in case of numerous topic consumers
• Fix import multiple converted csv columns
• Fix mem table merge when writing over committed deltas in tablets
• Check errors during read_table_scan and export
• Throttle datashard log messages

Features:

• Enable ANSI settings for generic query
• DML logging
• Request unit trailing header for ScanQuery/ScanYql sctipts
• Grpc KV api
• Change default metering mode of topics to request units
• Pqv0 support on server side
• Allow to maintain unavailable nodes in tablets checker
• Allow to reboot unavailable node
• Add Lazy Histogram for grpc request

YDB CLI 2.7.0

Released on October 23, 2023. To update to version 2.7.0, select the Downloads section.

Features:

• Added the ydb tools pg-convert command, which prepares a dump obtained by the pg_dump utility for loading into the YDB postgres-compatible layer.

• Added the ydb workload query load testing command, which loads the database with script execution queries in multiple threads.

• Added a command ydb scheme permissions list to list permissions.

• In the commands ydb table query execute, ydb table query explain, ydb yql, and ydb scripting yql, the --flame-graph option has been added, specifying the path to the file in which you need to save the visualization of query execution statistics.

• Special commands in the interactive query execution mode are now case-insensitive.

• Added validation for special commands and their parameters.

• Added table reading in the scenario with transactions in the command ydb workload transfer topic-to-table run.

• Added the --commit-messages option to the command ydb workload transfer topic-to-table run, specifying the number of messages in a single transaction.

• Added the options --only-table-in-tx and --only-topic-in-tx in the command ydb workload transfer topic-to-table run, specifying restrictions on the types of queries in a single transaction.

• Added new columns Select time and Upsert time in the statistics table in the command ydb workload transfer topic-to-table run.

Bug fixes:

• Fixed an error when loading an empty JSON list by commands: ydb table query execute, ydb scripting yql and ydb yql.

Full Changelog: CLI_2.6.0...CLI_2.7.0

YDB Server 23.3.13

Release notes: https://ydb.tech/en/docs/changelog-server#23-3