Skip to content

Commit

Permalink
include tiering configuration in ttl docs (ydb-platform#11704)
Browse files Browse the repository at this point in the history 
Co-authored-by: Ivan Blinkov <ivan@ydb.tech>
  • Loading branch information
@swalrus1 @blinkov @azevaykin
2 people authored and azevaykin committed Feb 3, 2025
1 parent 4559ca8 commit 0b976e2
Show file tree
Hide file tree
Showing 30 changed files with 935 additions and 581 deletions.
9 changes: 9 additions & 0 deletions ydb/docs/en/core/_includes/not_allow_for_oltp_note.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
{% if oss == true and backend_name == "YDB" %}

{% note warning %}

{% include [OLTP_not_allow_text](not_allow_for_oltp_text.md) %}

{% endnote %}

{% endif %}
1 change: 1 addition & 0 deletions ydb/docs/en/core/_includes/not_allow_for_oltp_text.md
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Supported only for [column-oriented](../concepts/datamodel/table.md#column-oriented-tables) tables. Support for [row-oriented](../concepts/datamodel/table.md#row-oriented-tables) tables is currently under development.
287 changes: 12 additions & 275 deletions ydb/docs/en/core/concepts/_includes/ttl.md
View file
Original file line number Diff line number Diff line change
@@ -1,26 +1,22 @@
# Time to Live (TTL)
# Time to Live (TTL) and eviction to external storage

This section describes how the TTL mechanism works and what its limits are. It also gives examples of commands and code snippets that can be used to enable, configure, and disable TTL.
This section describes how the TTL mechanism works and what its limits are.

## How it works {#how-it-works}

{{ ydb-short-name }} allows you to specify a TTL column in both [row-oriented](../datamodel/table.md#row-oriented-tables) and [column-oriented](../datamodel/table.md#column-oriented-tables) tables. Values in TTL columns determine the table rows lifetime.
The table's TTL is a sequence of storage tiers. Each tier contains an expression (TTL expression) and an action. When the expression is triggered, that tier is assigned to the row. When a tier is assigned to a row, the specified action is automatically performed: moving the row to external storage or deleting it. External storage is represented by the [external data source](../datamodel/external_data_source.md) object.

{% note warning %}

An item with the `NULL` value in the TTL column is never deleted.

{% endnote %}
{{ ydb-short-name }} allows you to specify a column (TTL column) whose values are used in TTL expressions. The expression is triggered when the specified number of seconds has passed since the time recorded in the TTL column. For rows with `NULL` value in TTL column, the expression is not triggered.

The timestamp for deleting a table item is determined by the formula:

```text
expiration_time = valueof(ttl_column) + expire_after_seconds
eviction_time = valueof(ttl_column) + evict_after_seconds
```

{% note info %}

TTL doesn't guarantee that the item will be deleted exactly at `expiration_time`, it might happen later. If it's important to exclude logically obsolete but not yet physically deleted items from the selection, use query-level filtering.
TTL doesn't guarantee that the item will be deleted exactly at `eviction_time`, it might happen later. If it's important to exclude logically obsolete but not yet physically deleted items from the selection, use query-level filtering.

{% endnote %}

Expand Down Expand Up @@ -58,273 +54,14 @@ The *BRO* has the following properties:
* Nanoseconds.

* You can't specify multiple TTL columns.
* You can't delete the TTL column. However, if this is required, you should first [disable TTL](#disable) for the table.
* You can't delete the TTL column. However, if this is required, you should first [disable TTL](../../recipes/yql/ttl.md#disable) for the table.
* Only {{ objstorage-name }} is supported as external storage.
* The delete action can only be specified for the last tier.

## Setup {#setting}

Currently, you can manage TTL settings using:

* [YQL](../../yql/reference/index.md).
* [{{ ydb-short-name }} console client](../../reference/ydb-cli/index.md).
* {{ ydb-short-name }} {% if oss %}C++, {% endif %}Go and Python [SDK](../../reference/ydb-sdk/index.md).

### Enabling TTL for an existing table {#enable-on-existent-table}

In the example below, the items of the `mytable` table will be deleted an hour after the time set in the `created_at` column:

{% list tabs group=tool %}

- YQL

```yql
ALTER TABLE `mytable` SET (TTL = Interval("PT1H") ON created_at);
```

- CLI

```bash
$ {{ ydb-cli }} -e <endpoint> -d <database> table ttl set --column created_at --expire-after 3600 mytable
```

{% if oss == true %}

- C++

```c++
session.AlterTable(
"mytable",
TAlterTableSettings()
.BeginAlterTtlSettings()
.Set("created_at", TDuration::Hours(1))
.EndAlterTtlSettings()
);
```
{% endif %}
- Go
```go
err := session.AlterTable(ctx, "mytable",
options.WithSetTimeToLiveSettings(
options.NewTTLSettings().ColumnDateType("created_at").ExpireAfter(time.Hour),
),
)
```

- Python

```python
session.alter_table('mytable', set_ttl_settings=ydb.TtlSettings().with_date_type_column('created_at', 3600))
```

{% endlist %}

{% note tip %}

When setting up TTL using YQL, an `Interval` is created from a string literal in [ISO 8601](https://en.wikipedia.org/wiki/ISO_8601) format with [some restrictions](../../yql/reference/builtins/basic#data-type-literals).

{% endnote %}

The example below shows how to use the `modified_at` column with a numeric type (`Uint32`) as a TTL column. The column value is interpreted as the number of seconds since the Unix epoch:

{% list tabs group=tool %}

- YQL

```yql
ALTER TABLE `mytable` SET (TTL = Interval("PT1H") ON modified_at AS SECONDS);
```

- CLI

```bash
$ {{ ydb-cli }} -e <endpoint> -d <database> table ttl set --column modified_at --expire-after 3600 --unit seconds mytable
```

{% if oss == true %}

- C++

```c++
session.AlterTable(
"mytable",
TAlterTableSettings()
.BeginAlterTtlSettings()
.Set("modified_at", TTtlSettings::EUnit::Seconds, TDuration::Hours(1))
.EndAlterTtlSettings()
);
```
{% endif %}
- Go
```go
err := session.AlterTable(ctx, "mytable",
options.WithSetTimeToLiveSettings(
options.NewTTLSettings().ColumnSeconds("modified_at").ExpireAfter(time.Hour),
),
)
```

- Python

```python
session.alter_table('mytable', set_ttl_settings=ydb.TtlSettings().with_value_since_unix_epoch('modified_at', UNIT_SECONDS, 3600))
```

{% endlist %}

### Enabling TTL for a newly created table {#enable-for-new-table}

For a newly created table, you can pass TTL settings along with the table description:

{% list tabs group=tool %}

- YQL

```yql
CREATE TABLE `mytable` (
id Uint64,
expire_at Timestamp,
PRIMARY KEY (id)
) WITH (
TTL = Interval("PT0S") ON expire_at
);
```

{% if oss == true %}

- C++

```c++
session.CreateTable(
"mytable",
TTableBuilder()
.AddNullableColumn("id", EPrimitiveType::Uint64)
.AddNullableColumn("expire_at", EPrimitiveType::Timestamp)
.SetPrimaryKeyColumn("id")
.SetTtlSettings("expire_at")
.Build()
);
```
{% endif %}
- Go
```go
err := session.CreateTable(ctx, "mytable",
options.WithColumn("id", types.Optional(types.TypeUint64)),
options.WithColumn("expire_at", types.Optional(types.TypeTimestamp)),
options.WithTimeToLiveSettings(
options.NewTTLSettings().ColumnDateType("expire_at"),
),
)
```

- Python

```python
session.create_table(
'mytable',
ydb.TableDescription()
.with_column(ydb.Column('id', ydb.OptionalType(ydb.DataType.Uint64)))
.with_column(ydb.Column('expire_at', ydb.OptionalType(ydb.DataType.Timestamp)))
.with_primary_key('id')
.with_ttl(ydb.TtlSettings().with_date_type_column('expire_at'))
)
```

{% endlist %}

### Disabling TTL {#disable}

{% list tabs group=tool %}

- YQL

```yql
ALTER TABLE `mytable` RESET (TTL);
```

- CLI

```bash
$ {{ ydb-cli }} -e <endpoint> -d <database> table ttl reset mytable
```

{% if oss == true %}

- C++

```c++
session.AlterTable(
"mytable",
TAlterTableSettings()
.BeginAlterTtlSettings()
.Drop()
.EndAlterTtlSettings()
);
```

{% endif %}

- Go

```go
err := session.AlterTable(ctx, "mytable",
options.WithDropTimeToLive(),
)
```

- Python

```python
session.alter_table('mytable', drop_ttl_settings=True)
```

{% endlist %}

### Getting TTL settings {#describe}

The current TTL settings can be obtained from the table description:

{% list tabs group=tool %}

- CLI

```bash
$ {{ ydb-cli }} -e <endpoint> -d <database> scheme describe mytable
```

{% if oss == true %}

- C++

```c++
auto desc = session.DescribeTable("mytable").GetValueSync().GetTableDescription();
auto ttl = desc.GetTtlSettings();
```

{% endif %}

- Go

```go
desc, err := session.DescribeTable(ctx, "mytable")
if err != nil {
// process error
}
ttl := desc.TimeToLiveSettings
```

- Python

```python
desc = session.describe_table('mytable')
ttl = desc.ttl_settings
```

{% endlist %}
* [YQL](../../recipes/yql/ttl.md).
* [{{ ydb-short-name }} console client](../../recipes/ydb-cli/ttl.md).
* {{ ydb-short-name }} {% if oss %}C++, {% endif %}Go and Python [SDK](../../recipes/ydb-sdk/ttl.md).
11 changes: 2 additions & 9 deletions ydb/docs/en/core/concepts/datamodel/_includes/table.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -134,16 +134,9 @@ If there are multiple followers, their delay from the leader may vary: although

| Parameter name | Type | Acceptable values | Update capability | Reset capability |
| ------------- | --- | ------------------- | --------------------- | ------------------ |
| `TTL` | Expression | `Interval("<literal>") ON <column> [AS <unit>]` | Yes | Yes |
| `TTL` | Expression | `Interval("<literal>") ON <column> [AS <unit>]` or `Interval("literal1") action1, ..., Interval("literal1") action1 ON <column> [AS <unit>]` | Yes | Yes |

Where `<unit>` is a unit of measurement, specified only for column with a [numeric type](../../../concepts/ttl.md#restrictions):

* `SECONDS`
* `MILLISECONDS`
* `MICROSECONDS`
* `NANOSECONDS`

For more information about deleting expired data, see [Time to Live (TTL)](../../../concepts/ttl.md).
Syntax of TTL value is described in the article [{#T}](../../../yql/reference/syntax/create_table/with.md#time-to-live). For more information about deleting expired data, see [Time to Live (TTL)](../../../concepts/ttl.md).

### Renaming a table {#rename}

Expand Down
2 changes: 1 addition & 1 deletion ydb/docs/en/core/concepts/toc_i.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ items:
- name: Change Data Capture (CDC)
href: cdc.md
when: feature_changefeed
- name: Time to Live (TTL)
- name: Time to live and eviction
href: ttl.md
- name: Scan queries
href: scan_query.md
Expand Down
1 change: 1 addition & 0 deletions ydb/docs/en/core/recipes/ydb-cli/index.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@ Table of contents:

* [{#T}](convert-table-type.md)
* [{#T}](benchmarks.md)
* [{#T}](ttl.md)

See also:

Expand Down
4 changes: 3 additions & 1 deletion ydb/docs/en/core/recipes/ydb-cli/toc_p.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -2,4 +2,6 @@ items:
- name: Convert table type
href: convert-table-type.md
- name: Benchmarks
href: benchmarks.md
href: benchmarks.md
- name: Time to live (TTL)
href: ttl.md
Loading

0 comments on commit 0b976e2

Please sign in to comment.