Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update the log redaction documentation #18487

Merged
merged 8 commits into from
Aug 14, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
40 changes: 31 additions & 9 deletions log-redaction.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,14 +9,12 @@ When TiDB provides detailed log information, it might print sensitive data (for

## Log redaction in TiDB side

To enable log redaction in the TiDB side, set the value of [`global.tidb_redact_log`](/system-variables.md#tidb_redact_log) to `1`. This configuration value defaults to `0`, which means that log redaction is disabled.
To enable log redaction in the TiDB side, set the value of [`global.tidb_redact_log`](/system-variables.md#tidb_redact_log) to `ON` or `MARKER`. This configuration value defaults to `OFF`, which means that log redaction is disabled.

You can use the `set` syntax to set the global variable `tidb_redact_log`:

{{< copyable "sql" >}}

```sql
set @@global.tidb_redact_log=1;
set @@global.tidb_redact_log = ON;
```

After the setting, all logs generated in new sessions are redacted:
Expand All @@ -32,19 +30,43 @@ ERROR 1062 (23000): Duplicate entry '1' for key 't.a'
The error log for the `INSERT` statement above is printed as follows:

```
[2020/10/20 11:45:49.539 +08:00] [INFO] [conn.go:800] ["command dispatched failed"] [conn=5] [connInfo="id:5, addr:127.0.0.1:57222 status:10, collation:utf8_general_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="insert into t values ( ? ) , ( ? )"] [txn_mode=OPTIMISTIC] [err="[kv:1062]Duplicate entry '?' for key 't.a'"]
[2024/07/02 11:35:32.686 +08:00] [INFO] [conn.go:1146] ["command dispatched failed"] [conn=1482686470] [session_alias=] [connInfo="id:1482686470, addr:127.0.0.1:52258 status:10, collation:utf8mb4_0900_ai_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="insert into `t` values ( ... )"] [txn_mode=PESSIMISTIC] [timestamp=450859193514065921] [err="[kv:1062]Duplicate entry '?' for key 't.a'"]
```

From the preceding error log, you can see that when the value of `tidb_redact_log` is set to `ON`, sensitive information is replaced by the `?` mark in the TiDB log to avoid data security risks.

In addition, TiDB provides the `MARKER` option. When the value of `tidb_redact_log` is set to `MARKER`, TiDB marks sensitive information in the log with `‹›` instead of replacing it directly, so you can customize the redaction rules.

```sql
set @@global.tidb_redact_log = MARKER;
```

After the preceding configuration, the sensitive information is marked rather than replaced in all logs generated by new sessions:

```sql
create table t (a int, unique key (a));
Query OK, 0 rows affected (0.07 sec)

insert into t values (1),(1);
ERROR 1062 (23000): Duplicate entry '‹1›' for key 't.a'
```

The error log is as follows:

```
[2024/07/02 11:35:01.426 +08:00] [INFO] [conn.go:1146] ["command dispatched failed"] [conn=1482686470] [session_alias=] [connInfo="id:1482686470, addr:127.0.0.1:52258 status:10, collation:utf8mb4_0900_ai_ci, user:root"] [command=Query] [status="inTxn:0, autocommit:1"] [sql="insert into `t` values ( ‹1› ) , ( ‹1› )"] [txn_mode=PESSIMISTIC] [timestamp=450859185309483010] [err="[kv:1062]Duplicate entry '‹1›' for key 't.a'"]
```

From the error log above, you can see that all sensitive information is shielded using `?` after `tidb_redact_log` is enabled. In this way, data security risks are avoided.
As you can see from the preceding error log, after you set `tidb_redact_log` to `MARKER`, TiDB marks sensitive information using `‹ ›` in the log. You can customize redaction rules to handle sensitive information in the log as needed.

## Log redaction in TiKV side

To enable log redaction in the TiKV side, set the value of [`security.redact-info-log`](/tikv-configuration-file.md#redact-info-log-new-in-v408) to `true`. This configuration value defaults to `false`, which means that log redaction is disabled.
To enable log redaction in the TiKV side, set the value of [`security.redact-info-log`](/tikv-configuration-file.md#redact-info-log-new-in-v408) to `true` or `"marker"`. This configuration value defaults to `false`, which means that log redaction is disabled.

## Log redaction in PD side

To enable log redaction in the PD side, set the value of [`security.redact-info-log`](/pd-configuration-file.md#redact-info-log-new-in-v50) to `true`. This configuration value defaults to `false`, which means that log redaction is disabled.
To enable log redaction in the PD side, set the value of [`security.redact-info-log`](/pd-configuration-file.md#redact-info-log-new-in-v50) to `true` or `"marker"`. This configuration value defaults to `false`, which means that log redaction is disabled.

## Log redaction in TiFlash side

To enable log redaction in the TiFlash side, set both the [`security.redact_info_log`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) value in tiflash-server and the [`security.redact-info-log`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) value in tiflash-learner to `true`. Both configuration values default to `false`, which means that log redaction is disabled.
To enable log redaction in the TiFlash side, set both the [`security.redact_info_log`](/tiflash/tiflash-configuration.md#configure-the-tiflashtoml-file) value in tiflash-server and the [`security.redact-info-log`](/tiflash/tiflash-configuration.md#configure-the-tiflash-learnertoml-file) value in tiflash-learner to `true` or `"marker"`. Both configuration values default to `false`, which means that log redaction is disabled.
3 changes: 2 additions & 1 deletion pd-configuration-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -198,8 +198,9 @@ Configuration items related to security
### `redact-info-log` <span class="version-mark">New in v5.0</span>

+ Controls whether to enable log redaction in the PD log
+ When you set the configuration value to `true`, user data is redacted in the PD log.
+ Optional value: `false`, `true`, `"marker"`
+ Default value: `false`
+ For details on how to use it, see [Log redaction in PD side](/log-redaction.md#log-redaction-in-pd-side).

## `log`

Expand Down
17 changes: 10 additions & 7 deletions tiflash/tiflash-configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -256,10 +256,11 @@ delta_index_cache_size = 0

## Security settings take effect starting from v4.0.5.
[security]
## New in v5.0. This configuration item enables or disables log redaction. Value options: true, false, marker. The marker option is introduced in v8.2.0.
## The default value is false, which means that log redaction is disabled.
## If the configuration item is set to true, all user data in the log is replaced by `?`.
## If the configuration item is set to marker, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed.
## New in v5.0. This configuration item enables or disables log redaction. Value options: `true`, `false`, `"on"`, `"off"`, and `"marker"`. The `"on"`, `"off"`, and `"marker"` options are introduced in v8.2.0.
## If the configuration item is set to `false` or `"off"`, log redaction is disabled.
## If the configuration item is set to `true` or `"on"`, all user data in the log is replaced by `?`.
## If the configuration item is set to `"marker"`, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed.
## The default value is `false`.
## Note that you also need to set security.redact-info-log for tiflash-learner's logging in tiflash-learner.toml.
# redact_info_log = false

Expand Down Expand Up @@ -307,9 +308,11 @@ The parameters in `tiflash-learner.toml` are basically the same as those in TiKV
snap-handle-pool-size = 2

[security]
## New in v5.0. This configuration item enables or disables log redaction. Value options: true, false.
## The default value is false, which means that log redaction is disabled.
## If the configuration item is set to true, all user data in the log is replaced by `?`.
## New in v5.0. This configuration item enables or disables log redaction. Value options: `true`, `false`, `"on"`, `"off"`, and `"marker"`. The `"on"`, `"off"`, and `"marker"` options are introduced in v8.3.0.
## If the configuration item is set to `false` or `"off"`, log redaction is disabled.
## If the configuration item is set to `true` or `"on"`, all user data in the log is replaced by `?`.
## If the configuration item is set to `"marker"`, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed.
## The default value is `false`.
redact-info-log = false

[security.encryption]
Expand Down
6 changes: 5 additions & 1 deletion tikv-configuration-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -2041,8 +2041,12 @@ Configuration items related to security.

### `redact-info-log` <span class="version-mark">New in v4.0.8</span>

+ This configuration item enables or disables log redaction. If the configuration value is set to `true`, all user data in the log will be replaced by `?`.
+ This configuration item enables or disables log redaction. Value options: `true`, `false`, `"on"`, `"off"`, and `"marker"`. The `"on"`, `"off"`, and `"marker"` options are introduced in v8.3.0.
+ If the configuration item is set to `false` or `"off"`, log redaction is disabled.
+ If the configuration item is set to `true` or `"on"`, all user data in the log is replaced by `?`.
+ If the configuration item is set to `"marker"`, all user data in the log is wrapped in `‹ ›`. If user data contains `‹` or `›`, `‹` is escaped as `‹‹`, and `›` is escaped as `››`. Based on the marked logs, you can decide whether to desensitize the marked information when the logs are displayed.
+ Default value: `false`
+ For details on how to use it, see [Log redaction in TiKV side](/log-redaction.md#log-redaction-in-tikv-side).

## security.encryption

Expand Down
Loading