insert_deduplication_token setting for INSERT statement

[devcrafter]

The setting allows a user to provide own deduplication semantic in Replicated*MergeTree
If provided, it's used instead of data digest to generate block ID
So, for example, by providing a unique value for the setting in each INSERT statement,
user can avoid the same inserted data being deduplicated

Issue: ClickHouse#7461

Changelog category (leave one):

• New Feature

Changelog entry (a user-readable short description of the changes that goes to CHANGELOG.md):

insert_deduplication_token setting for INSERT statement (only for Replicated*MergeTree tables)
The setting allows a user to provide own deduplication semantic in Replicated*MergeTree
For example, by providing a unique value for the setting in each INSERT statement,
the user can avoid the same inserted data being deduplicated

Detailed description / Documentation draft:
...

By adding documentation, you'll allow users to try your new feature immediately, not when someone else will have time to document it later. Documentation is necessary for all features that affect user experience in any way. You can add brief documentation draft above, or add documentation right into your patch as Markdown files in docs folder.

If you are doing this for the first time, it's recommended to read the lightweight Contributing to ClickHouse Documentation guide first.

Information about CI checks: https://clickhouse.tech/docs/en/development/continuous-integration/

[devcrafter]

I'd consider limiting the size of insert_deduplication_token. What could be a reasonable number? I guess 1 KB should be fine ...

[alexey-milovidov]

Not necessary, it is limited implicitly nevertheless by memory limit and other limits.

[devcrafter]

ClickHouse#31609 (comment)
ClickHouse#31609 (comment)

Done

[devcrafter]

@alesapin @alexey-milovidov Please let me know if you have any comments

[what-the-diff]

• Added a new setting insert_deduplication_token
• Changed IMergeTreeDataPart::getZeroLevelPartBlockID to accept dedup token as an argument and use it instead of data digest if provided
• Replaced calls to getZeroLevelPartBlockID with the ones that pass empty string or context's settings value for insert_deduplication_token depending on whether we need block id based on data digest or not (MergeTreeSink, MergeTreeData)
• Updated tests: 02124 - added test case for multiple blocks in one query; 02119 - updated expected result after adding support for dedup tokens into INSERT SELECT queries
• Added tests for deduplication token on MergeTree and ReplicatedMergeTree tables.
• Fixed bug with incorrect deduplication of blocks by data digest when insert_deduplication_token is set to non-empty value ([experiment] apt-get upgrade in docker/server/Dockerfile ClickHouse/ClickHouse#12124).