Create data stream aliases from template #73867

danhermann · 2021-06-07T21:13:00Z

Allows the creation of data stream aliases from a data stream template.

Relates to #66163

danhermann · 2021-06-16T19:53:44Z

@elasticmachine update branch

danhermann · 2021-07-13T16:54:25Z

@elasticmachine update branch

danhermann · 2021-07-16T13:09:03Z

@elasticmachine update branch

danhermann · 2021-07-16T14:08:06Z

@martijnvg, I have data stream aliases being created from templates in this PR. I did it using only the AliasMetadata class without using the o.e.cluster.metadata.DataStreamAlias class which works as long as we're willing to overload or rename some of the members of AliasMetadata to refer to indices or data streams. If you think this is a good approach, I'll go ahead and add some tests and get it ready for final review.

martijnvg

Thanks @danhermann, I think this pr is in the right direction, but I think we maybe can simplify even more (see 2 comments).

martijnvg · 2021-07-19T14:20:30Z

server/src/main/java/org/elasticsearch/cluster/metadata/AliasMetadata.java

@@ -51,8 +52,11 @@
    @Nullable
    private final Boolean isHidden;

+    @Nullable
+    private final Boolean isDataStream;


I don't think this field is needed? I also don't see that is currently is used in this pr.
In MetadataCreateDataStreamService, data stream aliases can be created without this information? The fact that a DataStreamTemplate is defined should be sufficient?

martijnvg · 2021-07-19T14:23:56Z

server/src/main/java/org/elasticsearch/cluster/metadata/ComposableIndexTemplate.java

        }

        private final boolean hidden;

+        @Nullable
+        private final Map<String, AliasMetadata> aliases;


I think we can reuse the alias definitions in ComposableIndexTemplate (defined inside the Template class)?
If a composable index template has a DataStreamTemplate and aliases definitions are present then data stream aliases are created. If just alias definitions are present in composable index template then indices aliases are created.

danhermann · 2021-07-21T19:33:10Z

@elasticmachine run elasticsearch-ci/part-2

martijnvg

Left 2 small comments, LGTM otherwise 👍.

martijnvg · 2021-07-22T09:34:08Z

...r/src/test/java/org/elasticsearch/cluster/metadata/MetadataCreateDataStreamServiceTests.java

+        Map<String, AliasMetadata> aliases = new HashMap<>(aliasCount);
+        for (int k = 0; k < aliasCount; k++) {
+            final String aliasName = randomAlphaOfLength(6);
+            aliases.put(aliasName, AliasMetadata.newAliasMetadataBuilder(aliasName).build());


maybe also test the case here with aliases with write data streams and filter? (Either by randomly adding and asserting it or a new test)

martijnvg · 2021-07-22T09:36:42Z

server/src/main/java/org/elasticsearch/cluster/metadata/MetadataCreateDataStreamService.java

@@ -222,6 +221,13 @@ static ClusterState createDataStream(MetadataCreateIndexService metadataCreateIn
        logger.info("adding data stream [{}] with write index [{}] and backing indices [{}]", dataStreamName,


maybe enhance this log line with the fact that aliases were created as well (if template has aliases)?

danhermann · 2021-07-22T18:12:08Z

Thanks, @martijnvg. I made those changes.

With #73867, you can use component and index templates to add data stream aliases.

…c#75688) With elastic#73867, you can use component and index templates to add data stream aliases.

#75700) With #73867, you can use component and index templates to add data stream aliases. Co-authored-by: James Rodewig <40268737+jrodewig@users.noreply.github.com>

…c#75688) With elastic#73867, you can use component and index templates to add data stream aliases.

wip

52c9402

danhermann added >non-issue v8.0.0 :Data Management/Data streams Data streams and their lifecycles v7.14.0 labels Jun 7, 2021

danhermann added 6 commits June 7, 2021 16:25

fix test compile error

ebdbce5

add missing parser

cc8c7c4

bwc

5d21f2f

fix compilation

738b9e6

unused import

b0848d6

fix test

36847c8

Merge branch 'master' into 66163_create_alias_from_template

0aebdb4

mark-vieira added v7.15.0 and removed v7.14.0 labels Jun 30, 2021

martijnvg mentioned this pull request Jun 30, 2021

Introduce aliases for data streams #66163

Open

7 tasks

elasticmachine and others added 3 commits July 13, 2021 18:54

Merge branch 'master' into 66163_create_alias_from_template

49968e4

fix merge error

001b0d5

auto-create alias when data stream is created

39973f6

Merge branch 'master' into 66163_create_alias_from_template

991a36b

martijnvg reviewed Jul 19, 2021

View reviewed changes

danhermann and others added 6 commits July 19, 2021 11:21

remove AliasMetadata::isDataStream

e12bd60

reuse existing alias definitions

1399903

Merge branch 'master' into 66163_create_alias_from_template

59e57af

fix NPE

0f5c6a9

add test

43e1d5f

remove a couple more unnecessary flags

45cc1f5

elasticmachine and others added 3 commits July 21, 2021 18:35

Merge branch 'master' into 66163_create_alias_from_template

77e2a32

fix merge error

8d1e1e7

fix test

33f2359

martijnvg approved these changes Jul 22, 2021

View reviewed changes

review comments

833677a

danhermann merged commit 0232675 into elastic:master Jul 22, 2021

danhermann deleted the 66163_create_alias_from_template branch July 22, 2021 18:12

danhermann added the backport pending label Jul 22, 2021

danhermann added a commit to danhermann/elasticsearch that referenced this pull request Jul 22, 2021

Create data stream aliases from template (elastic#73867)

a8ca922

danhermann mentioned this pull request Jul 22, 2021

[7.x] Create data stream aliases from template (#73867) #75647

Merged

danhermann added a commit that referenced this pull request Jul 22, 2021

[7.x] Create data stream aliases from template (#73867) (#75647)

30b3ae3

danhermann removed the backport pending label Jul 22, 2021

danhermann mentioned this pull request Jul 23, 2021

[DOCS] Updates for data stream aliases #75654

Merged

jrodewig mentioned this pull request Jul 26, 2021

[DOCS] Update index template API docs for data stream aliases #75688

Merged

jrodewig added a commit that referenced this pull request Jul 26, 2021

[DOCS] Update index template API docs for data stream aliases (#75688)

6078349

With #73867, you can use component and index templates to add data stream aliases.

jakelandis added v8.0.0-alpha1 v8.0.0 and removed v8.0.0 v8.0.0-alpha1 labels Jul 26, 2021

ywangd pushed a commit to ywangd/elasticsearch that referenced this pull request Jul 30, 2021

Create data stream aliases from template (elastic#73867)

4ca2113

ywangd pushed a commit to ywangd/elasticsearch that referenced this pull request Jul 30, 2021

[DOCS] Update index template API docs for data stream aliases (elasti…

868ce3e

…c#75688) With elastic#73867, you can use component and index templates to add data stream aliases.

This was referenced Aug 2, 2021

Create data stream aliases from component templates #75956

Merged

[7.x] Create data stream aliases from component templates (#75956) #76137

Merged

danhermann mentioned this pull request Oct 22, 2021

[7.16] [DOCS] Updates for data stream aliases (#75654) #79682

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Create data stream aliases from template #73867

Create data stream aliases from template #73867

danhermann commented Jun 7, 2021

danhermann commented Jun 16, 2021

danhermann commented Jul 13, 2021

danhermann commented Jul 16, 2021

danhermann commented Jul 16, 2021 •

edited

Loading

martijnvg left a comment

martijnvg Jul 19, 2021

martijnvg Jul 19, 2021

danhermann commented Jul 21, 2021

martijnvg left a comment

martijnvg Jul 22, 2021

martijnvg Jul 22, 2021

danhermann commented Jul 22, 2021

		@@ -222,6 +221,13 @@ static ClusterState createDataStream(MetadataCreateIndexService metadataCreateIn
		logger.info("adding data stream [{}] with write index [{}] and backing indices [{}]", dataStreamName,

Create data stream aliases from template #73867

Create data stream aliases from template #73867

Conversation

danhermann commented Jun 7, 2021

danhermann commented Jun 16, 2021

danhermann commented Jul 13, 2021

danhermann commented Jul 16, 2021

danhermann commented Jul 16, 2021 • edited Loading

martijnvg left a comment

Choose a reason for hiding this comment

martijnvg Jul 19, 2021

Choose a reason for hiding this comment

martijnvg Jul 19, 2021

Choose a reason for hiding this comment

danhermann commented Jul 21, 2021

martijnvg left a comment

Choose a reason for hiding this comment

martijnvg Jul 22, 2021

Choose a reason for hiding this comment

martijnvg Jul 22, 2021

Choose a reason for hiding this comment

danhermann commented Jul 22, 2021

danhermann commented Jul 16, 2021 •

edited

Loading