Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add min_* conditions to rollover #83345

Merged
merged 59 commits into from
Jul 26, 2022
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
59 commits
Select commit Hold shift + click to select a range
e6fe054
min conditions
probakowski Jan 20, 2022
12879c4
min conditions
probakowski Jan 31, 2022
2dfe23a
tests
probakowski Jan 31, 2022
a46dc91
Merge remote-tracking branch 'origin/master' into min-age
probakowski Jan 31, 2022
431f1a9
fix
probakowski Feb 1, 2022
d6a7569
remove unused method
probakowski Feb 1, 2022
4446623
tests
probakowski Feb 1, 2022
16b459f
import
probakowski Feb 1, 2022
c3438d1
spotless
probakowski Feb 1, 2022
84a89cd
Merge remote-tracking branch 'origin/master' into min-age
probakowski Feb 1, 2022
31c286b
fix test
probakowski Feb 1, 2022
3568399
fix imports
probakowski Feb 1, 2022
ac7a903
spotless
probakowski Feb 1, 2022
2aae5c8
spotless
probakowski Feb 1, 2022
6502e7d
fix compilation
probakowski Feb 1, 2022
3cf3dc9
fix tests
probakowski Feb 1, 2022
f74a590
spotless
probakowski Feb 1, 2022
b2821fc
Merge remote-tracking branch 'origin/master' into min-age
probakowski Feb 1, 2022
2d797c5
Merge remote-tracking branch 'origin/master' into min-age
probakowski Feb 24, 2022
721ab16
fix tests
probakowski Feb 24, 2022
6170581
fix tests
probakowski Feb 24, 2022
e036e58
spotless
probakowski Feb 24, 2022
39cf0bf
spotless
probakowski Feb 24, 2022
f7f377a
fix rolloveraction tests
probakowski Feb 24, 2022
a4ccd7a
Update docs/changelog/83345.yaml
probakowski Feb 24, 2022
c6ca948
Merge branch 'master' into min-age
joegallo Jun 29, 2022
aff95df
Bump min_* BWC version to 8.4.0
joegallo Jun 29, 2022
577e98d
Change visibility
joegallo Jun 29, 2022
b588c21
Add/rewrite these javadocs
joegallo Jun 29, 2022
7945472
Merge branch 'master' into min-age
joegallo Jul 22, 2022
9f54660
Typo, fix a comment
joegallo Jul 21, 2022
a50b1c7
Add min_size and min_primary_shard_docs
joegallo Jul 22, 2022
b6cc438
Actually, these tests were wrong as written
joegallo Jul 21, 2022
3ba144c
Test that we won't rollover with only min_* conditions
joegallo Jul 21, 2022
6d034f6
Add yaml rest tests
joegallo Jul 21, 2022
591de5c
Typo, fix variable case
joegallo Jul 22, 2022
dabb4d8
Fix a minor buglet
joegallo Jul 22, 2022
929077e
Pull out a randomByteSize test utility
joegallo Jul 22, 2022
1a3ee5f
Add min_* to ilm usage
joegallo Jul 22, 2022
7ee742d
Add some missing asserts in RolloverActionTests
joegallo Jul 22, 2022
bd91ba4
Fix these switch betweens
joegallo Jul 22, 2022
22b03f4
Oops, fix deserialization else branch
joegallo Jul 22, 2022
ebda57e
Nit: follow builder pattern
joegallo Jul 22, 2022
6e0d386
Add min_size, min_primary_shard_docs to NamedXContents
joegallo Jul 22, 2022
b4248ea
Don't leak a mutable map
joegallo Jul 22, 2022
573b384
Rewrite areConditionsMet slightly
joegallo Jul 22, 2022
f5bdfbe
Use areConditionsMet in the task executor, too
joegallo Jul 22, 2022
9c2c03d
Merge branch 'master' into min-age
joegallo Jul 22, 2022
e346d57
Whitespace
joegallo Jul 22, 2022
6658822
Add a validation rule to _rollover requests
joegallo Jul 22, 2022
7927ec9
Merge branch 'main' into min-age
joegallo Jul 25, 2022
3c990a8
Add/fix includedInVersion for min_* conditions
joegallo Jul 25, 2022
76249a4
Tidier RolloverAction Writeable implementation
joegallo Jul 25, 2022
becd6af
Nicer fromXContent error message for conditions
joegallo Jul 25, 2022
8a60331
Add a condition Type (MIN/MAX) and drop isRequired
joegallo Jul 25, 2022
4d1f3c1
Add javadoc for areConditionsMet
joegallo Jul 25, 2022
912120a
Simpler serialization for min_age and min_size
joegallo Jul 25, 2022
99769ed
Merge branch 'main' into min-age
joegallo Jul 25, 2022
bcfd250
Spotless
joegallo Jul 25, 2022
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 5 additions & 0 deletions docs/changelog/83345.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
pr: 83345
summary: Add min_* conditions to rollover
area: ILM+SLM
type: enhancement
issues: []
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,7 @@
body: { "foo": "hello world" }
# make this doc visible in index stats
refresh: true

- do:
get:
index: logs_search
Expand Down Expand Up @@ -149,3 +149,37 @@
conditions:
max_docs: 1

---
"Will not rollover with only min_* conditions":
- skip:
version: " - 8.3.99"
reason: introduced in 8.4.0

# create index with alias and replica
- do:
indices.create:
index: logs-1
wait_for_active_shards: 1
body:
aliases:
logs_search: {}

# index first document and wait for refresh
- do:
index:
index: logs-1
id: "1"
body: { "foo": "hello world" }
refresh: true

# perform alias rollover with no result
- do:
catch: bad_request
indices.rollover:
alias: "logs_search"
wait_for_active_shards: 1
body:
conditions:
min_age: "0s"
min_docs: 1
- match: { error.reason: "Validation Failed: 1: at least one max_* rollover condition must be set when using min_* conditions;" }
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
"Rollover with min_age condition":
- skip:
version: " - 8.3.99"
reason: introduced in 8.4.0

# create index with alias and replica
- do:
indices.create:
index: logs-1
wait_for_active_shards: 1
body:
aliases:
logs_search: {}

# index first document and wait for refresh
- do:
index:
index: logs-1
id: "1"
body: { "foo": "hello world" }
refresh: true

# perform alias rollover with no result
- do:
indices.rollover:
alias: "logs_search"
wait_for_active_shards: 1
body:
conditions:
max_docs: 1
min_age: "7d"

- match: { conditions: { "[max_docs: 1]": true, "[min_age: 7d]": false } }
- match: { rolled_over: false }

# perform alias rollover
- do:
indices.rollover:
alias: "logs_search"
wait_for_active_shards: 1
body:
conditions:
max_docs: 1
min_age: "0s"

- match: { conditions: { "[max_docs: 1]": true, "[min_age: 0s]": true } }
- match: { rolled_over: true }
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
"Rollover with min_docs condition":
- skip:
version: " - 8.3.99"
reason: introduced in 8.4.0

# create index with alias and replica
- do:
indices.create:
index: logs-1
wait_for_active_shards: 1
body:
aliases:
logs_search: {}

# index first document and wait for refresh
- do:
index:
index: logs-1
id: "1"
body: { "foo": "hello world" }
refresh: true

# perform alias rollover with no result
- do:
indices.rollover:
alias: "logs_search"
wait_for_active_shards: 1
body:
conditions:
max_docs: 1
min_docs: 10

- match: { conditions: { "[max_docs: 1]": true, "[min_docs: 10]": false } }
- match: { rolled_over: false }

# perform alias rollover
- do:
indices.rollover:
alias: "logs_search"
wait_for_active_shards: 1
body:
conditions:
max_docs: 1
min_docs: 1

- match: { conditions: { "[max_docs: 1]": true, "[min_docs: 1]": true } }
- match: { rolled_over: true }
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
"Rollover with min_primary_shard_docs condition":
- skip:
version: " - 8.3.99"
reason: introduced in 8.4.0

# create index with alias and replica
- do:
indices.create:
index: logs-1
wait_for_active_shards: 1
body:
aliases:
logs_search: {}

# index first document and wait for refresh
- do:
index:
index: logs-1
id: "1"
body: { "foo": "hello world" }
refresh: true

# perform alias rollover with no result
- do:
indices.rollover:
alias: "logs_search"
wait_for_active_shards: 1
body:
conditions:
max_docs: 1
min_primary_shard_docs: 10

- match: { conditions: { "[max_docs: 1]": true, "[min_primary_shard_docs: 10]": false } }
- match: { rolled_over: false }

# perform alias rollover
- do:
indices.rollover:
alias: "logs_search"
wait_for_active_shards: 1
body:
conditions:
max_docs: 1
min_primary_shard_docs: 1

- match: { conditions: { "[max_docs: 1]": true, "[min_primary_shard_docs: 1]": true } }
- match: { rolled_over: true }
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
"Rollover with min_primary_shard_size condition":
- skip:
version: " - 8.3.99"
reason: introduced in 8.4.0

# create index with alias and replica
- do:
indices.create:
index: logs-1
wait_for_active_shards: 1
body:
aliases:
logs_search: {}

# index first document and wait for refresh
- do:
index:
index: logs-1
id: "1"
body: { "foo": "hello world" }
refresh: true

# perform alias rollover with no result
- do:
indices.rollover:
alias: "logs_search"
wait_for_active_shards: 1
body:
conditions:
max_docs: 1
min_primary_shard_size: "50gb"

- match: { conditions: { "[max_docs: 1]": true, "[min_primary_shard_size: 50gb]": false } }
- match: { rolled_over: false }

# perform alias rollover
- do:
indices.rollover:
alias: "logs_search"
wait_for_active_shards: 1
body:
conditions:
max_docs: 1
min_primary_shard_size: "0b"

- match: { conditions: { "[max_docs: 1]": true, "[min_primary_shard_size: 0b]": true } }
- match: { rolled_over: true }
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
---
"Rollover with min_size condition":
- skip:
version: " - 8.3.99"
reason: introduced in 8.4.0

# create index with alias and replica
- do:
indices.create:
index: logs-1
wait_for_active_shards: 1
body:
aliases:
logs_search: {}

# index first document and wait for refresh
- do:
index:
index: logs-1
id: "1"
body: { "foo": "hello world" }
refresh: true

# perform alias rollover with no result
- do:
indices.rollover:
alias: "logs_search"
wait_for_active_shards: 1
body:
conditions:
max_docs: 1
min_size: "50gb"

- match: { conditions: { "[max_docs: 1]": true, "[min_size: 50gb]": false } }
- match: { rolled_over: false }

# perform alias rollover
- do:
indices.rollover:
alias: "logs_search"
wait_for_active_shards: 1
body:
conditions:
max_docs: 1
min_size: "0b"

- match: { conditions: { "[max_docs: 1]": true, "[min_size: 0b]": true } }
- match: { rolled_over: true }
Original file line number Diff line number Diff line change
Expand Up @@ -472,6 +472,7 @@ public void testRolloverMaxSize() throws Exception {
.indices()
.prepareRolloverIndex("test_alias")
.addMaxIndexSizeCondition(new ByteSizeValue(randomNonNegativeLong(), ByteSizeUnit.BYTES))
.addMinIndexDocsCondition(1)
.get();
assertThat(response.getOldIndex(), equalTo("test-000002"));
assertThat(response.getNewIndex(), equalTo("test-000003"));
Expand Down Expand Up @@ -532,6 +533,7 @@ public void testRolloverMaxPrimaryShardSize() throws Exception {
.indices()
.prepareRolloverIndex("test_alias")
.addMaxPrimaryShardSizeCondition(new ByteSizeValue(randomNonNegativeLong(), ByteSizeUnit.BYTES))
.addMinIndexDocsCondition(1)
.get();
assertThat(response.getOldIndex(), equalTo("test-000002"));
assertThat(response.getNewIndex(), equalTo("test-000003"));
Expand Down Expand Up @@ -597,6 +599,7 @@ public void testRolloverMaxPrimaryShardDocs() throws Exception {
.indices()
.prepareRolloverIndex("test_alias")
.addMaxPrimaryShardDocsCondition(randomNonNegativeLong())
.addMinIndexDocsCondition(1)
.get();
assertThat(response.getOldIndex(), equalTo("test-000002"));
assertThat(response.getNewIndex(), equalTo("test-000003"));
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -20,11 +20,21 @@
*/
public abstract class Condition<T> implements NamedWriteable, ToXContentFragment {

/**
* Describes the type of condition - a min_* condition (MIN) or max_* condition (MAX).
*/
public enum Type {
MIN,
MAX
}

protected T value;
protected final String name;
protected final Type type;

protected Condition(String name) {
protected Condition(String name, Type type) {
this.name = name;
this.type = type;
}

/**
Expand Down Expand Up @@ -67,6 +77,10 @@ public String name() {
return name;
}

public Type type() {
return type;
}

/**
* Holder for index stats used to evaluate conditions
*/
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -24,12 +24,12 @@ public class MaxAgeCondition extends Condition<TimeValue> {
public static final String NAME = "max_age";

public MaxAgeCondition(TimeValue value) {
super(NAME);
super(NAME, Type.MAX);
this.value = value;
}

public MaxAgeCondition(StreamInput in) throws IOException {
super(NAME);
super(NAME, Type.MAX);
this.value = TimeValue.timeValueMillis(in.readLong());
}

Expand Down Expand Up @@ -64,7 +64,7 @@ public static MaxAgeCondition fromXContent(XContentParser parser) throws IOExcep
if (parser.nextToken() == XContentParser.Token.VALUE_STRING) {
return new MaxAgeCondition(TimeValue.parseTimeValue(parser.text(), NAME));
} else {
throw new IllegalArgumentException("invalid token: " + parser.currentToken());
throw new IllegalArgumentException("invalid token when parsing " + NAME + " condition: " + parser.currentToken());
}
}
}
Loading