[CI] TimeSeriesLifecycleActionsIT testWaitForSnapshot failure #50781

martijnvg · 2020-01-09T08:46:21Z

Failure:

2> java.lang.AssertionError: 
    Expected: "completed"
         but: was "wait_for_snapshot"
        at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:18)
        at org.junit.Assert.assertThat(Assert.java:956)
        at org.junit.Assert.assertThat(Assert.java:923)
        at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.lambda$testWaitForSnapshot$23(TimeSeriesLifecycleActionsIT.java:344)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:879)
        at org.elasticsearch.test.ESTestCase.assertBusy(ESTestCase.java:852)
        at org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testWaitForSnapshot(TimeSeriesLifecycleActionsIT.java:344)

Reproduce line:

./gradlew ':x-pack:plugin:ilm:qa:multi-node:integTestRunner' --tests "org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT.testWaitForSnapshot" -Dtests.seed=EC28318704EF9660 -Dtests.security.manager=true -Dtests.locale=nb -Dtests.timezone=Pacific/Tarawa -Dcompiler.java=13

This didn't reproduce for me.

Build url: https://gradle-enterprise.elastic.co/s/uirpg2i73okrq

This test failed a few times already:
https://build-stats.elastic.co/app/kibana#/discover?_g=(refreshInterval:(pause:!t,value:0),time:(from:now-30d,mode:quick,to:now))&_a=(columns:!(_source),index:e58bf320-7efd-11e8-bf69-63c8ef516157,interval:auto,query:(language:lucene,query:'class:*TimeSeriesLifecycleActionsIT+AND+test:testWaitForSnapshot'),sort:!(time,desc))

The text was updated successfully, but these errors were encountered:

elasticmachine · 2020-01-09T08:46:23Z

Pinging @elastic/es-core-features (:Core/Features/ILM+SLM)

This one is failing a lot => mute Relates elastic#50781

This one is failing a lot => mute Relates #50781

original-brownbear · 2020-01-09T13:10:31Z

Muted in 170e100

original-brownbear · 2020-01-09T15:26:45Z

This looks like the same issue in testWaitForSnapshotSlmExecutedBefore https://gradle-enterprise.elastic.co/s/t65m4uewdrzlo

mark-vieira · 2020-01-09T19:26:11Z

Yeah, I'm seeing a number of failures for different tests in TimeSeriesLifecycleActionsIT starting around mid December.

Page takes a minute to load: https://gradle-enterprise.elastic.co/scans/tests?failures.failureClassification=non_verification&list.offset=0&list.size=50&list.sortColumn=startTime&list.sortOrder=desc&search.buildToolType=gradle&search.buildToolType=maven&search.startTimeMax=1578643199999&search.startTimeMin=1575187200000&tests.container=org.elasticsearch.xpack.ilm.TimeSeriesLifecycleActionsIT&tests.sortField=FAILED&tests.unstableOnly&trends.section=overview&trends.timeResolution=day&viewer.tzOffset=-480

nik9000 · 2020-01-09T19:37:36Z

This showed up with #50454

This change adds some randomness and cleanup step to TimeSeriesLifecycleActionsIT#testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore tests in attempt to make them stable. Reletes to elastic#50781

mark-vieira · 2020-01-09T23:10:21Z

Let's please give this some immediate attention. TimeSeriesLifecycleActionsIT is the most common test failure across CI in the past month excluding Windows packaging tests. That could just mean muting for now.

Muting the test while I work on fixing its flakiness. Relates elastic#50781

#50842) Muting the test while I work on fixing its flakiness. Relates #50781

probakowski · 2020-01-10T09:08:23Z

testWaitForSnapshotSlmExecutedBefore muted in ce50e8e

This change adds some randomness and cleanup step to TimeSeriesLifecycleActionsIT#testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore tests in attempt to make them stable. Reletes to elastic#50781

… and testWaitForSnapshotSlmExecutedBefore test (#50818) * Fix flaky TimeSeriesLifecycleActionsIT#testWaitForSnapshot test This change adds some randomness and cleanup step to TimeSeriesLifecycleActionsIT#testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore tests in attempt to make them stable. Reletes to #50781 * Formatting changes * Longer timeout

* ILM action to wait for SLM policy execution (#50454) This change add new ILM action to wait for SLM policy execution to ensure that index has snapshot before deletion. Closes #45067 * Fix flaky TimeSeriesLifecycleActionsIT#testWaitForSnapshot test This change adds some randomness and cleanup step to TimeSeriesLifecycleActionsIT#testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore tests in attempt to make them stable. Reletes to #50781 * Formatting changes * Longer timeout * Fix Map.of in Java8 * Unused import removed

probakowski · 2020-01-14T00:39:35Z

Closing this one, looks like it's successfully fixed by #50818

dakrone · 2020-01-15T18:13:29Z

The failed today on the master intake branch: https://gradle-enterprise.elastic.co/s/aalmk4jiei77w

The same error:

java.lang.AssertionError
: 

Expected: "completed"
     but: was "wait_for_snapshot"

tvernum · 2020-01-20T04:51:53Z

Failed again

Will re-mute.

This test was recently un-muted, but is still failing Relates: #50781

This test was recently un-muted, but is still failing Relates: elastic#50781 Backport of: elastic#51203

This test was recently un-muted, but is still failing Relates: #50781 Backport of: #51203

This test was recently un-muted, but is still failing Relates: elastic#50781 Backport of: elastic#51203

This test was recently un-muted, but is still failing Relates: #50781 Backport of: #51203

This one is failing a lot => mute Relates elastic#50781

elastic#50842) Muting the test while I work on fixing its flakiness. Relates elastic#50781

… and testWaitForSnapshotSlmExecutedBefore test (elastic#50818) * Fix flaky TimeSeriesLifecycleActionsIT#testWaitForSnapshot test This change adds some randomness and cleanup step to TimeSeriesLifecycleActionsIT#testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore tests in attempt to make them stable. Reletes to elastic#50781 * Formatting changes * Longer timeout

This test was recently un-muted, but is still failing Relates: elastic#50781

…edBefore Due to #50781

droberts195 · 2020-01-28T14:33:41Z

testWaitForSnapshotSlmExecutedBefore failed again on master with the same error as before in https://gradle-enterprise.elastic.co/s/zqejrnbnah5bu

I re-muted testWaitForSnapshotSlmExecutedBefore on master in c6c628b

cbuescher · 2020-01-29T12:06:48Z

Another failure today on 7.x:
https://gradle-enterprise.elastic.co/s/jfkhpckv7ohxk/console-log?task=:x-pack:plugin:ilm:qa:multi-node:integTestRunner
Will also mute on 7.x for now

…edBefore Due to #50781

dnhatn · 2020-02-13T00:06:54Z

Another instance https://gradle-enterprise.elastic.co/s/rd72cz5aa5lo2.

probakowski · 2020-02-20T21:21:09Z

I'll close this issue, given that it hasn't failed for over 900 builds since #51892 has been merged:
https://ela.st/wait-for-snapshot
https://ela.st/wait-for-snapshot-before

martijnvg added >test-failure Triaged test failures from CI :Data Management/ILM+SLM Index and Snapshot lifecycle management labels Jan 9, 2020

original-brownbear added a commit to original-brownbear/elasticsearch that referenced this issue Jan 9, 2020

Mute TimeSeriesLifecycleActionsIT.testWaitForSnapshot

399d832

This one is failing a lot => mute Relates elastic#50781

original-brownbear mentioned this issue Jan 9, 2020

Mute TimeSeriesLifecycleActionsIT.testWaitForSnapshot #50790

Merged

original-brownbear added a commit that referenced this issue Jan 9, 2020

Mute TimeSeriesLifecycleActionsIT.testWaitForSnapshot (#50790)

170e100

This one is failing a lot => mute Relates #50781

dakrone assigned probakowski Jan 9, 2020

probakowski mentioned this issue Jan 9, 2020

Increase timeouts in TimeSeriesLifecycleActionsIT#testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore test #50818

Merged

probakowski added a commit to probakowski/elasticsearch that referenced this issue Jan 10, 2020

Mute TimeSeriesLifecycleActionsIT.testWaitForSnapshotSlmExecutedBefore

0dd7ea9

Muting the test while I work on fixing its flakiness. Relates elastic#50781

probakowski mentioned this issue Jan 10, 2020

Mute TimeSeriesLifecycleActionsIT.testWaitForSnapshotSlmExecutedBefore #50842

Merged

probakowski added a commit that referenced this issue Jan 10, 2020

Mute TimeSeriesLifecycleActionsIT.testWaitForSnapshotSlmExecutedBefore (

ce50e8e

#50842) Muting the test while I work on fixing its flakiness. Relates #50781

amirhmd mentioned this issue Jan 10, 2020

Add the new 'maintenance' privilege containing 4 actions (#29998) #50643

Merged

probakowski closed this as completed Jan 14, 2020

dakrone reopened this Jan 15, 2020

tvernum mentioned this issue Jan 20, 2020

Mute TimeSeriesLifecycleActionsIT.waitForSnapshot #51203

Merged

tvernum added a commit that referenced this issue Jan 20, 2020

Mute TimeSeriesLifecycleActionsIT.waitForSnapshot (#51203)

7d74a4d

This test was recently un-muted, but is still failing Relates: #50781

tvernum added a commit to tvernum/elasticsearch that referenced this issue Jan 20, 2020

Mute TimeSeriesLifecycleActionsIT.waitForSnapshot

54faed9

This test was recently un-muted, but is still failing Relates: elastic#50781 Backport of: elastic#51203

tvernum mentioned this issue Jan 20, 2020

Mute TimeSeriesLifecycleActionsIT.waitForSnapshot #51208

Merged

tvernum added a commit that referenced this issue Jan 20, 2020

Mute TimeSeriesLifecycleActionsIT.waitForSnapshot (#51208)

a0ca824

This test was recently un-muted, but is still failing Relates: #50781 Backport of: #51203

tvernum added a commit to tvernum/elasticsearch that referenced this issue Jan 21, 2020

Mute TimeSeriesLifecycleActionsIT.waitForSnapshot

e55335e

This test was recently un-muted, but is still failing Relates: elastic#50781 Backport of: elastic#51203

tvernum mentioned this issue Jan 21, 2020

Mute TimeSeriesLifecycleActionsIT.waitForSnapshot #51245

Merged

tvernum added a commit that referenced this issue Jan 21, 2020

Mute TimeSeriesLifecycleActionsIT.waitForSnapshot (#51245)

25c422b

This test was recently un-muted, but is still failing Relates: #50781 Backport of: #51203

SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this issue Jan 23, 2020

Mute TimeSeriesLifecycleActionsIT.testWaitForSnapshot (elastic#50790)

c093f7f

This one is failing a lot => mute Relates elastic#50781

SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this issue Jan 23, 2020

Mute TimeSeriesLifecycleActionsIT.testWaitForSnapshotSlmExecutedBefore (

331c6c8

elastic#50842) Muting the test while I work on fixing its flakiness. Relates elastic#50781

SivagurunathanV pushed a commit to SivagurunathanV/elasticsearch that referenced this issue Jan 23, 2020

Mute TimeSeriesLifecycleActionsIT.waitForSnapshot (elastic#51203)

76427ef

This test was recently un-muted, but is still failing Relates: elastic#50781

droberts195 added a commit that referenced this issue Jan 28, 2020

[TEST] Mute TimeSeriesLifecycleActionsIT.testWaitForSnapshotSlmExecut…

c6c628b

…edBefore Due to #50781

cbuescher pushed a commit that referenced this issue Jan 29, 2020

[TEST] Mute TimeSeriesLifecycleActionsIT.testWaitForSnapshotSlmExecut…

e0e35b7

…edBefore Due to #50781

probakowski mentioned this issue Feb 4, 2020

Make TimeSeriesLifecycleActionsIT.testWaitForSnapshot and testWaitForSnapshotSlmExecutedBefore wait for snaphost #51892

Merged

probakowski closed this as completed Feb 20, 2020

probakowski reopened this Feb 27, 2020

probakowski closed this as completed Feb 27, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[CI] TimeSeriesLifecycleActionsIT testWaitForSnapshot failure #50781

[CI] TimeSeriesLifecycleActionsIT testWaitForSnapshot failure #50781

martijnvg commented Jan 9, 2020

elasticmachine commented Jan 9, 2020

original-brownbear commented Jan 9, 2020

original-brownbear commented Jan 9, 2020

mark-vieira commented Jan 9, 2020

nik9000 commented Jan 9, 2020

mark-vieira commented Jan 9, 2020

probakowski commented Jan 10, 2020

probakowski commented Jan 14, 2020

dakrone commented Jan 15, 2020

tvernum commented Jan 20, 2020

droberts195 commented Jan 28, 2020

cbuescher commented Jan 29, 2020

dnhatn commented Feb 13, 2020

probakowski commented Feb 20, 2020

[CI] TimeSeriesLifecycleActionsIT testWaitForSnapshot failure #50781

[CI] TimeSeriesLifecycleActionsIT testWaitForSnapshot failure #50781

Comments

martijnvg commented Jan 9, 2020

elasticmachine commented Jan 9, 2020

original-brownbear commented Jan 9, 2020

original-brownbear commented Jan 9, 2020

mark-vieira commented Jan 9, 2020

nik9000 commented Jan 9, 2020

mark-vieira commented Jan 9, 2020

probakowski commented Jan 10, 2020

probakowski commented Jan 14, 2020

dakrone commented Jan 15, 2020

tvernum commented Jan 20, 2020

droberts195 commented Jan 28, 2020

cbuescher commented Jan 29, 2020

dnhatn commented Feb 13, 2020

probakowski commented Feb 20, 2020