[Maps] update sample data functional test to use a longer time range to ensure stable results #132744

nreese · 2022-05-23T19:34:01Z

#91205 re-enabled Maps sample_data functional tests. The PR also switched the logic for setting the time range from setting the absolute range from 6 months in past to 6 months in the future to set a quick select time range 30 days in the past and 40 days in the future

The problem with this change is that the data sets extend into the future more than 40 days. Since sample data is shifted to be relative to current time, this can cause different results between runs. The screen shot below shows how web logs sample data set extends past 40 days in the future. Having a time range long enough to include all sample data ensures the results are the same between test runs.

The PR also

removes unused quick times to highlight the one used by the test.
Fixes same problem copied to upgrade test

…to ensure stable results

…-fix'

…ample_data

liza-mae · 2022-05-23T22:10:21Z

@nreese when this one is ready, I can run against cloud CI to get those results, let me know.

elasticmachine · 2022-05-24T17:46:19Z

Pinging @elastic/kibana-gis (Team:Geo)

nickpeihl

lgtm! 🤞

code review

nreese · 2022-05-25T14:03:16Z

@elasticmachine merge upstream

liza-mae · 2022-05-25T15:47:18Z

Results from cloud: https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-upgrade-snapshot-tests+main/11/testReport/

Looks like an improvement for web logs, but still needs a small tolerance increase and some others are failing for ElementClickInterceptedError need to investigate.

[Kibana Core Tests1.x-pack/test/upgrade/apps/maps/maps_smoke_tests·ts.upgrade upgrade maps smoke tests space: automation, name: web logs should load saved object and display layers](https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-upgrade-snapshot-tests+main/11/JOB=upgradeTest5n,TASK=kibana_upgrade_tests,node=ess-testing/testReport/Kibana%20Core%20Tests1/x-pack_test_upgrade_apps_maps_maps_smoke_tests%C2%B7ts/upgrade_upgrade_maps_smoke_tests_space__automation__name__web_logs_should_load_saved_object_and_display_layers/)
[ Stack Trace](javascript:hideFailureSummary('id1239215-stacktrace'))
Error: expected '0.032' to be below 0.031
    at Assertion.assert (/var/lib/jenkins/workspace/elastic+estf-cloud-upgrade-snapshot-tests+main/JOB/upgradeTest5n/TASK/kibana_upgrade_tests/node/ess-testing/ci/upgrade/kibana/build/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.lessThan.Assertion.below (/var/lib/jenkins/workspace/elastic+estf-cloud-upgrade-snapshot-tests+main/JOB/upgradeTest5n/TASK/kibana_upgrade_tests/node/ess-testing/ci/upgrade/kibana/build/kibana/node_modules/@kbn/expect/expect.js:336:8)
    at Function.lessThan (/var/lib/jenkins/workspace/elastic+estf-cloud-upgrade-snapshot-tests+main/JOB/upgradeTest5n/TASK/kibana_upgrade_tests/node/ess-testing/ci/upgrade/kibana/build/kibana/node_modules/@kbn/expect/expect.js:531:15)
    at Context.<anonymous> (test/upgrade/apps/maps/maps_smoke_tests.ts:133:54)
    at Object.apply (/var/lib/jenkins/workspace/elastic+estf-cloud-upgrade-snapshot-tests+main/JOB/upgradeTest5n/TASK/kibana_upgrade_tests/node/ess-testing/ci/upgrade/kibana/build/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

However a larger difference is now seen for flights in some cases:

[Kibana Core Tests1.x-pack/test/upgrade/apps/maps/maps_smoke_tests·ts.upgrade upgrade maps smoke tests space: default, name: flights should load saved object and display layers](https://internal-ci.elastic.co/view/Stack%20Tests/job/elastic+estf-cloud-upgrade-snapshot-tests+main/11/JOB=upgradeTest5d,TASK=kibana_upgrade_tests,node=ess-testing/testReport/Kibana%20Core%20Tests1/x-pack_test_upgrade_apps_maps_maps_smoke_tests%C2%B7ts/upgrade_upgrade_maps_smoke_tests_space__default__name__flights_should_load_saved_object_and_display_layers/)
[ Stack Trace](javascript:hideFailureSummary('id1239004-stacktrace'))
Error: expected '0.054' to be below 0.031
    at Assertion.assert (/var/lib/jenkins/workspace/elastic+estf-cloud-upgrade-snapshot-tests+main/JOB/upgradeTest5d/TASK/kibana_upgrade_tests/node/ess-testing/ci/upgrade/kibana/build/kibana/node_modules/@kbn/expect/expect.js:100:11)
    at Assertion.lessThan.Assertion.below (/var/lib/jenkins/workspace/elastic+estf-cloud-upgrade-snapshot-tests+main/JOB/upgradeTest5d/TASK/kibana_upgrade_tests/node/ess-testing/ci/upgrade/kibana/build/kibana/node_modules/@kbn/expect/expect.js:336:8)
    at Function.lessThan (/var/lib/jenkins/workspace/elastic+estf-cloud-upgrade-snapshot-tests+main/JOB/upgradeTest5d/TASK/kibana_upgrade_tests/node/ess-testing/ci/upgrade/kibana/build/kibana/node_modules/@kbn/expect/expect.js:531:15)
    at Context.<anonymous> (test/upgrade/apps/maps/maps_smoke_tests.ts:108:54)
    at Object.apply (/var/lib/jenkins/workspace/elastic+estf-cloud-upgrade-snapshot-tests+main/JOB/upgradeTest5d/TASK/kibana_upgrade_tests/node/ess-testing/ci/upgrade/kibana/build/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

nreese · 2022-05-25T19:42:17Z

@liza-mae

It looks like the screen sizes are different then the baselines. Could this be caused by cloud tests not using buildkite? Below are the diffs for those 2 tests. Maybe instead of a screen shot comparison, the tests could be updated to make sure data is successfully pulled and there are not unexpected errors?

Diff for flights test

Diff for e-commerce test

liza-mae · 2022-05-25T20:31:01Z

@nreese yes the upgrade tests are still in Jenkins. My goal right now is to unblock the 8.3 snapshot testing (which was recently added), so for me increasing the tolerance is okay for the existing tests.

Then later in another PR, I can pair up with you or someone from GIS team to see the best way to handle maps testing in upgrade feature. We can certainly change the testing, but I would need to understand what you mean by verify data is successfully pulled versus using screenshots, I expect the data to be dynamic. Since, I am not too familiar with maps tests and would need to get some additional information before rewriting the tests, it is going to hold up stabilizing the test runs for 8.3 while I do this.

Would you be okay with reverting the upgrade maps smoke test changes in this PR and I will add the existing screenshots to upgrade directory with the tolerance increase in the PR I have open now? Afterwards I will schedule a pair session to rewrite maps tests for upgrade.

liza-mae

Let's revert the changes in x-pack/test/upgrade/apps/maps/maps_smoke_tests.ts in this PR and just increase the tolerance on the existing screenshots in #132374. We will do another PR later to rewrite the maps smoke tests for upgrade testing.

…ample_data

nreese · 2022-05-25T20:37:29Z

Let's revert the changes in x-pack/test/upgrade/apps/maps/maps_smoke_tests.ts in this PR and just increase the tolerance on the existing screenshots in #132374. We will do another PR later to rewrite the maps smoke tests for upgrade testing.

I have reverted changes to x-pack/test/upgrade/apps/maps/maps_smoke_tests.ts

…-fix'

liza-mae

LGTM

nreese · 2022-06-06T12:25:00Z

@elasticmachine merge upstream

kibana-ci · 2022-06-06T13:25:08Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💔 Build #47735 failed b9b9dff
💚 Build #47617 succeeded e268885
💚 Build #47407 succeeded b30d74a
💚 Build #47320 succeeded 898d408
💔 Build #47247 failed 525b40b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

nreese added 2 commits May 23, 2022 13:24

[Maps] update sample data functional test to use a longer time range …

7b59490

…to ensure stable results

fix map upgrade tests

6954b08

nreese mentioned this pull request May 23, 2022

[Upgrade] Fix tests for 8.3 snapshots #132374

Merged

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

5fb8546

…-fix'

liza-mae self-requested a review May 23, 2022 20:50

nreese added 2 commits May 23, 2022 16:01

lower threshold to make tests fail to get new baseline from CI

712a22a

Merge branch 'fix_sample_data' of github.com:nreese/kibana into fix_s…

480e6d9

…ample_data

nreese added 3 commits May 24, 2022 06:40

update e-commerce snapshot

b1cf873

update flights baseline

525b40b

update web logs baseline image

898d408

nreese marked this pull request as ready for review May 24, 2022 17:46

nreese requested a review from a team as a code owner May 24, 2022 17:46

nreese added [Deprecated-Use Team:Presentation]Team:Geo Former Team Label for Geo Team. Now use Team:Presentation release_note:skip Skip the PR/issue when compiling release notes backport:skip This commit does not require backporting labels May 24, 2022

nreese requested a review from nickpeihl May 24, 2022 17:47

nreese added the v8.3.0 label May 24, 2022

use session images instead of baseline images

b30d74a

nickpeihl approved these changes May 25, 2022

View reviewed changes

Merge branch 'main' into fix_sample_data

e268885

liza-mae suggested changes May 25, 2022

View reviewed changes

nreese added 2 commits May 25, 2022 14:36

revert changes to x-pack/test/upgrade/apps/maps/maps_smoke_tests.ts

2e6db1c

Merge branch 'fix_sample_data' of github.com:nreese/kibana into fix_s…

fa568ac

…ample_data

[CI] Auto-commit changed files from 'node scripts/eslint --no-cache -…

b9b9dff

…-fix'

liza-mae approved these changes May 25, 2022

View reviewed changes

nreese added v8.4.0 and removed v8.3.0 labels Jun 6, 2022

Merge branch 'main' into fix_sample_data

a83e271

nreese merged commit d6cf39d into elastic:main Jun 6, 2022

sophiec20 added the Feature:Maps label Aug 21, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Maps] update sample data functional test to use a longer time range to ensure stable results #132744

[Maps] update sample data functional test to use a longer time range to ensure stable results #132744

nreese commented May 23, 2022 •

edited

Loading

liza-mae commented May 23, 2022

elasticmachine commented May 24, 2022

nickpeihl left a comment

nreese commented May 25, 2022

liza-mae commented May 25, 2022

nreese commented May 25, 2022

liza-mae commented May 25, 2022

liza-mae left a comment

nreese commented May 25, 2022

liza-mae left a comment

nreese commented Jun 6, 2022

kibana-ci commented Jun 6, 2022

[Maps] update sample data functional test to use a longer time range to ensure stable results #132744

[Maps] update sample data functional test to use a longer time range to ensure stable results #132744

Conversation

nreese commented May 23, 2022 • edited Loading

liza-mae commented May 23, 2022

elasticmachine commented May 24, 2022

nickpeihl left a comment

Choose a reason for hiding this comment

nreese commented May 25, 2022

liza-mae commented May 25, 2022

nreese commented May 25, 2022

liza-mae commented May 25, 2022

liza-mae left a comment

Choose a reason for hiding this comment

nreese commented May 25, 2022

liza-mae left a comment

Choose a reason for hiding this comment

nreese commented Jun 6, 2022

kibana-ci commented Jun 6, 2022

💚 Build Succeeded

Metrics [docs]

History

nreese commented May 23, 2022 •

edited

Loading