Do not drop errors when checking for templates

[urso]

• Bug

What does this PR do?

Return an error when the HTTP request to check if a template is installed failed due to IO or in case the status code indicates that ES is not stable.

Why is it important?

When connecting to Elasticsearch Beats do check if a template exists and attempt to create the template if that is not the case. If there is an error while checking for the template, the error should be handled. By not handling the error the Beat might attempt to install the template, even if it was already installed. This can lead to issues like Beats overwriting an already existing and modified index with settings that are not supposed to be applied. Plus if Elasticsearch is currently unstable (return code 429 or 5xx), we might trigger a thundering herd problem with hundreds of Beats attempting to overwrite templates in an unstable cluster. By returning early on error, the output will back off exponentially (with random jitter), potentially reducing the chance of breaking the existing template or finally bringing Elasticsearch down to its knees.

Checklist

• My code follows the style guidelines of this project
• I have commented my code, particularly in hard-to-understand areas
  - [ ] I have made corresponding changes to the documentation
  - [ ] I have made corresponding change to the default configuration files
  - [ ] I have added tests that prove my fix is effective or that my feature works
• I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Related issues

• Closes Elasticsearch template check treats non-200 response as "template not found" #24144

[elasticmachine]

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

• Build Cause: urso commented: /test metricbeat

• Start Time: 2021-03-08T11:52:48.275+0000

• Duration: 62 min 36 sec

• Commit: 46de4f5

Test stats 🧪

Test	Results
Failed	0
Passed	45789
Skipped	4916
Total	50705

Trends 🧪

💚 Flaky test report

Tests succeeded.

Expand to view the summary

Test stats 🧪

Test	Results
Failed	0
Passed	45789
Skipped	4916
Total	50705

[elasticmachine]

Pinging @elastic/agent (Team:Agent)

[urso]

/test metricbeat