Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Fleet] Simulate template doesn't always use the right index template #164269

Closed
juliaElastic opened this issue Aug 21, 2023 · 5 comments · Fixed by #166941
Closed

[Fleet] Simulate template doesn't always use the right index template #164269

juliaElastic opened this issue Aug 21, 2023 · 5 comments · Fixed by #166941
Assignees
@juliaElastic
Labels
bug Fixes for quality problems that affect the customer experience Team:Fleet Team label for Observability Data Collection Fleet team v8.10.3 v8.11.0

Comments

@juliaElastic
Copy link
Contributor

Fleet tries to guess the index template name from the data stream name during simulateTemplate see this logic:

kibana/x-pack/plugins/fleet/server/services/epm/elasticsearch/template/template.ts

Lines 537 to 540 in 8532b99

function dataStreamNameToIndexTemplateName(dataStreamName: string): string {
const [type, dataset] = dataStreamName.split('-'); // ignore namespace at the end
return [type, dataset].join('-');
}

It seems that this logic doesn't work well in all cases, there should be a more robust way to find an index template of a data stream.

Steps to reproduce:

  • Install apm package with stack version 8.7.1, use non-default namespace.
  • Upgrade the stack to 8.8.2 or 8.9.0 and see errors with simulate template e.g. unable to simulate template [metrics-apm.app.event_gateway] that does not exist
  • E.g. Datastream metrics-apm.app.event_gateway-<dataset> is attempted to be simulated as index template metrics-apm.app.event_gateway while the actual index template is metrics-apm.app
@juliaElastic juliaElastic added bug Fixes for quality problems that affect the customer experience Team:Fleet Team label for Observability Data Collection Fleet team labels Aug 21, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/fleet (Team:Fleet)

@juliaElastic
Copy link
Contributor Author

juliaElastic commented Aug 22, 2023 •
edited
Loading

I think this issue is related (the same?) to #160775

@jlind23
Copy link
Contributor

jlind23 commented Aug 22, 2023

@juliaElastic Correct, closing.

@jlind23 jlind23 closed this as not planned Won't fix, can't repro, duplicate, stale Aug 22, 2023
@lucabelluccini
Copy link
Contributor

Just to clarify, metrics-apm.app which is owned by APM integration is not related to the built-in index templates mentioned in #160775.

Also, the problem could occur in case a user clones an integration Index Template to customize their behavior per namespace. Example:

data stream name attempted simulate actual index template
metrics-apm.app.somename-somenamespace metrics-apm.app.somename metrics-apm.app-somenamespace

Also I think the dataStreamNameToIndexTemplateName should rely on the Data Streams API (but it might expensive) as it contains the name of the Index Template which originated the Data stream.

@hop-dev
Copy link
Contributor

hop-dev commented Aug 23, 2023

I think this issue is different to #160775.

@lucabelluccini is right, if we could use the API to find the origin datastream (and ensure that it is a managed datastream) then this would be better.

@hop-dev hop-dev reopened this Aug 23, 2023
@juliaElastic juliaElastic self-assigned this Sep 21, 2023
This was referenced Sep 21, 2023
Merged
Closed
juliaElastic added a commit that referenced this issue Sep 22, 2023
@juliaElastic
[Fleet] fix index template from datastream name (#166941)
3ff82f2 
## Summary

Resolve #164269

Some context why I picked this up now:
#162772 (comment)

To verify:
- Make sure 8.8+ apm package is installed
- Create data stream `PUT _data_stream/metrics-apm.app.default-default`
- Reinstall apm package from API or UI
- Check kibana info logs, expect to not see simulate template error and
rollover like below

```
[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet] Mappings update for metrics-apm.app.default-default failed due to ResponseError: illegal_argument_exception
        Root causes:
                illegal_argument_exception: unable to simulate template [metrics-apm.app.default] that does not exist
[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet] Triggering a rollover for metrics-apm.app.default-default
```


### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Sep 22, 2023
@juliaElastic
[Fleet] fix index template from datastream name (elastic#166941)
528b930 
## Summary

Resolve elastic#164269

Some context why I picked this up now:
elastic#162772 (comment)

To verify:
- Make sure 8.8+ apm package is installed
- Create data stream `PUT _data_stream/metrics-apm.app.default-default`
- Reinstall apm package from API or UI
- Check kibana info logs, expect to not see simulate template error and
rollover like below

```
[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet] Mappings update for metrics-apm.app.default-default failed due to ResponseError: illegal_argument_exception
        Root causes:
                illegal_argument_exception: unable to simulate template [metrics-apm.app.default] that does not exist
[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet] Triggering a rollover for metrics-apm.app.default-default
```

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

(cherry picked from commit 3ff82f2)
kibanamachine referenced this issue Sep 26, 2023
@kibanamachine @juliaElastic
[8.10] [Fleet] fix index template from datastream name (#166941) (#16…
09ee1ee 
…7008)

# Backport

This will backport the following commits from `main` to `8.10`:
- [[Fleet] fix index template from datastream name
(#166941)](#166941)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Julia
Bardi","email":"90178898+juliaElastic@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-09-22T08:20:05Z","message":"[Fleet]
fix index template from datastream name (#166941)\n\n##
Summary\r\n\r\nResolve
https://github.com/elastic/kibana/issues/164269\r\n\r\nSome context why
I picked this up
now:\r\nhttps://github.com//issues/162772#issuecomment-1728031080\r\n\r\nTo
verify:\r\n- Make sure 8.8+ apm package is installed\r\n- Create data
stream `PUT _data_stream/metrics-apm.app.default-default`\r\n- Reinstall
apm package from API or UI\r\n- Check kibana info logs, expect to not
see simulate template error and\r\nrollover like
below\r\n\r\n```\r\n[2023-09-21T15:54:36.559+02:00][INFO
][plugins.fleet] Mappings update for metrics-apm.app.default-default
failed due to ResponseError: illegal_argument_exception\r\n Root
causes:\r\n illegal_argument_exception: unable to simulate template
[metrics-apm.app.default] that does not
exist\r\n[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet]
Triggering a rollover for
metrics-apm.app.default-default\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"3ff82f2c17e532fed5d5544ed9bbae4f6e7331af","branchLabelMapping":{"^v8.11.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Fleet","backport:prev-minor","v8.11.0"],"number":166941,"url":"https://github.com/elastic/kibana/pull/166941","mergeCommit":{"message":"[Fleet]
fix index template from datastream name (#166941)\n\n##
Summary\r\n\r\nResolve
https://github.com/elastic/kibana/issues/164269\r\n\r\nSome context why
I picked this up
now:\r\nhttps://github.com//issues/162772#issuecomment-1728031080\r\n\r\nTo
verify:\r\n- Make sure 8.8+ apm package is installed\r\n- Create data
stream `PUT _data_stream/metrics-apm.app.default-default`\r\n- Reinstall
apm package from API or UI\r\n- Check kibana info logs, expect to not
see simulate template error and\r\nrollover like
below\r\n\r\n```\r\n[2023-09-21T15:54:36.559+02:00][INFO
][plugins.fleet] Mappings update for metrics-apm.app.default-default
failed due to ResponseError: illegal_argument_exception\r\n Root
causes:\r\n illegal_argument_exception: unable to simulate template
[metrics-apm.app.default] that does not
exist\r\n[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet]
Triggering a rollover for
metrics-apm.app.default-default\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"3ff82f2c17e532fed5d5544ed9bbae4f6e7331af"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.11.0","labelRegex":"^v8.11.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/166941","number":166941,"mergeCommit":{"message":"[Fleet]
fix index template from datastream name (#166941)\n\n##
Summary\r\n\r\nResolve
https://github.com/elastic/kibana/issues/164269\r\n\r\nSome context why
I picked this up
now:\r\nhttps://github.com//issues/162772#issuecomment-1728031080\r\n\r\nTo
verify:\r\n- Make sure 8.8+ apm package is installed\r\n- Create data
stream `PUT _data_stream/metrics-apm.app.default-default`\r\n- Reinstall
apm package from API or UI\r\n- Check kibana info logs, expect to not
see simulate template error and\r\nrollover like
below\r\n\r\n```\r\n[2023-09-21T15:54:36.559+02:00][INFO
][plugins.fleet] Mappings update for metrics-apm.app.default-default
failed due to ResponseError: illegal_argument_exception\r\n Root
causes:\r\n illegal_argument_exception: unable to simulate template
[metrics-apm.app.default] that does not
exist\r\n[2023-09-21T15:54:36.559+02:00][INFO ][plugins.fleet]
Triggering a rollover for
metrics-apm.app.default-default\r\n```\r\n\r\n\r\n### Checklist\r\n\r\n-
[x] [Unit or
functional\r\ntests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)\r\nwere
updated or added to match the most common
scenarios","sha":"3ff82f2c17e532fed5d5544ed9bbae4f6e7331af"}}]}]
BACKPORT-->

Co-authored-by: Julia Bardi <90178898+juliaElastic@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@juliaElastic juliaElastic

Labels
bug Fixes for quality problems that affect the customer experience Team:Fleet Team label for Observability Data Collection Fleet team v8.10.3 v8.11.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Fleet] fix index template from datastream name juliaElastic/kibana
5 participants
@lucabelluccini @hop-dev @elasticmachine @jlind23 @juliaElastic