Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Add new index for synthetics #61665

Closed
andrewvc opened this issue Aug 27, 2020 · 9 comments · Fixed by #62193
Closed

Add new index for synthetics #61665

andrewvc opened this issue Aug 27, 2020 · 9 comments · Fixed by #62193
Assignees
@dakrone
Labels
>enhancement Team:Data Management Meta label for data/management team

Comments

@andrewvc
Copy link
Contributor

For the new Elastic agent, there are two current index types are logs and metrics. Synthetics doesn't really fit with that, we should add a new synthetics index for data reported from heartbeat. This would be under the 'monitoring' category.

A question I have is about the fields that should be included in ES. We don't really have a 'minimal' subset. We need all our fields for the Uptime UI to be rendered correctly. Should we include all our field mappings with the index in ES?

CC @ruflin @ph
@andrewvc andrewvc added >enhancement needs:triage Requires assignment of a team area label labels Aug 27, 2020
@dakrone dakrone added :Core/Features/Features and removed needs:triage Requires assignment of a team area label labels Aug 27, 2020
@elasticmachine
Copy link
Collaborator

Pinging @elastic/es-core-features (:Core/Features/Features)

@elasticmachine elasticmachine added the Team:Data Management Meta label for data/management team label Aug 27, 2020
@dakrone
Copy link
Member

dakrone commented Aug 27, 2020

@andrewvc as I understand it, we only need a template (similar to our logs-*-* and metrics-*-* templates. We don't need the index to be pre-created, is that right?

@ruflin
Copy link
Contributor

ruflin commented Aug 31, 2020

I would suggest we use the same templates we have for logs / metrics at the moment for synthetics. The detailed data_stream templates will be set up Uptime before data is shipped but having the template in Elasticsearch ensures nobody messed with the indices/data streams in advance.

@andrewvc
Copy link
Contributor Author

andrewvc commented Sep 1, 2020

@ruflin that's fine with me.

@dakrone yes, I believe that's correct. @ruflin is there anything else needed?

@ruflin
Copy link
Contributor

ruflin commented Sep 1, 2020

A PR to get it into ES :-)

@andrewvc
Copy link
Contributor Author

andrewvc commented Sep 3, 2020

@dakrone I take it you're the one to open the PR?

@andrewvc andrewvc mentioned this issue Sep 3, 2020
Closed
10 tasks
@dakrone
Copy link
Member

dakrone commented Sep 8, 2020

@andrewvc I have been on vacation for the past 11 days, but yes, either me or someone on the core/features team would likely work on this

@dakrone
Copy link
Member

dakrone commented Sep 9, 2020

@ruflin

I would suggest we use the same templates we have for logs / metrics at the moment for synthetics.

These two templates have slightly different mappings (logs have a "message" field), should the synthetics-*-* template mappings include a "message" field?

dakrone added a commit to dakrone/elasticsearch that referenced this issue Sep 9, 2020
@dakrone
Add "synthetics-*-*" templates for synthetics fleet data
af938d8 
For the Elastic Agent we currently have `logs` and `metrics`, however, synthetic data doesn't belong
with those and thus we should have a place for it to live. This would be data reported from
heartbeat and under the 'monitoring' category.

This commit adds a composable index template for `synthetics-*-*` indices similar to the work in
 elastic#56709 and elastic#57629.

Resolves elastic#61665
@dakrone dakrone mentioned this issue Sep 9, 2020
Merged
@ruflin
Copy link
Contributor

ruflin commented Sep 10, 2020

My guess is that we don't need the message as so far synthetics will load its own templates for all indices.

dakrone added a commit that referenced this issue Sep 14, 2020
@dakrone
Add "synthetics-*-*" templates for synthetics fleet data (#62193)
d21ddb5 
* Add "synthetics-*-*" templates for synthetics fleet data

For the Elastic Agent we currently have `logs` and `metrics`, however, synthetic data doesn't belong
with those and thus we should have a place for it to live. This would be data reported from
heartbeat and under the 'monitoring' category.

This commit adds a composable index template for `synthetics-*-*` indices similar to the work in
 #56709 and #57629.

Resolves #61665
dakrone added a commit to dakrone/elasticsearch that referenced this issue Sep 14, 2020
@dakrone
Add "synthetics-*-*" templates for synthetics fleet data (elastic#62193)
9f84054 
* Add "synthetics-*-*" templates for synthetics fleet data

For the Elastic Agent we currently have `logs` and `metrics`, however, synthetic data doesn't belong
with those and thus we should have a place for it to live. This would be data reported from
heartbeat and under the 'monitoring' category.

This commit adds a composable index template for `synthetics-*-*` indices similar to the work in
 elastic#56709 and elastic#57629.

Resolves elastic#61665
dakrone added a commit that referenced this issue Sep 14, 2020
@dakrone
[7.x] Add "synthetics-*-*" templates for synthetics fleet data (#62193)…
6b2af30 
… (#62346)

* Add "synthetics-*-*" templates for synthetics fleet data

For the Elastic Agent we currently have `logs` and `metrics`, however, synthetic data doesn't belong
with those and thus we should have a place for it to live. This would be data reported from
heartbeat and under the 'monitoring' category.

This commit adds a composable index template for `synthetics-*-*` indices similar to the work in
 #56709 and #57629.

Resolves #61665
@paulb-elastic paulb-elastic mentioned this issue Oct 27, 2020
Closed
@dominiqueclarke dominiqueclarke mentioned this issue Jan 20, 2021
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dakrone dakrone

Labels
>enhancement Team:Data Management Meta label for data/management team
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add "synthetics-*-*" templates for synthetics fleet data dakrone/elasticsearch
4 participants
@dakrone @andrewvc @ruflin @elasticmachine