Add single, combined ECS component template

[BenB196]

Summary

ECS component templates were added as part of #1156 and this provides a useful way of adding ECS fields to indexes. However, when you need to use a lot of ECS objects using the broken down component templates results in a long list of component templates, and when you need to change them (update ECS version), you need to change them all. It would be nice if a single large component template with all ECS fields was also generated that could be used.

Motivation:

This large component template would allow for easier management of indexes that use a lot of ECS fields, and currently require the management of multiple ECS component templates.

Other:

I'm not sure if this would be considered an anti-pattern with Elastic component templates or not, but I'd consider this a quality of life thing for index field management.

[ebeahan]

One challenge as the number of ECS fields grows is managing larger index mappings, and different use cases may not need all ECS fields defined. Part of the motivation for splitting the different field sets into their own component template example component templates was to demonstrate a more flexible approach - only include the ECS field sets you need where you need them.

I agree with you, though, that for some use cases having a single component template to reference would be simpler for pulling all ECS fields and there's also the mentioned overhead when you need to make changes to each composable template.

The ECS tooling supports users adding their own custom fields. We handle any custom field sets the same way we handle a ECS field set - it's generated in its own component template. With this proposed "all-in-one" ECS component template, I still see placing custom fields into their own component templates. This would enable the ECS tooling to generate an all-in-one component template and also any custom component templates.

We want to keep the legacy index templates for now, but I can see when we eventually remove those templates having the two options:

• The "all-in-one" component template with all ECS fields. Custom field sets would be placed in their own component templates.
• The smaller component templates, one for each ECS field set. Custom fields would continue to be placed in their own component templates.

Thoughts?

[djptek]

Hi @BenB196 I like this idea from the point of view of addressing multiple use cases, my concern is that there might be different ways to define those use cases, and their expected inputs and outputs which could generate a requirement for a overly complex merge/filter tool. In this case, rather than impose a convention through a tool, it might be simpler to suggest each user to define their own tooling to address their use case, for example user A might want a given subset expressed in flattened dot notation, user B might want the intersection of two subsets, defined in nested notation etc.

[BenB196]

I've actually thought about this a bit more, and I think my original request might be trying to solve the problem in the wrong way.

My original thought was that a singular merged component template would reduce management overhead of handling index templates with many component templates. However, I think this would just shift the management from the index template side to the component template side.

I now think merging two (or more) existing components templates would be an anti-pattern, as why have component templates if you're just going to merge them into something larger, and should probably be managed at the Index template level with additional tooling/API.

That being said a somewhat tangential thing (and might be worth a new issue), I do think that as ECS grows in general, knowing and understanding all of the fields and sections will become more of a challenge. I don't much input on this as I think its just an issue the comes with additional complexity, but it could possibly be handled more at the documentation level rather than implementation/tooling level.

[djptek]

Thanks for the feedback @BenB196

I totally agree about navigating one's way around all the fields, it's a lot to load into your stack

possibly be handled more at the documentation level rather than implementation/tooling level.

we recently added a link to a full list of fields which should be available in the next version of ECS, see: https://github.com/elastic/ecs/blob/master/generated/csv/fields.csv - it's just a raw table, however, I've found it useful for just <ctrl>F-ing my way around a single page rather then going up and down the docs tree

[ebeahan]

A new discussion on a similar topic has started in #1869. Let's take any further conversations to that thread.