[Security Solution][Alerts] Detection alerts indices are missing data_stream ECS field mappings

[marshallmain]

In #123486 we removed the constant_keyword data stream fields to avoid situations where alerts became un-indexable if they had varying values for those fields. However, we didn't add regular keyword field mappings to replace them similar to how we did for the legacy .siem-signals mappings. The result is that queries on data_stream.* fields don't work for alerts in 8.0+.

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[marshallmain]

We're looking at resolving this for 8.4 now. The 2 options we have considered are:

1. Add the data_stream.* fields as keyword to alerts indices. This deviates from the ECS standard slightly and also means that the fields will not conform to the data stream naming scheme in which the data_stream fields are supposed to correspond to the actual name of the ES data stream the document lives in. However, the fields will be queryable in the same ways and with the same queries that customers use on their source data.
2. Add data_stream.* fields in a nested location to avoid any possibility of conflicting with ECS or actual data stream usages. The fields might be something like kibana.data_stream.*. However, user queries will need to account for the different location of this field since we'd be copying data_stream.* fields in source documents to kibana.data_stream.* in alerts.

Re: option 1, @kobelb mentioned that more of ECS may become a first-class concept in the future. If we populate data_stream.* fields in non-data stream indices (the alerts indices) and map them as keyword instead of constant_keyword, could that cause conflicts with future ECS related features, e.g. enforcing the data stream naming scheme? @jpountz @javanna

In addition, the Security alerts indices use large portions of the ECS mappings. The solutions in elastic/ecs#1869 sound like they could reduce the burden (both in code and in the cluster state) of maintaining those ECS mappings by building them in to ES. If we map these fields as keyword now, would that be likely to prevent us from adopting ES built-in ECS mappings?

cc @elastic/response-ops @elastic/observability @MikePaquette @jethr0null

[matschaffer]

Drive-by here, but it seems to me that data_stream for a given data stream (index, mapping, etc) should stay consistent with the data stream that's storing the documents.

What #123420 seems to be highlighting is a need for something like (source).data_stream.* which could deviate from the name of the destination data stream for the document.

Since source is already used in a network context, it might make sense to introduce to introduce another field set (like option 2).

Naming is hard but maybe derivative or from which is a field set expected to contain a copy of an original ECS document from another data stream?

[dgieselaar]

@marshallmain for future reference, you'll probably want to tag @elastic/observability-ui rather than @elastic/observability.

[michaelhyatt]

bumping it, a customer is looking for a fix for it.

[JAndritsch]

bumping it, a customer is looking for a fix for it.

+1

[michaelhyatt]

Any news? Apologies for bumping it again.

[acumen-kevinr]

Is this still being tracked and worked on? We've implemented a workaround with runtime fields but this is still not perfect.

[LaZyDK]

We would love this working also. We have the biggest issues where we have a single cluster SIEM polling multiple other cluster alert indexes over CCS.

[yctercero]

@elastic/response-ops-ram would it be ok to move this ticket over to your team?

[vulnerabivoro]

• 1 to this, we need to manually modify alerts-security* template and reindex every time elastic is updated.