Count number of documents with at least one ignored field

[salvatore-campagna]

When it comes to counting documents including at least one ignored field, aggregations might
result in poor performance because of having to aggregate too many documents over multiple
indices targeted either though index patterns or data streams.

As part of the effort behind making ingestion of logs more reliable and improve on explaining indexing
issues, we would like to provide users with the fraction of documents including at least one
ignored field. For this purpose we introduce a new index metric, ignored_field, to the index stats api which allows fetching statistics about ignored fields.

Ignored field stats includes the following:

• total_docs: total number of documents in the index
• docs_with_ignored_fields: total number of documents with at least one ignored field
• sum_doc_freq_terms_ignored_fields: the sum of term frequencies for the _ignored field

This solution has some drawbacks anyway:

1. it ignores deleted documents (it would need to filter out and subtract deleted documents which would make stats latency worse)
2. it ignores documents which are still in the IndexWriter buffer waiting to be flushed to Lucene segments

Resolves #108092

[elasticsearchmachine]

Pinging @elastic/es-storage-engine (Team:StorageEngine)

[elasticsearchmachine]

Hi @salvatore-campagna, I've created a changelog YAML for you.

[elasticsearchmachine]

Hi @salvatore-campagna, I've updated the changelog YAML for you.

[salvatore-campagna]

This happens for source only indices which do not include inverted index and doc values.

[martijnvg]

Should this invoke getDocCount() instead? Which returns: Returns the number of documents that have at least one term for this field, which matches more closely with the method name and what we are trying to include in the doc stats?

[salvatore-campagna]

Will put this back.

[salvatore-campagna]

@martijnvg this is the logic I was talking about. I had to add a new source to avoid FrozenStorageDeciderIT#testScale fail. If I use "doc_stats" as source if fails at the assertion

assert false : "doc stats are eagerly loaded";

in FrozenEngine#openSearcher

[salvatore-campagna]

@martijnvg I added this entry which, if I understand correctly results in not opening a new searcher but just reusing one that is already open.

[salvatore-campagna]

Full BWC started to fail after I merged main I think something has been merged introducing some serialization issue. Probably another PR was merged without testing for full BWC.

[salvatore-campagna]

I don't see a way to filter stats based on tier other than using the tier_preference.

Filtering on it anyway returning ignored field stats only in case the tier preference is data_hot, data_warm or data_content means that:

• we would be able to get such stats only for data streams
• we would be able to get such stats only for indices created by version 8 and above since the tier preference is required and a default value is injected only for indices created by version 8 and above

[salvatore-campagna]

Another option would be to just return an empty result when the request hits a node whose role is data_frozen or data_cold. This way in a cluster hitting multiple nodes with different (data) roles we would return nothing from nodes with slow storage and the actual stats from other nodes like data_hot, data_warm and data_content.

[salvatore-campagna]

Summarising what we discussed in a meeting about how to proceed.

We will implement this feature keeping the ignored_field stats request and adding a filter that, by default, filters out results when fetching stats whose fetching might be slow (data_frozen, data_cold, and snapshots). We will give users an option to eventually ask for ignored field stats for such (slow) cases too, adding an additional option...something like allow_slow_queries=true.

[salvatore-campagna]

With this PR: #101373 we introduced doc values for the _ignored field. That should make the aggregation on _ignored field faster (for new indices).

[felixbarny]

In addition to enabling aggregations, it also makes it faster to count documents that have a _ignored field with the exists query. Since the dataset quality page has a time filter, it makes more sense to use an exists query in combination with a range query on the timestamp rather than getting stats from an entire index.

[flash1293]

The reason we started looking into this approach is that we can't do read queries to collect telemetry with the current permission setup in Kibana. I realize that this is not a strong point in favor of doing it, back then our reasoning was that when it helps this use case and is a helpful feature in general, we should do it.