-
Notifications
You must be signed in to change notification settings - Fork 25k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[ML] Data frame analytics - exclude fields from dest index #49531
Labels
Comments
Pinging @elastic/ml-core (:ml) |
dimitris-athanasiou
changed the title
data frame analytics - exclude fields from dest index
[ML] Data frame analytics - exclude fields from dest index
Nov 25, 2019
dimitris-athanasiou
added a commit
to dimitris-athanasiou/elasticsearch
that referenced
this issue
Nov 29, 2019
This adds a `_source` setting under the `source` setting of a data frame analytics config. The new `_source` is reusing the structure of a `FetchSourceContext` like `analyzed_fields` does. Specifying includes and excludes for source allows selecting which fields will get reindexed and will be available in the destination index. Closes elastic#49531
dimitris-athanasiou
added a commit
that referenced
this issue
Nov 29, 2019
This adds a `_source` setting under the `source` setting of a data frame analytics config. The new `_source` is reusing the structure of a `FetchSourceContext` like `analyzed_fields` does. Specifying includes and excludes for source allows selecting which fields will get reindexed and will be available in the destination index. Closes #49531
dimitris-athanasiou
added a commit
to dimitris-athanasiou/elasticsearch
that referenced
this issue
Nov 29, 2019
…lastic#49690) This adds a `_source` setting under the `source` setting of a data frame analytics config. The new `_source` is reusing the structure of a `FetchSourceContext` like `analyzed_fields` does. Specifying includes and excludes for source allows selecting which fields will get reindexed and will be available in the destination index. Closes elastic#49531 Backport of elastic#49690
dimitris-athanasiou
added a commit
that referenced
this issue
Nov 29, 2019
…49690) (#49718) This adds a `_source` setting under the `source` setting of a data frame analytics config. The new `_source` is reusing the structure of a `FetchSourceContext` like `analyzed_fields` does. Specifying includes and excludes for source allows selecting which fields will get reindexed and will be available in the destination index. Closes #49531 Backport of #49690
SivagurunathanV
pushed a commit
to SivagurunathanV/elasticsearch
that referenced
this issue
Jan 23, 2020
…tic#49690) This adds a `_source` setting under the `source` setting of a data frame analytics config. The new `_source` is reusing the structure of a `FetchSourceContext` like `analyzed_fields` does. Specifying includes and excludes for source allows selecting which fields will get reindexed and will be available in the destination index. Closes elastic#49531
This was referenced Feb 3, 2020
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Describe the feature:
Data frame analytics always reindexes all fields of an index. While it is possible to exclude individual documents, it is not possible to exclude a set of fields from being written to the dest index.
For outlier detection only numeric fields are required for analytics. However, all text fields also get copied to the output index.
I suggest adding
_source
filtering as part of the source definition like this.This way the behaviour would be consistent with source filtering in queries or when using _reindex.
This is easy to implement too as we can make use of source filtering in the
_reindex
request that we makeThe text was updated successfully, but these errors were encountered: