When courier:ignoreFilterIfFieldNotInIndex enabled, geographic filters no longer working #153595
Assignees
Labels
bug
Fixes for quality problems that affect the customer experience
Feature:Maps
impact:high
Addressing this issue will have a high level of impact on the quality/strength of our product.
Team:Presentation
Presentation Team for Dashboard, Input Controls, and Canvas
Comments
|
Pinging @elastic/kibana-presentation (Team:Presentation) |
|
any advice on this? I would like to have two indexes on a dashboard that are filterable by geographies/shapes, but they have different fields so need to be ignored when one index is filtered by a field that the other does not have. thanks. |
Since 7.14, geospatial filters support multiple fields from multiple indexes. If you are only generating filters from the map, then you should be able to turn off |
|
Yes, that would make it easy if filters were only generated from the map, but filters are generated mostly from controls and tables and queries. ex: data is filtered by type/category via a control and then filtered by geography using the map. With courier:ignoreFilterIfFieldNotInIndex off it works for only one of the indexes. With it on, it works for the controls/table filters but geographic filters do not get applied. |
|
I thought maybe it was because the point data indexes have geo_point fields called 'location' while the polygon shapes have geo_area fields called 'geometry', I thought if changing the point 'location' field to be named 'geometry' so they match but after testing this out, it doesn't seem to make a difference. geography filters do not work when Ignore Filter(s) is enabled. Can this be fixed? |
|
Yes it can be fixed, but it will take a code fix and an upgrade. As a work around, you could disable |
|
Great! Would this be the place to request it be looked into? I'm currently using the workaround but would really need to have both indexes filterable via controls and geometry. An example of the two indexes are of citizen service requests and police crime/accident data, so the need is to mix filter analyze (different types of service requests compared to different types of accidents/crime) to find correlations/trends both in chart form and spatially. |
nreese
added a commit
that referenced
this issue
Mar 31, 2023
…phic filters no longer working (#153816) Fixes #153595 ### Background When advanced setting `courier:ignoreFilterIfFieldNotInIndex` is enabled, filters are ignored when data view does not contain the filtering field. The logic on how this works is captured below for context. https://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/es_query/from_filters.ts#L83 ``` .filter((filter) => { const indexPattern = findIndexPattern(inputDataViews, filter.meta?.index); return !ignoreFilterIfFieldNotInIndex || filterMatchesIndex(filter, indexPattern); }) ``` https://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/es_query/filter_matches_index.ts#L20 ``` export function filterMatchesIndex(filter: Filter, indexPattern?: DataViewBase | null) { if (!filter.meta?.key || !indexPattern) { return true; } // Fixes #89878 // Custom filters may refer multiple fields. Validate the index id only. if (filter.meta?.type === 'custom') { return filter.meta.index === indexPattern.id; } return indexPattern.fields.some((field) => field.name === filter.meta.key); } ``` The problem is that [mapSpatialFilter](https://github.com/elastic/kibana/blob/8.7/src/plugins/data/public/query/filter_manager/lib/mappers/map_spatial_filter.ts#L12) is setting `meta.key` property. This causes `filterMatchesIndex` check to fail for spatial filters. Spatial filters are designed to work across data views and avoid the problems that `courier:ignoreFilterIfFieldNotInIndex` solves. As such, spatial filters should not set `meta.key` property and not get removed when `courier:ignoreFilterIfFieldNotInIndex` is enabled. ### Test * set advanced setting `courier:ignoreFilterIfFieldNotInIndex` and refresh browser * install 2 or more sample data sets * create a new map * add documents layer from one sample data set * add documents layer from another sample data set * create spatial filter on map, https://www.elastic.co/guide/en/kibana/8.6/maps-create-filter-from-map.html#maps-spatial-filters * Verify filter is applied to both layers --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>
kibanamachine
pushed a commit
that referenced
this issue
Mar 31, 2023
…phic filters no longer working (#153816) Fixes #153595 ### Background When advanced setting `courier:ignoreFilterIfFieldNotInIndex` is enabled, filters are ignored when data view does not contain the filtering field. The logic on how this works is captured below for context. https://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/es_query/from_filters.ts#L83 ``` .filter((filter) => { const indexPattern = findIndexPattern(inputDataViews, filter.meta?.index); return !ignoreFilterIfFieldNotInIndex || filterMatchesIndex(filter, indexPattern); }) ``` https://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/es_query/filter_matches_index.ts#L20 ``` export function filterMatchesIndex(filter: Filter, indexPattern?: DataViewBase | null) { if (!filter.meta?.key || !indexPattern) { return true; } // Fixes #89878 // Custom filters may refer multiple fields. Validate the index id only. if (filter.meta?.type === 'custom') { return filter.meta.index === indexPattern.id; } return indexPattern.fields.some((field) => field.name === filter.meta.key); } ``` The problem is that [mapSpatialFilter](https://github.com/elastic/kibana/blob/8.7/src/plugins/data/public/query/filter_manager/lib/mappers/map_spatial_filter.ts#L12) is setting `meta.key` property. This causes `filterMatchesIndex` check to fail for spatial filters. Spatial filters are designed to work across data views and avoid the problems that `courier:ignoreFilterIfFieldNotInIndex` solves. As such, spatial filters should not set `meta.key` property and not get removed when `courier:ignoreFilterIfFieldNotInIndex` is enabled. ### Test * set advanced setting `courier:ignoreFilterIfFieldNotInIndex` and refresh browser * install 2 or more sample data sets * create a new map * add documents layer from one sample data set * add documents layer from another sample data set * create spatial filter on map, https://www.elastic.co/guide/en/kibana/8.6/maps-create-filter-from-map.html#maps-spatial-filters * Verify filter is applied to both layers --------- Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com> (cherry picked from commit 4504cd1)
kibanamachine
referenced
this issue
Mar 31, 2023
…geographic filters no longer working (#153816) (#154162) # Backport This will backport the following commits from `main` to `8.7`: - [[maps] fix When courier:ignoreFilterIfFieldNotInIndex enabled, geographic filters no longer working (#153816)](#153816) <!--- Backport version: 8.9.7 --> ### Questions ? Please refer to the [Backport tool documentation](https://github.com/sqren/backport) <!--BACKPORT [{"author":{"name":"Nathan Reese","email":"reese.nathan@elastic.co"},"sourceCommit":{"committedDate":"2023-03-31T13:59:01Z","message":"[maps] fix When courier:ignoreFilterIfFieldNotInIndex enabled, geographic filters no longer working (#153816)\n\nFixes https://github.com/elastic/kibana/issues/153595\r\n\r\n### Background\r\n\r\nWhen advanced setting `courier:ignoreFilterIfFieldNotInIndex` is\r\nenabled, filters are ignored when data view does not contain the\r\nfiltering field. The logic on how this works is captured below for\r\ncontext.\r\n\r\n\r\nhttps://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/es_query/from_filters.ts#L83\r\n```\r\n .filter((filter) => {\r\n const indexPattern = findIndexPattern(inputDataViews, filter.meta?.index);\r\n return !ignoreFilterIfFieldNotInIndex || filterMatchesIndex(filter, indexPattern);\r\n })\r\n```\r\n\r\n\r\nhttps://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/es_query/filter_matches_index.ts#L20\r\n```\r\nexport function filterMatchesIndex(filter: Filter, indexPattern?: DataViewBase | null) {\r\n if (!filter.meta?.key || !indexPattern) {\r\n return true;\r\n }\r\n\r\n // Fixes https://github.com/elastic/kibana/issues/89878\r\n // Custom filters may refer multiple fields. Validate the index id only.\r\n if (filter.meta?.type === 'custom') {\r\n return filter.meta.index === indexPattern.id;\r\n }\r\n\r\n return indexPattern.fields.some((field) => field.name === filter.meta.key);\r\n}\r\n```\r\n\r\nThe problem is that\r\n[mapSpatialFilter](https://github.com/elastic/kibana/blob/8.7/src/plugins/data/public/query/filter_manager/lib/mappers/map_spatial_filter.ts#L12)\r\nis setting `meta.key` property. This causes `filterMatchesIndex` check\r\nto fail for spatial filters.\r\n\r\nSpatial filters are designed to work across data views and avoid the\r\nproblems that `courier:ignoreFilterIfFieldNotInIndex` solves. As such,\r\nspatial filters should not set `meta.key` property and not get removed\r\nwhen `courier:ignoreFilterIfFieldNotInIndex` is enabled.\r\n\r\n### Test\r\n* set advanced setting `courier:ignoreFilterIfFieldNotInIndex` and\r\nrefresh browser\r\n* install 2 or more sample data sets\r\n* create a new map\r\n* add documents layer from one sample data set\r\n* add documents layer from another sample data set\r\n* create spatial filter on map,\r\nhttps://www.elastic.co/guide/en/kibana/8.6/maps-create-filter-from-map.html#maps-spatial-filters\r\n* Verify filter is applied to both layers\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"4504cd18c07c0f0af94830558f7906dd77a2b217","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:Presentation","auto-backport","Feature:Maps","v8.8.0","v8.7.1"],"number":153816,"url":"https://github.com/elastic/kibana/pull/153816","mergeCommit":{"message":"[maps] fix When courier:ignoreFilterIfFieldNotInIndex enabled, geographic filters no longer working (#153816)\n\nFixes https://github.com/elastic/kibana/issues/153595\r\n\r\n### Background\r\n\r\nWhen advanced setting `courier:ignoreFilterIfFieldNotInIndex` is\r\nenabled, filters are ignored when data view does not contain the\r\nfiltering field. The logic on how this works is captured below for\r\ncontext.\r\n\r\n\r\nhttps://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/es_query/from_filters.ts#L83\r\n```\r\n .filter((filter) => {\r\n const indexPattern = findIndexPattern(inputDataViews, filter.meta?.index);\r\n return !ignoreFilterIfFieldNotInIndex || filterMatchesIndex(filter, indexPattern);\r\n })\r\n```\r\n\r\n\r\nhttps://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/es_query/filter_matches_index.ts#L20\r\n```\r\nexport function filterMatchesIndex(filter: Filter, indexPattern?: DataViewBase | null) {\r\n if (!filter.meta?.key || !indexPattern) {\r\n return true;\r\n }\r\n\r\n // Fixes https://github.com/elastic/kibana/issues/89878\r\n // Custom filters may refer multiple fields. Validate the index id only.\r\n if (filter.meta?.type === 'custom') {\r\n return filter.meta.index === indexPattern.id;\r\n }\r\n\r\n return indexPattern.fields.some((field) => field.name === filter.meta.key);\r\n}\r\n```\r\n\r\nThe problem is that\r\n[mapSpatialFilter](https://github.com/elastic/kibana/blob/8.7/src/plugins/data/public/query/filter_manager/lib/mappers/map_spatial_filter.ts#L12)\r\nis setting `meta.key` property. This causes `filterMatchesIndex` check\r\nto fail for spatial filters.\r\n\r\nSpatial filters are designed to work across data views and avoid the\r\nproblems that `courier:ignoreFilterIfFieldNotInIndex` solves. As such,\r\nspatial filters should not set `meta.key` property and not get removed\r\nwhen `courier:ignoreFilterIfFieldNotInIndex` is enabled.\r\n\r\n### Test\r\n* set advanced setting `courier:ignoreFilterIfFieldNotInIndex` and\r\nrefresh browser\r\n* install 2 or more sample data sets\r\n* create a new map\r\n* add documents layer from one sample data set\r\n* add documents layer from another sample data set\r\n* create spatial filter on map,\r\nhttps://www.elastic.co/guide/en/kibana/8.6/maps-create-filter-from-map.html#maps-spatial-filters\r\n* Verify filter is applied to both layers\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"4504cd18c07c0f0af94830558f7906dd77a2b217"}},"sourceBranch":"main","suggestedTargetBranches":["8.7"],"targetPullRequestStates":[{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/153816","number":153816,"mergeCommit":{"message":"[maps] fix When courier:ignoreFilterIfFieldNotInIndex enabled, geographic filters no longer working (#153816)\n\nFixes https://github.com/elastic/kibana/issues/153595\r\n\r\n### Background\r\n\r\nWhen advanced setting `courier:ignoreFilterIfFieldNotInIndex` is\r\nenabled, filters are ignored when data view does not contain the\r\nfiltering field. The logic on how this works is captured below for\r\ncontext.\r\n\r\n\r\nhttps://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/es_query/from_filters.ts#L83\r\n```\r\n .filter((filter) => {\r\n const indexPattern = findIndexPattern(inputDataViews, filter.meta?.index);\r\n return !ignoreFilterIfFieldNotInIndex || filterMatchesIndex(filter, indexPattern);\r\n })\r\n```\r\n\r\n\r\nhttps://github.com/elastic/kibana/blob/main/packages/kbn-es-query/src/es_query/filter_matches_index.ts#L20\r\n```\r\nexport function filterMatchesIndex(filter: Filter, indexPattern?: DataViewBase | null) {\r\n if (!filter.meta?.key || !indexPattern) {\r\n return true;\r\n }\r\n\r\n // Fixes https://github.com/elastic/kibana/issues/89878\r\n // Custom filters may refer multiple fields. Validate the index id only.\r\n if (filter.meta?.type === 'custom') {\r\n return filter.meta.index === indexPattern.id;\r\n }\r\n\r\n return indexPattern.fields.some((field) => field.name === filter.meta.key);\r\n}\r\n```\r\n\r\nThe problem is that\r\n[mapSpatialFilter](https://github.com/elastic/kibana/blob/8.7/src/plugins/data/public/query/filter_manager/lib/mappers/map_spatial_filter.ts#L12)\r\nis setting `meta.key` property. This causes `filterMatchesIndex` check\r\nto fail for spatial filters.\r\n\r\nSpatial filters are designed to work across data views and avoid the\r\nproblems that `courier:ignoreFilterIfFieldNotInIndex` solves. As such,\r\nspatial filters should not set `meta.key` property and not get removed\r\nwhen `courier:ignoreFilterIfFieldNotInIndex` is enabled.\r\n\r\n### Test\r\n* set advanced setting `courier:ignoreFilterIfFieldNotInIndex` and\r\nrefresh browser\r\n* install 2 or more sample data sets\r\n* create a new map\r\n* add documents layer from one sample data set\r\n* add documents layer from another sample data set\r\n* create spatial filter on map,\r\nhttps://www.elastic.co/guide/en/kibana/8.6/maps-create-filter-from-map.html#maps-spatial-filters\r\n* Verify filter is applied to both layers\r\n\r\n---------\r\n\r\nCo-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>","sha":"4504cd18c07c0f0af94830558f7906dd77a2b217"}},{"branch":"8.7","label":"v8.7.1","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"}]}] BACKPORT--> Co-authored-by: Nathan Reese <reese.nathan@elastic.co>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Kibana version: 8.6.2
Elasticsearch version: 8.6.2
Server OS version: Windows 19
Browser version: Chrome, Edge
Browser OS version:
Original install method (e.g. download page, yum, from source, etc.):
download zip windows 64bit
Describe the bug:
With 2 indexes displayed/filterable on a dashboard/map, ingnoreFilterFieldNotInIndex is enabled so to filter on the different indexes. This performs well with the regular controls/filters, however, it seems to take away the ability to use geographic filters: Draw Shape to filter, etc.., on the map.
Steps to reproduce:
Expected behavior:
All point locations from any of the visible indexes should be filtered on the map and other dashboard charts/visualizations
Screenshots (if relevant):
Errors in browser console (if relevant):
Provide logs and/or server output (if relevant):
Any additional context:
The text was updated successfully, but these errors were encountered: