Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Kibana Alerts: not able to use date_nanos in ES Query #145855

Closed
psanz-estc opened this issue Nov 21, 2022 · 1 comment · Fixed by #151374
Closed

Kibana Alerts: not able to use date_nanos in ES Query #145855

psanz-estc opened this issue Nov 21, 2022 · 1 comment · Fixed by #151374
Assignees
@ersin-erdal
Labels
bug Fixes for quality problems that affect the customer experience Feature:Alerting/RuleTypes Issues related to specific Alerting Rules Types good first issue low hanging fruit Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)

Comments

@psanz-estc
Copy link

Kibana version: 8.4

Elasticsearch version: 8.4

Server OS version:

Browser version:

Browser OS version:

Original install method (e.g. download page, yum, from source, etc.):

Describe the bug:
When trying to create a Kibana alert using Elasticsearch query, Kibana does not work if timestamps is defined as date_nanos

Steps to reproduce:

  1. Create a simple index with a time field using date_nanos
PUT timestamp-nanos
{
  "mappings": {
    "properties": {
      "timestamp": {
        "type": "date_nanos"
      }
    }
  }
}
  1. Add documents
PUT timestamp-nanos/_doc/3
{
  "timestamp": "2022-11-20T12:10:30.123456789Z"
}

PUT timestamp-nanos/_doc/4
{
  "timestamp": "2022-11-20T12:10:30.123498765Z"
}
  1. Trying to create an alert via ES Query fails to list the time field in the index:
    image

Expected behavior:
Kibana should be able to use date_nanos fields the same way it uses regular date fields

Screenshots (if relevant):

Errors in browser console (if relevant):

Provide logs and/or server output (if relevant):

Any additional context:
@psanz-estc psanz-estc added bug Fixes for quality problems that affect the customer experience Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Feature:Alerting/RuleTypes Issues related to specific Alerting Rules Types labels Nov 21, 2022
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@mikecote mikecote moved this from Awaiting Triage to Todo in AppEx: ResponseOps - Execution & Connectors Dec 8, 2022
@ymao1 ymao1 added the good first issue low hanging fruit label Feb 2, 2023
@ersin-erdal ersin-erdal self-assigned this Feb 15, 2023
@ersin-erdal ersin-erdal mentioned this issue Feb 15, 2023
Merged
@ersin-erdal ersin-erdal moved this from In Progress to In Review in AppEx: ResponseOps - Execution & Connectors Feb 15, 2023
ersin-erdal added a commit that referenced this issue Feb 21, 2023
@ersin-erdal
Add date_nanos support to Time field (#151374)
58204cf 
Fixes: #145855 

`Time field` in rule update/create forms shows only the fields with
`date` data type. This PR adds `date_nanos` type as well.

## To verify:

- Create an index that has fields with `date` and `date_nanos` types.
```
PUT test-index-date
{
  "mappings": {
    "properties": {
      "date": {
        "type": "date"
      },
      "timestamp": {
        "type": "date_nanos"
      }
    }
  }
}
```
- Add data with `date` and `date_nanos` data types
Note: You can use
https://www.site24x7.com/tools/time-stamp-converter.html address to
generate date_nanos timestamps with a date you desired.
```
PUT test-index-date/_bulk?refresh
{ "index" : { "_id" : "1" } }
{ "date": "2023-02-15T18:00:00.000Z", "timestamp": 1676480400000} 
{ "index" : { "_id" : "2" } }
{ "date": "2023-02-15T18:00:00.000Z", "timestamp": 1676480400000 } 
```
- Open a Rule create form and select `Elasticsearch query`
- Select `Query DSL`
- Click on `select an index`
- Type the above index name (test-index-date) in `Indices to query`
input
- Select `timestamp` from time field
- Click on `Test Query` button and expect to see `Query matched 2
documents in the last 5m.`
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Feb 21, 2023
@ersin-erdal
Add date_nanos support to Time field (elastic#151374)
dc16d40 
Fixes: elastic#145855

`Time field` in rule update/create forms shows only the fields with
`date` data type. This PR adds `date_nanos` type as well.

## To verify:

- Create an index that has fields with `date` and `date_nanos` types.
```
PUT test-index-date
{
  "mappings": {
    "properties": {
      "date": {
        "type": "date"
      },
      "timestamp": {
        "type": "date_nanos"
      }
    }
  }
}
```
- Add data with `date` and `date_nanos` data types
Note: You can use
https://www.site24x7.com/tools/time-stamp-converter.html address to
generate date_nanos timestamps with a date you desired.
```
PUT test-index-date/_bulk?refresh
{ "index" : { "_id" : "1" } }
{ "date": "2023-02-15T18:00:00.000Z", "timestamp": 1676480400000}
{ "index" : { "_id" : "2" } }
{ "date": "2023-02-15T18:00:00.000Z", "timestamp": 1676480400000 }
```
- Open a Rule create form and select `Elasticsearch query`
- Select `Query DSL`
- Click on `select an index`
- Type the above index name (test-index-date) in `Indices to query`
input
- Select `timestamp` from time field
- Click on `Test Query` button and expect to see `Query matched 2
documents in the last 5m.`

(cherry picked from commit 58204cf)
kibanamachine added a commit that referenced this issue Feb 21, 2023
@kibanamachine @ersin-erdal
[8.7] Add date_nanos support to Time field (#151374) (#151724)
b4c69c4 
# Backport

This will backport the following commits from `main` to `8.7`:
- [Add date_nanos support to Time field
(#151374)](#151374)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ersin
Erdal","email":"92688503+ersin-erdal@users.noreply.github.com"},"sourceCommit":{"committedDate":"2023-02-21T15:25:09Z","message":"Add
date_nanos support to Time field (#151374)\n\nFixes: #145855
\r\n\r\n`Time field` in rule update/create forms shows only the fields
with\r\n`date` data type. This PR adds `date_nanos` type as
well.\r\n\r\n## To verify:\r\n\r\n- Create an index that has fields with
`date` and `date_nanos` types.\r\n```\r\nPUT test-index-date\r\n{\r\n
\"mappings\": {\r\n \"properties\": {\r\n \"date\": {\r\n \"type\":
\"date\"\r\n },\r\n \"timestamp\": {\r\n \"type\": \"date_nanos\"\r\n
}\r\n }\r\n }\r\n}\r\n```\r\n- Add data with `date` and `date_nanos`
data types\r\nNote: You can
use\r\nhttps://www.site24x7.com/tools/time-stamp-converter.html address
to\r\ngenerate date_nanos timestamps with a date you
desired.\r\n```\r\nPUT test-index-date/_bulk?refresh\r\n{ \"index\" : {
\"_id\" : \"1\" } }\r\n{ \"date\": \"2023-02-15T18:00:00.000Z\",
\"timestamp\": 1676480400000} \r\n{ \"index\" : { \"_id\" : \"2\" }
}\r\n{ \"date\": \"2023-02-15T18:00:00.000Z\", \"timestamp\":
1676480400000 } \r\n```\r\n- Open a Rule create form and select
`Elasticsearch query`\r\n- Select `Query DSL`\r\n- Click on `select an
index`\r\n- Type the above index name (test-index-date) in `Indices to
query`\r\ninput\r\n- Select `timestamp` from time field\r\n- Click on
`Test Query` button and expect to see `Query matched 2\r\ndocuments in
the last
5m.`","sha":"58204cf2a0fd2b921d5f5c3e138b725bce1c5229","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:fix","Team:ResponseOps","v8.7.0","v8.8.0"],"number":151374,"url":"https://github.com/elastic/kibana/pull/151374","mergeCommit":{"message":"Add
date_nanos support to Time field (#151374)\n\nFixes: #145855
\r\n\r\n`Time field` in rule update/create forms shows only the fields
with\r\n`date` data type. This PR adds `date_nanos` type as
well.\r\n\r\n## To verify:\r\n\r\n- Create an index that has fields with
`date` and `date_nanos` types.\r\n```\r\nPUT test-index-date\r\n{\r\n
\"mappings\": {\r\n \"properties\": {\r\n \"date\": {\r\n \"type\":
\"date\"\r\n },\r\n \"timestamp\": {\r\n \"type\": \"date_nanos\"\r\n
}\r\n }\r\n }\r\n}\r\n```\r\n- Add data with `date` and `date_nanos`
data types\r\nNote: You can
use\r\nhttps://www.site24x7.com/tools/time-stamp-converter.html address
to\r\ngenerate date_nanos timestamps with a date you
desired.\r\n```\r\nPUT test-index-date/_bulk?refresh\r\n{ \"index\" : {
\"_id\" : \"1\" } }\r\n{ \"date\": \"2023-02-15T18:00:00.000Z\",
\"timestamp\": 1676480400000} \r\n{ \"index\" : { \"_id\" : \"2\" }
}\r\n{ \"date\": \"2023-02-15T18:00:00.000Z\", \"timestamp\":
1676480400000 } \r\n```\r\n- Open a Rule create form and select
`Elasticsearch query`\r\n- Select `Query DSL`\r\n- Click on `select an
index`\r\n- Type the above index name (test-index-date) in `Indices to
query`\r\ninput\r\n- Select `timestamp` from time field\r\n- Click on
`Test Query` button and expect to see `Query matched 2\r\ndocuments in
the last
5m.`","sha":"58204cf2a0fd2b921d5f5c3e138b725bce1c5229"}},"sourceBranch":"main","suggestedTargetBranches":["8.7"],"targetPullRequestStates":[{"branch":"8.7","label":"v8.7.0","labelRegex":"^v(\\d+).(\\d+).\\d+$","isSourceBranch":false,"state":"NOT_CREATED"},{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/151374","number":151374,"mergeCommit":{"message":"Add
date_nanos support to Time field (#151374)\n\nFixes: #145855
\r\n\r\n`Time field` in rule update/create forms shows only the fields
with\r\n`date` data type. This PR adds `date_nanos` type as
well.\r\n\r\n## To verify:\r\n\r\n- Create an index that has fields with
`date` and `date_nanos` types.\r\n```\r\nPUT test-index-date\r\n{\r\n
\"mappings\": {\r\n \"properties\": {\r\n \"date\": {\r\n \"type\":
\"date\"\r\n },\r\n \"timestamp\": {\r\n \"type\": \"date_nanos\"\r\n
}\r\n }\r\n }\r\n}\r\n```\r\n- Add data with `date` and `date_nanos`
data types\r\nNote: You can
use\r\nhttps://www.site24x7.com/tools/time-stamp-converter.html address
to\r\ngenerate date_nanos timestamps with a date you
desired.\r\n```\r\nPUT test-index-date/_bulk?refresh\r\n{ \"index\" : {
\"_id\" : \"1\" } }\r\n{ \"date\": \"2023-02-15T18:00:00.000Z\",
\"timestamp\": 1676480400000} \r\n{ \"index\" : { \"_id\" : \"2\" }
}\r\n{ \"date\": \"2023-02-15T18:00:00.000Z\", \"timestamp\":
1676480400000 } \r\n```\r\n- Open a Rule create form and select
`Elasticsearch query`\r\n- Select `Query DSL`\r\n- Click on `select an
index`\r\n- Type the above index name (test-index-date) in `Indices to
query`\r\ninput\r\n- Select `timestamp` from time field\r\n- Click on
`Test Query` button and expect to see `Query matched 2\r\ndocuments in
the last 5m.`","sha":"58204cf2a0fd2b921d5f5c3e138b725bce1c5229"}}]}]
BACKPORT-->

Co-authored-by: Ersin Erdal <92688503+ersin-erdal@users.noreply.github.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ersin-erdal ersin-erdal

Labels
bug Fixes for quality problems that affect the customer experience Feature:Alerting/RuleTypes Issues related to specific Alerting Rules Types good first issue low hanging fruit Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams)
Projects
No open projects
AppEx: ResponseOps - Execution & Connectors
Status: Done
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Add date_nanos support to Time field ersin-erdal/kibana
4 participants
@ymao1 @elasticmachine @psanz-estc @ersin-erdal