Add support for multi-core thread_id in filebeat postgresql module #9431
Conversation
Remove default version qualifier and rename the environment variable to set it from `BEAT_VERSION_QUALIFIER` to `VERSION_QUALIFIER` this will align with other parts of the stack. **Tested with filebeat.** ``` ❯ ./filebeat version [08:39:01] filebeat version 7.0.0 (amd64), libbeat 7.0.0 [0a0c267 built 2018-11-19 13:38:15 +0000 UTC] ``` **Without the patch** ``` ❯ ./filebeat version [08:40:07] filebeat version 7.0.0-alpha1 (amd64), libbeat 7.0.0-alpha1 [b007837 built 2018-11-19 13:39:59 +0000 UTC] ``` Fixes: #8384
|
|
|
Pinging @elastic/infrastructure |
| @@ -6,8 +6,6 @@ | |||
| "input.type": "log", | |||
| "log.offset": 0, | |||
| "message": "2017-07-31 13:36:42.585 CEST [4974] LOG: database system was shut down at 2017-06-17 16:58:04 CEST", | |||
| "postgresql.log.level": "LOG", | |||
There was a problem hiding this comment.
Why are these values missing now?
|
I have seen that the input log lines can be very tricky so don't forget that you can try with more than one Grok patterns too. |
|
@ruflin yes for the changelog. Now I have this pr number, I can add it into the changelog :-) |
|
@sayden Good point! I just updated |
| @@ -0,0 +1,1092 @@ | |||
| [ | |||
There was a problem hiding this comment.
No need to add this expected file, some longer test files are only to check that the content can be parsed, not to check the generated events.
| "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timezone} \\[(%{NUMBER:postgresql.log.thread_id}|%{NUMBER:postgresql.log.thread_id}-%{BASE16FLOAT:postgresql.log.core_id})\\] \\[%{USERNAME:postgresql.log.user}\\]@\\[%{HOSTNAME:postgresql.log.database}\\] %{WORD:postgresql.log.level}: duration: %{NUMBER:postgresql.log.duration} ms statement: %{MULTILINEQUERY:postgresql.log.query}", | ||
| "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timezone} \\[(%{NUMBER:postgresql.log.thread_id}|%{NUMBER:postgresql.log.thread_id}-%{BASE16FLOAT:postgresql.log.core_id})\\] %{USERNAME:postgresql.log.user}@%{HOSTNAME:postgresql.log.database} %{WORD:postgresql.log.level}: ?%{GREEDYDATA:postgresql.log.message}", | ||
| "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timezone} \\[(%{NUMBER:postgresql.log.thread_id}|%{NUMBER:postgresql.log.thread_id}-%{BASE16FLOAT:postgresql.log.core_id})\\] \\[%{USERNAME:postgresql.log.user}\\]@\\[%{HOSTNAME:postgresql.log.database}\\] %{WORD:postgresql.log.level}: ?%{GREEDYDATA:postgresql.log.message}", | ||
| "%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timezone} \\[(%{NUMBER:postgresql.log.thread_id}|%{NUMBER:postgresql.log.thread_id}-%{BASE16FLOAT:postgresql.log.core_id})\\] %{WORD:postgresql.log.level}: ?%{GREEDYDATA:postgresql.log.message}", |
There was a problem hiding this comment.
I think this is not related to the postgresql module, it is intended to test the fields generator, so these changes are probably not needed. But now that you have added them, should we add the core_id to the list of expected fields below?
| @@ -6,7 +6,7 @@ | |||
| "field": "message", | |||
| "ignore_missing": true, | |||
| "patterns": [ | |||
| "^%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timezone} \\[%{NUMBER:postgresql.log.thread_id}\\] ((\\[%{USERNAME:postgresql.log.user}\\]@\\[%{POSTGRESQL_DB_NAME:postgresql.log.database}\\]|%{USERNAME:postgresql.log.user}@%{POSTGRESQL_DB_NAME:postgresql.log.database}) )?%{WORD:postgresql.log.level}: (duration: %{NUMBER:postgresql.log.duration} ms statement: %{GREEDYDATA:postgresql.log.query}|%{GREEDYDATA:postgresql.log.message})" | |||
| "^%{LOCALDATETIME:postgresql.log.timestamp} %{WORD:postgresql.log.timezone} \\[(%{NUMBER:postgresql.log.thread_id}|%{NUMBER:postgresql.log.thread_id}-%{BASE16FLOAT:postgresql.log.core_id})\\] ((\\[%{USERNAME:postgresql.log.user}\\]@\\[%{POSTGRESQL_DB_NAME:postgresql.log.database}\\]|%{USERNAME:postgresql.log.user}@%{POSTGRESQL_DB_NAME:postgresql.log.database}) )?%{WORD:postgresql.log.level}: (duration: %{NUMBER:postgresql.log.duration} ms statement: %{GREEDYDATA:postgresql.log.query}|%{GREEDYDATA:postgresql.log.message})" | |||
There was a problem hiding this comment.
Could we add the core_id as optional so thread_id appears only one in the expression? i.e: \\[%{NUMBER:postgresql.log.thread_id}(-%{BASE16FLOAT:postgresql.log.core_id})?\\]
There was a problem hiding this comment.
yes! good point! Thanks! I destroyed this pr somehow ... I will create a new PR instead...
|
Thanks for all the comments. I addressed them in #9482 and closing this one... I somehow managed to destroy it again... |
No description provided.