Skip to content

Commit

Permalink
[Kafka] update kafka ECS version and adding event.original option (el…
Browse files Browse the repository at this point in the history
…astic#1116)

* update kafka ECS version and adding event.original option

* update changelog

* run elastic-package format

* add parsing kafka log thread

* run elastic-package test pipeline

Co-authored-by: kaiyan-sheng <kaiyan.sheng@elastic.co>
  • Loading branch information
P1llus and kaiyan-sheng authored Jun 10, 2021
1 parent d1f5a71 commit 3fa4791
Show file tree
Hide file tree
Showing 25 changed files with 6,324 additions and 70 deletions.
5 changes: 5 additions & 0 deletions packages/kafka/changelog.yml
Original file line number Diff line number Diff line change
@@ -1,4 +1,9 @@
# newer versions go on top
- version: "0.4.0"
changes:
- description: Update to ECS 1.10.0 and adding event.original option
type: enhancement
link: https://github.com/elastic/integrations/pull/1116
- version: "0.3.8"
changes:
- description: Updating package owner
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
dynamic_fields:
event.ingested: ".*"
multiline:
first_line_pattern: '^\['
negate: true
match: after
fields:
"@timestamp": "2020-04-28T11:07:58.223Z"
tags:
- preserve_original_event
Original file line number Diff line number Diff line change
@@ -0,0 +1,23 @@
[2018-10-31 15:03:32,474] TRACE [Controller id=10] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
[2018-10-31 15:03:32,474] DEBUG [Controller id=10] Preferred replicas by broker Map(20 -> Map(__consumer_offsets-22 -> Vector(20), __consumer_offsets-4 -> Vector(20), __consumer_offsets-7 -> Vector(20), __consumer_offsets-46 -> Vector(20), __consumer_offsets-25 -> Vector(20), __consumer_offsets-49 -> Vector(20), __consumer_offsets-16 -> Vector(20), test-0 -> Vector(20, 30, 10), __consumer_offsets-28 -> Vector(20), __consumer_offsets-31 -> Vector(20), test-2-2 -> Vector(20, 30), __consumer_offsets-37 -> Vector(20), filebeat-system-0 -> Vector(20), test-3-3 -> Vector(20, 30), __consumer_offsets-19 -> Vector(20), __consumer_offsets-13 -> Vector(20), __consumer_offsets-43 -> Vector(20), __consumer_offsets-1 -> Vector(20), __consumer_offsets-34 -> Vector(20), __consumer_offsets-10 -> Vector(20), test-3 -> Vector(20, 10, 30), __consumer_offsets-40 -> Vector(20)), 10 -> Map(__consumer_offsets-30 -> Vector(10), __consumer_offsets-21 -> Vector(10), __consumer_offsets-27 -> Vector(10), __consumer_offsets-9 -> Vector(10), __consumer_offsets-33 -> Vector(10), __consumer_offsets-36 -> Vector(10), __consumer_offsets-42 -> Vector(10), __consumer_offsets-3 -> Vector(10), __consumer_offsets-18 -> Vector(10), test-5 -> Vector(10, 30, 20), __consumer_offsets-15 -> Vector(10), __consumer_offsets-24 -> Vector(10), test-3-1 -> Vector(10, 20), __consumer_offsets-48 -> Vector(10), filebeat-kafka-0 -> Vector(10), __consumer_offsets-6 -> Vector(10), test-2-1 -> Vector(10, 20), test-3-2 -> Vector(10, 30), __consumer_offsets-0 -> Vector(10), __consumer_offsets-39 -> Vector(10), __consumer_offsets-12 -> Vector(10), __consumer_offsets-45 -> Vector(10), test-2 -> Vector(10, 20, 30)), 30 -> Map(__consumer_offsets-8 -> Vector(30), __consumer_offsets-35 -> Vector(30), __consumer_offsets-41 -> Vector(30), __consumer_offsets-23 -> Vector(30), __consumer_offsets-47 -> Vector(30), metricbeat-0 -> Vector(30), test-3-0 -> Vector(30, 10), metricbeat-kafka-0 -> Vector(30), filebeat-0 -> Vector(30), test-2-0 -> Vector(30, 10), __consumer_offsets-38 -> Vector(30), __consumer_offsets-17 -> Vector(30), test-1 -> Vector(30, 10, 20), test-2-3 -> Vector(30, 20), __consumer_offsets-11 -> Vector(30), __consumer_offsets-2 -> Vector(30), __consumer_offsets-14 -> Vector(30), test-4 -> Vector(30, 20, 10), metricbeat-system-0 -> Vector(30), __consumer_offsets-20 -> Vector(30), __consumer_offsets-44 -> Vector(30), __consumer_offsets-5 -> Vector(30), __consumer_offsets-26 -> Vector(30), __consumer_offsets-29 -> Vector(30), __consumer_offsets-32 -> Vector(30))) (kafka.controller.KafkaController)
[2018-10-31 15:03:32,474] DEBUG [Controller id=10] Topics not in preferred replica for broker 20 Map() (kafka.controller.KafkaController)
[2018-10-31 15:03:32,475] TRACE [Controller id=10] Leader imbalance ratio for broker 20 is 0.0 (kafka.controller.KafkaController)
[2018-10-31 15:03:32,475] DEBUG [Controller id=10] Topics not in preferred replica for broker 10 Map(test-3-1 -> Vector(10, 20)) (kafka.controller.KafkaController)
[2018-10-31 15:03:32,475] TRACE [Controller id=10] Leader imbalance ratio for broker 10 is 0.043478260869565216 (kafka.controller.KafkaController)
[2018-10-31 15:03:32,475] DEBUG [Controller id=10] Topics not in preferred replica for broker 30 Map() (kafka.controller.KafkaController)
[2018-10-31 15:03:32,475] TRACE [Controller id=10] Leader imbalance ratio for broker 30 is 0.0 (kafka.controller.KafkaController)
[2018-10-31 15:08:32,475] TRACE [Controller id=10] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)
[2018-10-31 15:08:32,475] DEBUG [Controller id=10] Preferred replicas by broker Map(20 -> Map(__consumer_offsets-22 -> Vector(20), __consumer_offsets-4 -> Vector(20), __consumer_offsets-7 -> Vector(20), __consumer_offsets-46 -> Vector(20), __consumer_offsets-25 -> Vector(20), __consumer_offsets-49 -> Vector(20), __consumer_offsets-16 -> Vector(20), test-0 -> Vector(20, 30, 10), __consumer_offsets-28 -> Vector(20), __consumer_offsets-31 -> Vector(20), test-2-2 -> Vector(20, 30), __consumer_offsets-37 -> Vector(20), filebeat-system-0 -> Vector(20), test-3-3 -> Vector(20, 30), __consumer_offsets-19 -> Vector(20), __consumer_offsets-13 -> Vector(20), __consumer_offsets-43 -> Vector(20), __consumer_offsets-1 -> Vector(20), __consumer_offsets-34 -> Vector(20), __consumer_offsets-10 -> Vector(20), test-3 -> Vector(20, 10, 30), __consumer_offsets-40 -> Vector(20)), 10 -> Map(__consumer_offsets-30 -> Vector(10), __consumer_offsets-21 -> Vector(10), __consumer_offsets-27 -> Vector(10), __consumer_offsets-9 -> Vector(10), __consumer_offsets-33 -> Vector(10), __consumer_offsets-36 -> Vector(10), __consumer_offsets-42 -> Vector(10), __consumer_offsets-3 -> Vector(10), __consumer_offsets-18 -> Vector(10), test-5 -> Vector(10, 30, 20), __consumer_offsets-15 -> Vector(10), __consumer_offsets-24 -> Vector(10), test-3-1 -> Vector(10, 20), __consumer_offsets-48 -> Vector(10), filebeat-kafka-0 -> Vector(10), __consumer_offsets-6 -> Vector(10), test-2-1 -> Vector(10, 20), test-3-2 -> Vector(10, 30), __consumer_offsets-0 -> Vector(10), __consumer_offsets-39 -> Vector(10), __consumer_offsets-12 -> Vector(10), __consumer_offsets-45 -> Vector(10), test-2 -> Vector(10, 20, 30)), 30 -> Map(__consumer_offsets-8 -> Vector(30), __consumer_offsets-35 -> Vector(30), __consumer_offsets-41 -> Vector(30), __consumer_offsets-23 -> Vector(30), __consumer_offsets-47 -> Vector(30), metricbeat-0 -> Vector(30), test-3-0 -> Vector(30, 10), metricbeat-kafka-0 -> Vector(30), filebeat-0 -> Vector(30), test-2-0 -> Vector(30, 10), __consumer_offsets-38 -> Vector(30), __consumer_offsets-17 -> Vector(30), test-1 -> Vector(30, 10, 20), test-2-3 -> Vector(30, 20), __consumer_offsets-11 -> Vector(30), __consumer_offsets-2 -> Vector(30), __consumer_offsets-14 -> Vector(30), test-4 -> Vector(30, 20, 10), metricbeat-system-0 -> Vector(30), __consumer_offsets-20 -> Vector(30), __consumer_offsets-44 -> Vector(30), __consumer_offsets-5 -> Vector(30), __consumer_offsets-26 -> Vector(30), __consumer_offsets-29 -> Vector(30), __consumer_offsets-32 -> Vector(30))) (kafka.controller.KafkaController)
[2018-10-31 15:08:32,475] DEBUG [Controller id=10] Topics not in preferred replica for broker 20 Map() (kafka.controller.KafkaController)
[2018-10-31 15:08:32,475] TRACE [Controller id=10] Leader imbalance ratio for broker 20 is 0.0 (kafka.controller.KafkaController)
[2018-10-31 15:08:32,475] DEBUG [Controller id=10] Topics not in preferred replica for broker 10 Map(test-3-1 -> Vector(10, 20)) (kafka.controller.KafkaController)
[2018-10-31 15:08:32,475] TRACE [Controller id=10] Leader imbalance ratio for broker 10 is 0.043478260869565216 (kafka.controller.KafkaController)
[2018-10-31 15:08:32,475] DEBUG [Controller id=10] Topics not in preferred replica for broker 30 Map() (kafka.controller.KafkaController)
[2018-10-31 15:08:32,475] TRACE [Controller id=10] Leader imbalance ratio for broker 30 is 0.0 (kafka.controller.KafkaController)
[2018-10-31 15:09:30,306] INFO [Controller id=10] New topics: [Set(foo)], deleted topics: [Set()], new partition replica assignment [Map(foo-0 -> Vector(20))] (kafka.controller.KafkaController)
[2018-10-31 15:09:30,307] INFO [Controller id=10] New partition creation callback for foo-0 (kafka.controller.KafkaController)
[2018-10-31 15:09:30,396] INFO [RequestSendThread controllerId=10] Controller 10 connected to 10.122.220.20:9094 (id: 20 rack: null) for sending state change requests (kafka.controller.RequestSendThread)
[2018-10-31 15:09:30,397] INFO [RequestSendThread controllerId=10] Controller 10 connected to 10.122.220.20:9093 (id: 10 rack: null) for sending state change requests (kafka.controller.RequestSendThread)
[2018-10-31 15:09:30,396] INFO [RequestSendThread controllerId=10] Controller 10 connected to 10.122.220.20:9095 (id: 30 rack: null) for sending state change requests (kafka.controller.RequestSendThread)
[2018-10-31 15:13:32,475] TRACE [Controller id=10] Checking need to trigger auto leader balancing (kafka.controller.KafkaController)

Loading

0 comments on commit 3fa4791

Please sign in to comment.