Add Azure Event Hub and Log Analytics Operators #287
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
|
… type, because we cannot predict which log fields will be present.
|
… prevent Elasticsearch record.data conflicts
|
|
djaglowski
previously approved these changes
May 4, 2021
…oved test cases for parse() as they are redundant with TestParseEvent() and not part of the public interface. Updated test cases for loganalytics to reflect the new resource.
… used once, and we check to see if it is not true.
…a's persistence package handles locking on its own.
…, in order to reuse the code between both operators. Input configuration is unchanged due to yaml:",inline"
|
… the configuration of both azure operators
|
|
Codecov Report
@@ Coverage Diff @@
## master #287 +/- ##
==========================================
- Coverage 71.72% 70.62% -1.10%
==========================================
Files 104 112 +8
Lines 5725 5956 +231
==========================================
+ Hits 4106 4206 +100
- Misses 1181 1304 +123
- Partials 438 446 +8
Continue to review full report at Codecov.
|
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Description of Changes
This PR replaces #280. and #285.
operator/builtin/input/azureoperator/builtin/input/azure/eventhuboperator/builtin/input/azure/loganalyticsBoth operators rely heavily upon methods provided by the azure packages
EventHubtype, for connecting to Event Hub and streaming events.Azure Event Hub Input Operator
This operator acts as a thin wrapper around
azure.EventHub. It relies solely onazure.ParseEventfor event parsing. It makes no assumptions on the structure of the data, or which fields to expect.record.datawill contain the event's message.{ "timestamp": "2021-04-27T18:34:15.524Z", "severity": 0, "record": { "data": "hello, world!", "id": "195766f2-662f-4dda-95fc-9d3a26450fbb", "system_properties": { "x-opt-enqueued-time": "2021-04-27T18:34:15.524Z", "x-opt-offset": 12917788488, "x-opt-sequence-number": 13997 } } }Azure Log Analytics Input Operator
Azure Log Analytics can forward logs to Azure Event Hub. The event's
datafield contains batch log entries as serialized json. The raw message looks like this when coming though the Azure Event Hub Input Operator:{ "timestamp": "2021-04-26T18:09:02.531Z", "severity": 0, "record": { "data": "{\"records\": [{ \"Computer\": \"aks-agentpool-39365618-vmss000001\", \"ContainerID\": \"93f4537223ae81d1c39e12e684de25c65207549d1003d153356055a6137f82b0\", \"LogEntry\": \"[SpanData(name='Recv.grpc.health.v1.Health.Check', context=SpanContext(trace_id=5422791cd0f04c79823f21da4ed6a225, span_id=None, trace_options=TraceOptions(enabled=True), tracestate=None), span_id='f0225892b66f4bec', parent_span_id=None, attributes={'component': 'grpc'}, start_time='2021-04-26T18:08:46.110348Z', end_time='2021-04-26T18:08:46.110427Z', child_span_count=0, stack_trace=None, time_events=[<opencensus.trace.time_event.TimeEvent object at 0x7f5d9fe46550>, <opencensus.trace.time_event.TimeEvent object at 0x7f5d9fc53e50>], links=[], status=None, same_process_as_parent_span=None, span_kind=1)]\", \"LogEntrySource\": \"stdout\", \"MG\": \"00000000-0000-0000-0000-000000000002\", \"SourceSystem\": \"Containers\", \"TenantId\": \"ae0db88b-40bb-40b7-b056-57980214436c\", \"TimeGenerated\": \"2021-04-26T18:08:46.1100000Z\", \"TimeOfCommand\": \"2021-04-26T18:08:59.0000000Z\", \"Type\": \"ContainerLog\", \"_Internal_WorkspaceResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourcegroups/bindplane-integration/providers/microsoft.operationalinsights/workspaces/bp-integration1\", \"_ResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourceGroups/devops/providers/Microsoft.ContainerService/managedClusters/log-analytics\"},{ \"Computer\": \"aks-agentpool-39365618-vmss000001\", \"ContainerID\": \"93f4537223ae81d1c39e12e684de25c65207549d1003d153356055a6137f82b0\", \"LogEntry\": \"[SpanData(name='Recv.grpc.health.v1.Health.Check', context=SpanContext(trace_id=85ffc46e7e2a4425ae911aecdca81908, span_id=None, trace_options=TraceOptions(enabled=True), tracestate=None), span_id='24c11702523f4ed6', parent_span_id=None, attributes={'component': 'grpc'}, start_time='2021-04-26T18:08:46.358194Z', end_time='2021-04-26T18:08:46.358268Z', child_span_count=0, stack_trace=None, time_events=[<opencensus.trace.time_event.TimeEvent object at 0x7f5d9fc47c50>, <opencensus.trace.time_event.TimeEvent object at 0x7f5d9fc50790>], links=[], status=None, same_process_as_parent_span=None, span_kind=1)]\", \"LogEntrySource\": \"stdout\", \"MG\": \"00000000-0000-0000-0000-000000000002\", \"SourceSystem\": \"Containers\", \"TenantId\": \"ae0db88b-40bb-40b7-b056-57980214436c\", \"TimeGenerated\": \"2021-04-26T18:08:46.3580000Z\", \"TimeOfCommand\": \"2021-04-26T18:08:59.0000000Z\", \"Type\": \"ContainerLog\", \"_Internal_WorkspaceResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourcegroups/bindplane-integration/providers/microsoft.operationalinsights/workspaces/bp-integration1\", \"_ResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourceGroups/devops/providers/Microsoft.ContainerService/managedClusters/log-analytics\"},{ \"Computer\": \"aks-agentpool-39365618-vmss000001\", \"ContainerID\": \"93f4537223ae81d1c39e12e684de25c65207549d1003d153356055a6137f82b0\", \"LogEntry\": \"[SpanData(name='Recv.grpc.health.v1.Health.Check', context=SpanContext(trace_id=d90501de1d574a39b7c61bd5c82a22fb, span_id=None, trace_options=TraceOptions(enabled=True), tracestate=None), span_id='2bba12b0bf9d4dc0', parent_span_id=None, attributes={'component': 'grpc'}, start_time='2021-04-26T18:08:51.128478Z', end_time='2021-04-26T18:08:51.128578Z', child_span_count=0, stack_trace=None, time_events=[<opencensus.trace.time_event.TimeEvent object at 0x7f5d9fc50050>, <opencensus.trace.time_event.TimeEvent object at 0x7f5d9fc503d0>], links=[], status=None, same_process_as_parent_span=None, span_kind=1)]\", \"LogEntrySource\": \"stdout\", \"MG\": \"00000000-0000-0000-0000-000000000002\", \"SourceSystem\": \"Containers\", \"TenantId\": \"ae0db88b-40bb-40b7-b056-57980214436c\", \"TimeGenerated\": \"2021-04-26T18:08:51.1280000Z\", \"TimeOfCommand\": \"2021-04-26T18:08:59.0000000Z\", \"Type\": \"ContainerLog\", \"_Internal_WorkspaceResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourcegroups/bindplane-integration/providers/microsoft.operationalinsights/workspaces/bp-integration1\", \"_ResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourceGroups/devops/providers/Microsoft.ContainerService/managedClusters/log-analytics\"},{ \"Computer\": \"aks-agentpool-39365618-vmss000001\", \"ContainerID\": \"93f4537223ae81d1c39e12e684de25c65207549d1003d153356055a6137f82b0\", \"LogEntry\": \"[SpanData(name='Recv.grpc.health.v1.Health.Check', context=SpanContext(trace_id=d0d1bb4aa26241e389fb8932d522595b, span_id=None, trace_options=TraceOptions(enabled=True), tracestate=None), span_id='e51b79b362574ab2', parent_span_id=None, attributes={'component': 'grpc'}, start_time='2021-04-26T18:08:51.360458Z', end_time='2021-04-26T18:08:51.360542Z', child_span_count=0, stack_trace=None, time_events=[<opencensus.trace.time_event.TimeEvent object at 0x7f5d9fe46690>, <opencensus.trace.time_event.TimeEvent object at 0x7f5d9fc49dd0>], links=[], status=None, same_process_as_parent_span=None, span_kind=1)]\", \"LogEntrySource\": \"stdout\", \"MG\": \"00000000-0000-0000-0000-000000000002\", \"SourceSystem\": \"Containers\", \"TenantId\": \"ae0db88b-40bb-40b7-b056-57980214436c\", \"TimeGenerated\": \"2021-04-26T18:08:51.3600000Z\", \"TimeOfCommand\": \"2021-04-26T18:08:59.0000000Z\", \"Type\": \"ContainerLog\", \"_Internal_WorkspaceResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourcegroups/bindplane-integration/providers/microsoft.operationalinsights/workspaces/bp-integration1\", \"_ResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourceGroups/devops/providers/Microsoft.ContainerService/managedClusters/log-analytics\"},{ \"Computer\": \"aks-agentpool-39365618-vmss000001\", \"ContainerID\": \"93f4537223ae81d1c39e12e684de25c65207549d1003d153356055a6137f82b0\", \"LogEntry\": \"[SpanData(name='Recv.grpc.health.v1.Health.Check', context=SpanContext(trace_id=83d055d4436a4ddd976574008e77d009, span_id=None, trace_options=TraceOptions(enabled=True), tracestate=None), span_id='0657173a31394c89', parent_span_id=None, attributes={'component': 'grpc'}, start_time='2021-04-26T18:08:56.111434Z', end_time='2021-04-26T18:08:56.111513Z', child_span_count=0, stack_trace=None, time_events=[<opencensus.trace.time_event.TimeEvent object at 0x7f5d9fc535d0>, <opencensus.trace.time_event.TimeEvent object at 0x7f5d9fe43e90>], links=[], status=None, same_process_as_parent_span=None, span_kind=1)]\", \"LogEntrySource\": \"stdout\", \"MG\": \"00000000-0000-0000-0000-000000000002\", \"SourceSystem\": \"Containers\", \"TenantId\": \"ae0db88b-40bb-40b7-b056-57980214436c\", \"TimeGenerated\": \"2021-04-26T18:08:56.1110000Z\", \"TimeOfCommand\": \"2021-04-26T18:08:59.0000000Z\", \"Type\": \"ContainerLog\", \"_Internal_WorkspaceResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourcegroups/bindplane-integration/providers/microsoft.operationalinsights/workspaces/bp-integration1\", \"_ResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourceGroups/devops/providers/Microsoft.ContainerService/managedClusters/log-analytics\"},{ \"Computer\": \"aks-agentpool-39365618-vmss000001\", \"ContainerID\": \"93f4537223ae81d1c39e12e684de25c65207549d1003d153356055a6137f82b0\", \"LogEntry\": \"[SpanData(name='Recv.grpc.health.v1.Health.Check', context=SpanContext(trace_id=03d699a233924d1892e915af9f564e03, span_id=None, trace_options=TraceOptions(enabled=True), tracestate=None), span_id='ca56064504344297', parent_span_id=None, attributes={'component': 'grpc'}, start_time='2021-04-26T18:08:56.358351Z', end_time='2021-04-26T18:08:56.358447Z', child_span_count=0, stack_trace=None, time_events=[<opencensus.trace.time_event.TimeEvent object at 0x7f5d9fc53710>, <opencensus.trace.time_event.TimeEvent object at 0x7f5d9fc4d850>], links=[], status=None, same_process_as_parent_span=None, span_kind=1)]\", \"LogEntrySource\": \"stdout\", \"MG\": \"00000000-0000-0000-0000-000000000002\", \"SourceSystem\": \"Containers\", \"TenantId\": \"ae0db88b-40bb-40b7-b056-57980214436c\", \"TimeGenerated\": \"2021-04-26T18:08:56.3580000Z\", \"TimeOfCommand\": \"2021-04-26T18:08:59.0000000Z\", \"Type\": \"ContainerLog\", \"_Internal_WorkspaceResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourcegroups/bindplane-integration/providers/microsoft.operationalinsights/workspaces/bp-integration1\", \"_ResourceId\": \"/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourceGroups/devops/providers/Microsoft.ContainerService/managedClusters/log-analytics\"}]}", "system_properties": { "x-opt-enqueued-time": "2021-04-26T18:09:02.531Z", "x-opt-offset": 13446160, "x-opt-sequence-number": 1443 } } }This operator builds on top of
azure.EventHubby:azure.ParseEventfor initial event parsingazure_log_analytics_typelabel from the Log Analyticstyperecord fieldtimegeneratedrecord field as the entry's timestampRecordsdatafield into multiple entriesParsed entries look like this. Note that all log analytics fields are nested under
record.containerlog, wherecontainerlogis the name of the Log Analytics table that the logs originated from:{ "timestamp": "2021-04-30T17:24:45.024Z", "severity": 0, "labels": { "azure_log_analytics_table": "containerlog" }, "record": { "containerlog": { "_internal_workspaceresourceid": "/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourcegroups/bindplane-integration/providers/microsoft.operationalinsights/workspaces/bp-integration1", "_resourceid": "/subscriptions/09373b6b-bc8b-4093-925d-eb87334c7d56/resourceGroups/devops/providers/Microsoft.ContainerService/managedClusters/log-analytics", "computer": "aks-agentpool-39365618-vmss000001", "containerid": "597c8960fc6d7c3420515ef89f7aa6e9d180ceca05e7d98032e8f6ecc6d6a0ab", "logentry": "{\"http.req.id\":\"4e452a78-d51a-4a02-85a8-ddca29c572ec\",\"http.req.method\":\"GET\",\"http.req.path\":\"/_healthz\",\"http.resp.bytes\":2,\"http.resp.status\":200,\"http.resp.took_ms\":0,\"message\":\"request complete\",\"session\":\"x-readiness-probe\",\"severity\":\"debug\",\"timestamp\":\"2021-04-30T17:24:45.024421855Z\"}", "logentrysource": "stdout", "mg": "00000000-0000-0000-0000-000000000002", "sourcesystem": "Containers", "tenantid": "ae0db88b-40bb-40b7-b056-57980214436c", "timegenerated": "2021-04-30T17:24:45.0240000Z", "timeofcommand": "2021-04-30T17:24:59.0000000Z" }, "system_properties": { "x-opt-enqueued-time": "2021-04-30T17:25:06.579Z", "x-opt-offset": 34371131600, "x-opt-sequence-number": 29331 } } }Each entry derived from the event's batch retains shared records such as
system_properties.Please check that the PR fulfills these requirements