Poor performance of EMF Exporter causes metric lost in the long run

[bjrara]

What happened

Metrics exported by EMF exporter were lost by 1-3 minutes during the long run, causing gaps in the dashboard. Followed by a false burst, it automatically recovered from the stuck.

Usually metrics were reported at a constant rate, but when the collector "stucked", it was unsteady.

After adding more debugging logs, we found it took 30 seconds to digest one chunk of metrics by average, e.g. metrics from one Pod in terms of using prometheus receiver.

2021-03-05T22:47:12.090Z	INFO	awsemfexporter@v0.0.0/emf_exporter.go:125	Start processing resource metrics	{"component_kind": "exporter", "component_type": "awsemf", "component_name": "awsemf", "labels": {"ClusterName":"otel-sample-cluster","TaskId":"kubernetes-pod-appmesh-envoy","host.name":"192.168.39.233","port":"9901","scheme":"http","service.name":"kubernetes-pod-appmesh-envoy"}}
2021-03-05T22:47:48.359Z	INFO	awsemfexporter@v0.0.0/emf_exporter.go:152	Finish processing resource metrics	{"component_kind": "exporter", "component_type": "awsemf", "component_name": "awsemf", "labels": {"ClusterName":"otel-sample-cluster","TaskId":"kubernetes-pod-appmesh-envoy","host.name":"192.168.39.233","port":"9901","scheme":"http","service.name":"kubernetes-pod-appmesh-envoy"}}

It took 3 minutes to finish the same process at the time of metrics lost.

2021-03-05T22:44:37.697Z	INFO	awsemfexporter@v0.0.0/emf_exporter.go:125	Start processing resource metrics	{"component_kind": "exporter", "component_type": "awsemf", "component_name": "awsemf", "labels": {"ClusterName":"otel-sample-cluster","TaskId":"kubernetes-pod-appmesh-envoy","host.name":"192.168.39.233","port":"9901","scheme":"http","service.name":"kubernetes-pod-appmesh-envoy"}}
2021-03-05T22:47:11.933Z	INFO	awsemfexporter@v0.0.0/emf_exporter.go:152	Finish processing resource metrics	{"component_kind": "exporter", "component_type": "awsemf", "component_name": "awsemf", "labels": {"ClusterName":"otel-sample-cluster","TaskId":"kubernetes-pod-appmesh-envoy","host.name":"192.168.39.233","port":"9901","scheme":"http","service.name":"kubernetes-pod-appmesh-envoy"}}

Root cause

Although metrics are grouped before the pushing happens, every logEventBatch is sent one by one, causing high network latencies, which drags down the performance.

How to solve it

Pushing EMF logs in batch.

[bjrara]

The performance is improved from 30s -> 200ms after applying changes from open-telemetry/opentelemetry-collector-contrib#2572

2021-03-06T06:56:27.196Z	INFO	awsemfexporter@v0.0.0/emf_exporter.go:125	Start processing resource metrics	{"component_kind": "exporter", "component_type": "awsemf", "component_name": "awsemf", "labels": {"ClusterName":"otel-sample-cluster","TaskId":"kubernetes-pod-appmesh-envoy","host.name":"192.168.42.106","port":"9901","scheme":"http","service.name":"kubernetes-pod-appmesh-envoy"}}
2021-03-06T06:56:27.399Z	INFO	awsemfexporter@v0.0.0/emf_exporter.go:162	Finish processing resource metrics	{"component_kind": "exporter", "component_type": "awsemf", "component_name": "awsemf", "labels": {"ClusterName":"otel-sample-cluster","TaskId":"kubernetes-pod-appmesh-envoy","host.name":"192.168.42.106","port":"9901","scheme":"http","service.name":"kubernetes-pod-appmesh-envoy"}}

[bjrara]

Close this issue because the fix is merged.