Updated traces performance test results for Datadog exporter against real backend

[mx-psi]

Hi,

Please find the updated performance results for the datadog trace exporter as of commit a8603ce below. The tests were run against an internal environment following the instructions of the aws-otel-test-framework repository.

100 TPS

{
   "testcase":"datadog_exporter_trace_mock",
   "instanceType":"m5.2xlarge",
   "receivers":[
      "otlp"
   ],
   "processors":[
      "batch"
   ],
   "exporters":[
      "datadog"
   ],
   "dataType":"otlp",
   "dataMode":"trace",
   "dataRate":100,
   "avgCpu":8.565867782510242,
   "avgMem":74.50371413333333,
   "commitId":"dummy_commit",
   "collectionPeriod":10,
   "testingAmi":"soaking_linux"
}

1000 TPS

{
   "testcase":"datadog_exporter_trace_mock",
   "instanceType":"m5.2xlarge",
   "receivers":[
      "otlp"
   ],
   "processors":[
      "batch"
   ],
   "exporters":[
      "datadog"
   ],
   "dataType":"otlp",
   "dataMode":"trace",
   "dataRate":1000,
   "avgCpu":28.722534286136238,
   "avgMem":75.89580799999999,
   "commitId":"dummy_commit",
   "collectionPeriod":10,
   "testingAmi":"soaking_linux"
}

5000 TPS

{
   "testcase":"datadog_exporter_trace_mock",
   "instanceType":"m5.2xlarge",
   "receivers":[
      "otlp"
   ],
   "processors":[
      "batch"
   ],
   "exporters":[
      "datadog"
   ],
   "dataType":"otlp",
   "dataMode":"trace",
   "dataRate":5000,
   "avgCpu":99.33122504914861,
   "avgMem":89.73421226666667,
   "commitId":"dummy_commit",
   "collectionPeriod":10,
   "testingAmi":"soaking_linux"
}

We also conducted end to end testing to validate the behavior.

---

cc @alolita for OTLP Tracing stability

[alolita]

Hi @mx-psi Thanks for the perf results for the trace exporter. Did you use the current Collector release (https://github.com/open-telemetry/opentelemetry-collector/releases/tag/v0.31.0) for these tests?

[mx-psi]

@alolita I ran the tests using the latest available AWS OpenTelemetry Collector commit, which uses v0.30.0 of all components. I can re-run the tests with the contrib distro v0.31.0 or wait until AWS releases an updated version of their Collector. Would either of those work?

[alolita]

Hi @mx-psi once the final Collector version is released on OTEL, we will build the ADOT Collector thereafter. Would recommend doing a final run with both releases. We require these tests to be run with the ADOT Collector release.

[mx-psi]

@alolita That makes sense, we can keep this issue open and I will re-run the test when that happens. Thanks for the help here.

[mx-psi]

Superseded by aws-observability/aws-otel-test-framework#340