Enable the resource_to_telemetry_conversion to true in amp config.yaml files

[vasireddy99]

This PR will add dimensions/tags and more detailed information to AMP metrics which helps to identify the metrics using cluster_name, task_id etc.,

Modified the following ecs-amp config files by enabling the resource_to_telemetry_conversion to true

Paths to config files
config/ecs/ecs-amp.yaml
config/ecs/ecs-amp-xray.yaml
config/ecs/ecs-amp-xray-prometheus.yaml
config/ecs/ecs-amp-prometheus.yaml

[anuraaga]

Oh it looks like this setting is available, cool.

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/main/exporter/prometheusremotewriteexporter/config.go#L49

It's not documented at all 😱

https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/exporter/awsprometheusremotewriteexporter

[anuraaga]

Unfortunately, unlike the EMF exporter setting, this does not seem to support a list of labels to allow list

https://github.com/open-telemetry/opentelemetry-collector-contrib/blob/87ba2fa130a3b1353d717499e3b62c7bea4924a0/pkg/resourcetotelemetry/resource_to_telemetry.go#L31

We can't enable this because extremely high cardinality resource information will be sent and kill the prometheus. Perhaps that's why it's not documented :P

[codecov-commenter]

Codecov Report

Merging #739 (2a3af5e) into main (d2fcd75) will not change coverage.
The diff coverage is n/a.

❗ Current head 2a3af5e differs from pull request most recent head cbdf7b2. Consider uploading reports for the commit cbdf7b2 to get more accurate results

@@           Coverage Diff           @@
##             main     #739   +/-   ##
=======================================
  Coverage   54.69%   54.69%           
=======================================
  Files          11       11           
  Lines         298      298           
=======================================
  Hits          163      163           
  Misses        118      118           
  Partials       17       17           

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d2fcd75...cbdf7b2. Read the comment docs.

[vasireddy99]

Thanks for the comments, will determine the need with teams.

Please find the related PR and issues

Issue : open-telemetry/opentelemetry-collector#2412

PR : open-telemetry/opentelemetry-collector#2899

[alolita]

@anuraaga Agree with your concerns. Each service has to take a call on cardinality related performance concerns.

[alolita]

This is approved for AMP based on feedback from service team on no known limitations.

[anuraaga]

I'm still worried about sending resource data without an allow list - assuming the cardinality can be handled without slowing down queries, it's still a lot of storage used (process.command_line can be on the kilobyte scale for some apps) and could cause cost balooning without value to customers.

[alolita]

@anuraaga The service team has confirmed that they can handle high cardinality data.

[alolita]

lgtm based on service team feedback.