Improve conversion of resource attributes to/from prometheus (#2381)

* update resource attributes used by prometheus, and round-trip other resource attributes

* address feedbck

* address feedbck

* map target_info to resource, and use service.instance.id for instance

* formatting

* be clear on metric family vs metric point

* Update specification/metrics/datamodel.md

Co-authored-by: Anthony Mirabella <a9@aneurysm9.com>

* change deliniator to /

* fix formatting

Co-authored-by: Anthony Mirabella <a9@aneurysm9.com>
Co-authored-by: Bogdan Drutu <bogdandrutu@gmail.com>

specification/metrics/datamodel.md

@@ -1170,18 +1170,38 @@ Prometheus Cumulative metrics do not include the start time of the metric. When
 
 #### Resource Attributes
 
-When scraping a Prometheus endpoint, resource attributes MUST be added to the scraped metrics to distinguish them from metrics from other Prometheus endpoints.  In particular, `job` and `instance`, [as defined by Prometheus](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances), are needed to ensure Prometheus exporters can disambiguate metrics as [described below](#resource-attributes-1).
+When scraping a Prometheus endpoint, resource attributes MUST be added to the
+scraped metrics to distinguish them from metrics from other Prometheus
+endpoints. In particular, `service.name` and `service.instance.id`, are needed
+to ensure Prometheus exporters can disambiguate metrics using
+[`job` and `instance` labels](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances)
+as [described below](#resource-attributes-1).
 
-The following attributes MUST be associated with scraped metrics as resource attributes, and MUST NOT be added as metric attributes:
+The following attributes MUST be associated with scraped metrics as resource
+attributes, and MUST NOT be added as metric attributes:
 
 | OTLP Resource Attribute | Description |
 | ----------------------- | ----------- |
 | `service.name` | The configured name of the service that the target belongs to |
-| `job` | Identical to `service.name` |
-| `instance` | The <host>:<port> of the target's URL that was scraped. |
-| `host.name` | `instance` The <host> portion of `instance` |
-| `port` | `instance` The <port> portion of `instance` |
-| `scheme` | `http` or `https` |
+| `service.instance.id` | A unique identifier of the target.  By default, it should be the `<host>:<port>` of the scraped URL |
+
+The following attributes SHOULD be associated with scraped metrics as resource
+attributes, and MUST NOT be added as metric attributes:
+
+| OTLP Resource Attribute | Description |
+| ----------------------- | ----------- |
+| `net.host.name` | The `<host>` portion of the target's URL that was scraped |
+| `net.host.port` | The `<port>` portion of the target's URL that was scraped |
+| `http.scheme` | `http` or `https` |
+
+In addition to the attributes above, the
+["target" info](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems)
+metric family is used to supply additional resource attributes. If present,
+"target" info MUST be dropped from the batch of metrics, and all labels from
+the "target" info metric family MUST be converted to resource attributes
+attached to all other metrics which are part of the scrape. By default, label
+keys and values MUST NOT be altered (such as replacing `_` with `.` characters
+in keys).
 
 ### OTLP Metric points to Prometheus
 
@@ -1240,9 +1260,35 @@ OpenTelemetry Metric Attributes MUST be converted to [Prometheus labels](https:/
 
 #### Resource Attributes
 
-In SDK Prometheus (pull) exporters, all resource attributes MUST be dropped, and MUST NOT be attached as labels. The scraper of the endpoint is expected to discover resource attributes of the endpoint it is scraping.
-
-In the Collector's Prometheus pull and push (remote-write) exporters, it is possible for metrics from multiple targets to be sent together, so targets must be disambiguated from one another.  However, the Prometheus exposition format and [remote-write](https://github.com/Prometheus/Prometheus/blob/main/prompb/remote.proto) formats do not include a notion of resource, and expect metric labels to distinguish scraped targets.  By convention, [`job` and `instance`](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances) labels distinguish targets and are expected to be present on metrics exposed on a Prometheus pull exporter (a ["federated"](https://Prometheus.io/docs/Prometheus/latest/federation/) Prometheus endpoint) or pushed via Prometheus remote-write. In the collector Prometheus exporters, the `job` and `instance` resource attributes MUST be converted to Prometheus metric labels, and other resource attributes SHOULD NOT be converted to metric labels.
+In SDK Prometheus (pull) exporters, resource attributes SHOULD be converted to the ["target" info](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems) metric family; otherwise, they MUST be dropped, and MUST NOT be attached as labels to other metric families. The "target" info metric family MUST be an info-typed metric family whose labels MUST include the resource attributes, and MUST NOT include any other labels. There MUST be at most one "target" info metric family exposed on a Prometheus endpoint.
+
+In the Collector's Prometheus pull and push (remote-write) exporters, it is
+possible for metrics from multiple targets to be sent together, so targets must
+be disambiguated from one another. However, the Prometheus exposition format
+and [remote-write](https://github.com/Prometheus/Prometheus/blob/main/prompb/remote.proto)
+formats do not include a notion of resource, and expect metric labels to
+distinguish scraped targets. By convention, [`job` and `instance`](https://Prometheus.io/docs/concepts/jobs_instances/#jobs-and-instances)
+labels distinguish targets and are expected to be present on metrics exposed on
+a Prometheus pull exporter (a ["federated"](https://Prometheus.io/docs/Prometheus/latest/federation/)
+Prometheus endpoint) or pushed via Prometheus remote-write. In OTLP, the
+`service.name`, `service.namespace`, and `service.instance.id` triplet is
+[required to be unique](https://github.com/open-telemetry/opentelemetry-specification/blob/da74730bf835010229a9612c281f052e26553218/specification/resource/semantic_conventions/README.md#service),
+which makes them good candidates to use to construct `job` and `instance`. In
+the collector Prometheus exporters, the `service.name` and `service.namespace`
+attributes MUST be combined as `<service.namespace>/<service.name>`, or
+`<service.name>` if namespace is empty, to form the `job` metric label.  The
+`service.instance.id` attribute, if present, MUST be converted to the
+`instance` label; otherwise, `instance` should be added with an empty value.
+Other resource attributes SHOULD be converted to a
+["target" info](https://github.com/OpenObservability/OpenMetrics/blob/main/specification/OpenMetrics.md#supporting-target-metadata-in-both-push-based-and-pull-based-systems)
+metric family, or MUST be dropped. The "target" info metric family is an info-typed metric family
+whose labels MUST include the resource attributes, and MUST NOT include any
+other labels other than `job` and `instance`.  There MUST be at most one
+"target" info metric point exported for each unique combination of `job` and `instance`.
+
+If info-typed metric families are not yet supported by the language Prometheus client library, a gauge-typed metric family named "target" info with a constant value of 1 MUST be used instead.
+
+To convert OTLP resource attributes to Prometheus labels, string Attribute values are converted directly to labels, and non-string Attribute values MUST be converted to string attributes following the [attribute specification](https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/common/common.md#attribute).
 
 ## Footnotes