Skip to content

Commit

Permalink
Messaging metrics: early draft
Browse files Browse the repository at this point in the history
  • Loading branch information
lmolkova committed Jun 14, 2023
1 parent 0e24f77 commit cdbdf57
Show file tree
Hide file tree
Showing 4 changed files with 226 additions and 23 deletions.
55 changes: 55 additions & 0 deletions semantic_conventions/metrics/messaging.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
groups:
- id: metric.messaging.attributes
type: attribute_group
brief: "Common messaging metrics attributes."
extends: messaging.attributes.common
attributes:
- ref: messaging.destination.name
requirement_level:
conditionally_required: if and only if `messaging.destination.name` is known to have low cardinality. Otherwise, `messaging.destination.template` MAY be populated.
- ref: messaging.destination.template
requirement_level:
conditionally_required: if available.
- id: messaging.todo.status
type: string
brief: "The status of the operation: TODO"
requirement_level: required
examples: ["ok", "error", "timeout"]

- id: metric.messaging.publish.duration
type: metric
metric_name: messaging.publish.duration
brief: "Measures the duration of publish operation."
instrument: histogram
unit: "s"
extends: metric.messaging.attributes

- id: metric.messaging.receive.duration
type: metric
metric_name: messaging.receive.duration
brief: "Measures the duration of receive operation."
instrument: histogram
unit: "s"
extends: metric.messaging.attributes

- id: metric.messaging.delivery.duration
type: metric
metric_name: messaging.delivery.duration
brief: "Measures the duration of receive operation."
instrument: histogram
unit: "s"
extends: metric.messaging.attributes

- id: metric.messaging.settle.duration
type: metric
metric_name: messaging.settle.duration
brief: "Measures the duration of settle operation."
instrument: histogram
unit: "s"
extends: metric.messaging.attributes
attributes:
- id: messaging.todo_settlement.status
type: string
brief: "The status of the settlement: acknowledged, rejected, TODO"
examples: ["ack", "nack"]

38 changes: 23 additions & 15 deletions semantic_conventions/trace/messaging.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -85,18 +85,36 @@ groups:
type: boolean
brief: 'A boolean that is true if the message source is anonymous (could be unnamed or have auto-generated name).'

- id: messaging
- id: messaging.attributes.common
type: attribute_group
brief: "Common messaging attributes."
prefix: messaging
type: span
brief: >
This document defines general attributes used in
messaging systems.
attributes:
- id: system
type: string
requirement_level: required
brief: 'A string identifying the messaging system.'
examples: ['kafka', 'rabbitmq', 'rocketmq', 'activemq', 'AmazonSQS']
- ref: server.address
requirement_level:
conditionally_required: If available.
- ref: server.port
requirement_level:
conditionally_required: If available.
- ref: server.socket.address
tag: connection-level
- ref: network.protocol.name
examples: ['amqp', 'mqtt']
- ref: network.protocol.version

- id: messaging
prefix: messaging
type: span
brief: >
This document defines general attributes used in
messaging systems.
extends: messaging.attributes.common
attributes:
- id: operation
type:
allow_custom_values: true
Expand Down Expand Up @@ -141,13 +159,6 @@ groups:
- ref: messaging.message.payload_compressed_size_bytes
requirement_level:
recommended: Only if span represents operation on a single message.
- ref: server.address
note: >
This should be the IP/hostname of the broker (or other network-level peer) this specific message is sent to/received from.
requirement_level:
conditionally_required: If available.
- ref: server.socket.address
tag: connection-level
- ref: server.socket.port
tag: connection-level
- ref: network.transport
Expand All @@ -158,9 +169,6 @@ groups:
tag: connection-level
requirement_level:
recommended: If different than `server.address` and if `server.socket.address` is set.
- ref: network.protocol.name
examples: ['amqp', 'mqtt']
- ref: network.protocol.version

- id: messaging.producer
prefix: messaging
Expand Down
126 changes: 126 additions & 0 deletions specification/metrics/semantic_conventions/messaging-metrics.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,126 @@
<!--- Hugo front matter used to generate the website version of this page:
linkTitle: HTTP
--->

# Semantic Conventions for Messaging Metrics

**Status**: [Experimental][DocumentStatus]

The conventions described in this section are messaging specific.

**Disclaimer:** These are initial messaging metric instruments and attributes but more may be added in the future.

<!-- toc -->
<!-- tocstop -->

## Metric: `messaging.publish.duration`

This metric is required.

When this metric is reported alongside an messaging publish span, the metric value SHOULD be the same as the corresponding span duration.

This metric SHOULD be specified with
[`ExplicitBucketBoundaries`](https://github.com/open-telemetry/opentelemetry-specification/tree/v1.21.0/specification/metrics/api.md#instrument-advice)
of `[ 0, 0.005, 0.01, 0.025, 0.05, 0.075, 0.1, 0.25, 0.5, 0.75, 1, 2.5, 5, 7.5, 10 ]`.

<!-- semconv metric.messaging.publish.duration(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `messaging.publish.duration` | Histogram | `s` | Measures the duration of publish operation. |
<!-- endsemconv -->

<!-- semconv metric.messaging.publish.duration(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `messaging.system` | string | A string identifying the messaging system. | `kafka`; `rabbitmq`; `rocketmq`; `activemq`; `AmazonSQS` | Required |
| `messaging.todo.status` | string | The status of the operation: TODO | `ok`; `error`; `timeout` | Required |
| `messaging.destination.name` | string | The message destination name [1] | `MyQueue`; `MyTopic` | Conditionally Required: [2] |
| `messaging.destination.template` | string | Low cardinality representation of the messaging destination name [3] | `/customers/{customerId}` | Conditionally Required: if available. |
| [`network.protocol.name`](../../trace/semantic_conventions/span-general.md) | string | [OSI Application Layer](https://osi-model.com/application-layer/) or non-OSI equivalent. The value SHOULD be normalized to lowercase. | `amqp`; `mqtt` | Recommended |
| [`network.protocol.version`](../../trace/semantic_conventions/span-general.md) | string | Version of the application layer protocol used. See note below. [4] | `3.1.1` | Recommended |
| [`server.address`](../../trace/semantic_conventions/span-general.md) | string | Logical server hostname, matches server FQDN if available, and IP or socket address if FQDN is not known. | `example.com` | Conditionally Required: If available. |
| [`server.port`](../../trace/semantic_conventions/span-general.md) | int | Logical server port number | `80`; `8080`; `443` | Conditionally Required: If available. |
| [`server.socket.address`](../../trace/semantic_conventions/span-general.md) | string | Physical server IP address or Unix socket address. | `10.5.3.2` | Recommended: If different than `server.address`. |

**[1]:** Destination name SHOULD uniquely identify a specific queue, topic or other entity within the broker. If
the broker does not have such notion, the destination name SHOULD uniquely identify the broker.

**[2]:** if and only if `messaging.destination.name` is known to have low cardinality. Otherwise, `messaging.destination.template` MAY be populated.

**[3]:** Destination names could be constructed from templates. An example would be a destination name involving a user name or product id. Although the destination name in this case is of high cardinality, the underlying template is of low cardinality and can be effectively used for grouping and aggregation.

**[4]:** `network.protocol.version` refers to the version of the protocol used and might be different from the protocol client's version. If the HTTP client used has a version of `0.27.2`, but sends HTTP version `1.1`, this attribute should be set to `1.1`.
<!-- endsemconv -->

## Metric: `messaging.receive.duration`

This metric is recommended when messaging system supports poll-based receive operations.

<!-- semconv metric.messaging.receive.duration(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `messaging.receive.duration` | Histogram | `s` | Measures the duration of receive operation. |
<!-- endsemconv -->

<!-- semconv metric.messaging.publish.duration(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `messaging.system` | string | A string identifying the messaging system. | `kafka`; `rabbitmq`; `rocketmq`; `activemq`; `AmazonSQS` | Required |
| `messaging.todo.status` | string | The status of the operation: TODO | `ok`; `error`; `timeout` | Required |
| `messaging.destination.name` | string | The message destination name [1] | `MyQueue`; `MyTopic` | Conditionally Required: [2] |
| `messaging.destination.template` | string | Low cardinality representation of the messaging destination name [3] | `/customers/{customerId}` | Conditionally Required: if available. |
| [`network.protocol.name`](../../trace/semantic_conventions/span-general.md) | string | [OSI Application Layer](https://osi-model.com/application-layer/) or non-OSI equivalent. The value SHOULD be normalized to lowercase. | `amqp`; `mqtt` | Recommended |
| [`network.protocol.version`](../../trace/semantic_conventions/span-general.md) | string | Version of the application layer protocol used. See note below. [4] | `3.1.1` | Recommended |
| [`server.address`](../../trace/semantic_conventions/span-general.md) | string | Logical server hostname, matches server FQDN if available, and IP or socket address if FQDN is not known. | `example.com` | Conditionally Required: If available. |
| [`server.port`](../../trace/semantic_conventions/span-general.md) | int | Logical server port number | `80`; `8080`; `443` | Conditionally Required: If available. |
| [`server.socket.address`](../../trace/semantic_conventions/span-general.md) | string | Physical server IP address or Unix socket address. | `10.5.3.2` | Recommended: If different than `server.address`. |

**[1]:** Destination name SHOULD uniquely identify a specific queue, topic or other entity within the broker. If
the broker does not have such notion, the destination name SHOULD uniquely identify the broker.

**[2]:** if and only if `messaging.destination.name` is known to have low cardinality. Otherwise, `messaging.destination.template` MAY be populated.

**[3]:** Destination names could be constructed from templates. An example would be a destination name involving a user name or product id. Although the destination name in this case is of high cardinality, the underlying template is of low cardinality and can be effectively used for grouping and aggregation.

**[4]:** `network.protocol.version` refers to the version of the protocol used and might be different from the protocol client's version. If the HTTP client used has a version of `0.27.2`, but sends HTTP version `1.1`, this attribute should be set to `1.1`.
<!-- endsemconv -->


## Metric: `messaging.settle.duration`

This metric is recommended when messaging system supports settle operations.

<!-- semconv metric.messaging.settle.duration(metric_table) -->
| Name | Instrument Type | Unit (UCUM) | Description |
| -------- | --------------- | ----------- | -------------- |
| `messaging.settle.duration` | Histogram | `s` | Measures the duration of settle operation. |
<!-- endsemconv -->

<!-- semconv metric.messaging.settle.duration(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `messaging.system` | string | A string identifying the messaging system. | `kafka`; `rabbitmq`; `rocketmq`; `activemq`; `AmazonSQS` | Required |
| `messaging.todo.status` | string | The status of the operation: TODO | `ok`; `error`; `timeout` | Required |
| `messaging.todo_settlement.status` | string | The status of the settlement: acknowledged, rejected, TODO | `ack`; `nack` | Recommended |
| `messaging.destination.name` | string | The message destination name [1] | `MyQueue`; `MyTopic` | Conditionally Required: [2] |
| `messaging.destination.template` | string | Low cardinality representation of the messaging destination name [3] | `/customers/{customerId}` | Conditionally Required: if available. |
| [`network.protocol.name`](../../trace/semantic_conventions/span-general.md) | string | [OSI Application Layer](https://osi-model.com/application-layer/) or non-OSI equivalent. The value SHOULD be normalized to lowercase. | `amqp`; `mqtt` | Recommended |
| [`network.protocol.version`](../../trace/semantic_conventions/span-general.md) | string | Version of the application layer protocol used. See note below. [4] | `3.1.1` | Recommended |
| [`server.address`](../../trace/semantic_conventions/span-general.md) | string | Logical server hostname, matches server FQDN if available, and IP or socket address if FQDN is not known. | `example.com` | Conditionally Required: If available. |
| [`server.port`](../../trace/semantic_conventions/span-general.md) | int | Logical server port number | `80`; `8080`; `443` | Conditionally Required: If available. |
| [`server.socket.address`](../../trace/semantic_conventions/span-general.md) | string | Physical server IP address or Unix socket address. | `10.5.3.2` | Recommended: If different than `server.address`. |

**[1]:** Destination name SHOULD uniquely identify a specific queue, topic or other entity within the broker. If
the broker does not have such notion, the destination name SHOULD uniquely identify the broker.

**[2]:** if and only if `messaging.destination.name` is known to have low cardinality. Otherwise, `messaging.destination.template` MAY be populated.

**[3]:** Destination names could be constructed from templates. An example would be a destination name involving a user name or product id. Although the destination name in this case is of high cardinality, the underlying template is of low cardinality and can be effectively used for grouping and aggregation.

**[4]:** `network.protocol.version` refers to the version of the protocol used and might be different from the protocol client's version. If the HTTP client used has a version of `0.27.2`, but sends HTTP version `1.1`, this attribute should be set to `1.1`.
<!-- endsemconv -->

## Others:

- batch size (payload, in bytes): probably transport-level concept, not necessary here
- consumer lag/queue size - need to define new attributes (enqueue/creation time)
30 changes: 22 additions & 8 deletions specification/trace/semantic_conventions/messaging.md
Original file line number Diff line number Diff line change
Expand Up @@ -208,7 +208,7 @@ The following operations related to messages are defined for these semantic conv

## Messaging attributes

<!-- semconv messaging -->
<!-- semconv messaging(full) -->
| Attribute | Type | Description | Examples | Requirement Level |
|---|---|---|---|---|
| `messaging.system` | string | A string identifying the messaging system. | `kafka`; `rabbitmq`; `rocketmq`; `activemq`; `AmazonSQS` | Required |
Expand All @@ -223,9 +223,10 @@ The following operations related to messages are defined for these semantic conv
| [`network.protocol.version`](span-general.md) | string | Version of the application layer protocol used. See note below. [8] | `3.1.1` | Recommended |
| [`network.transport`](span-general.md) | string | [OSI Transport Layer](https://osi-model.com/transport-layer/) or [Inter-process Communication method](https://en.wikipedia.org/wiki/Inter-process_communication). The value SHOULD be normalized to lowercase. | `tcp`; `udp` | Recommended |
| [`network.type`](span-general.md) | string | [OSI Network Layer](https://osi-model.com/network-layer/) or non-OSI equivalent. The value SHOULD be normalized to lowercase. | `ipv4`; `ipv6` | Recommended |
| [`server.address`](span-general.md) | string | Logical server hostname, matches server FQDN if available, and IP or socket address if FQDN is not known. [9] | `example.com` | Conditionally Required: If available. |
| [`server.address`](span-general.md) | string | Logical server hostname, matches server FQDN if available, and IP or socket address if FQDN is not known. | `example.com` | Conditionally Required: If available. |
| [`server.port`](span-general.md) | int | Logical server port number | `80`; `8080`; `443` | Conditionally Required: If available. |
| [`server.socket.address`](span-general.md) | string | Physical server IP address or Unix socket address. | `10.5.3.2` | Recommended: If different than `server.address`. |
| [`server.socket.domain`](span-general.md) | string | The domain name of an immediate peer. [10] | `proxy.example.com` | Recommended: [11] |
| [`server.socket.domain`](span-general.md) | string | The domain name of an immediate peer. [9] | `proxy.example.com` | Recommended: [10] |
| [`server.socket.port`](span-general.md) | int | Physical server port. | `16456` | Recommended: If different than `server.port`. |

**[1]:** If a custom value is used, it MUST be of low cardinality.
Expand All @@ -244,11 +245,9 @@ The following operations related to messages are defined for these semantic conv

**[8]:** `network.protocol.version` refers to the version of the protocol used and might be different from the protocol client's version. If the HTTP client used has a version of `0.27.2`, but sends HTTP version `1.1`, this attribute should be set to `1.1`.

**[9]:** This should be the IP/hostname of the broker (or other network-level peer) this specific message is sent to/received from.
**[9]:** Typically observed from the client side, and represents a proxy or other intermediary domain name.

**[10]:** Typically observed from the client side, and represents a proxy or other intermediary domain name.

**[11]:** If different than `server.address` and if `server.socket.address` is set.
**[10]:** If different than `server.address` and if `server.socket.address` is set.

`messaging.operation` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

Expand All @@ -257,9 +256,24 @@ The following operations related to messages are defined for these semantic conv
| `publish` | publish |
| `receive` | receive |
| `process` | process |

`network.transport` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

| Value | Description |
|---|---|
| `tcp` | TCP |
| `udp` | UDP |
| `pipe` | Named or anonymous pipe. See note below. |
| `unix` | Unix domain socket |

`network.type` has the following list of well-known values. If one of them applies, then the respective value MUST be used, otherwise a custom value MAY be used.

| Value | Description |
|---|---|
| `ipv4` | IPv4 |
| `ipv6` | IPv6 |
<!-- endsemconv -->

Additionally `server.port` from the [network attributes][] is recommended.
Furthermore, it is strongly recommended to add the [`network.transport`][] attribute and follow its guidelines, especially for in-process queueing systems (like [Hangfire][], for example).
These attributes should be set to the broker to which the message is sent/from which it is received.

Expand Down

0 comments on commit cdbdf57

Please sign in to comment.