From 332ea52e2050a2970c1c806437c0ae7227796f49 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Wed, 22 Jan 2020 16:20:54 +0100 Subject: [PATCH 01/34] Carve out messaging from semantic conventions draft --- specification/data-messaging.md | 81 +++++++++++++++++++++++++++++++++ 1 file changed, 81 insertions(+) create mode 100644 specification/data-messaging.md diff --git a/specification/data-messaging.md b/specification/data-messaging.md new file mode 100644 index 00000000000..8929cadf4e4 --- /dev/null +++ b/specification/data-messaging.md @@ -0,0 +1,81 @@ +# Messaging systems + +> 🚧 WORK IN PROGRESS + + + + + +- [Definitions](#definitions) +- [Conventions](#conventions) +- [Messaging attributes](#messaging-attributes) + + + +## Definitions + +Although messaging systems are not as standardized as, e.g., HTTP, it is assumed that the following definitions are applicable to most of them that have similar concepts at all (names borrowed mostly from JMS): + +A *message* usually consists of headers (or properties, or meta information) and an optional body. It is sent by a single message *producer* to: + +* Physically: some message *broker* (which can be e.g., a single server, or a cluster, or a local process reached via IPC). The broker handles the actual routing, delivery, re-delivery, persistence, etc. In some messaging systems the broker may be identical or co-located with (some) message consumers. +* Logically: some particular message *destination*. + +A destination is usually identified by some name unique within the messaging system instance, which might look like an URL or a simple one-word identifier. Two kinds of targets are are distinguished: *topic*s and *queue*s. +A message that is sent (the send-operation is often called "*publish*" in this context) to a *topic* is broadcasted to all *subscribers* of the topic. +A message submitted to a queue is processed by a message *consumer* (usually exactly once although some message systems support a more performant at least once-mode for messages with [idempotent][] processing). + +The consumption of a message can happen in multiple steps: +First, the lower-level receiving of a message at a consumer, and then the logical processing of the message. +For receiving, the consumer may have the choice between peeking a destination to get a message back only if one is currently available +(for the purpose of this document, it is not relevant whether the message is removed from the destination or not by the "peek" operation), +or waiting until a message is received. +Often, the waiting for a message is not particularly interesting and hidden away in a framework that only invokes some handler function to process a message once one is received +(in the same way that the listening on a TCP port for an incoming HTTP message is not particularly interesting). +However, in a synchronous conversation, the wait time for a message is important. + +In some messaging systems, a message can receive a reply message that answers a particular other message that was sent earlier. All messages that are grouped together by such a reply-relationship are called a *conversation*. The grouping usually happens through some sort of "In-Reply-To:" meta information or an explicit conversation ID. Sometimes a conversation can span multiple message targets (e.g. initiated via a topic, continued on a temporary one-to-one queue). + +Some messaging systems support the concept of *temporary destination* (often only temporary queues) that are established just for a particular set of communication partners (often one to one) or conversation. Often such destinations are unnamed or have an auto-generated name. + +[idempotent]: https://en.wikipedia.org/wiki/Idempotence + +## Conventions + +Given these definitions, the remainder of this section describes the semantic conventions that shall be followed for Spans describing interactions with messaging systems. + +**Span name:** The span name should usually be set to the message destination name. +The conversation ID should be used instead when it is expected to have lower cardinality. +In particular, the conversation ID must be used if the message destination is unnamed or temporary. + +**Span kind:** A producer of a message should set the span kind to `PRODUCER` unless it synchronously waits for a response: then it should use `CLIENT`. +The processor of the message should set the kind to `CONSUMER`, unless it always sends back a reply that is directed to the producer of the message +(as opposed to e.g., a queue on which the producer happens to listen): then it should use `SERVER`. + +## Messaging attributes + +| Attribute name | Notes and examples | Required? | +| -------------- | ---------------------------------------------------------------------- | --------- | +| `component` | Denotes the type of the span and needs to be `"msg"`. | Yes | +| `msg.flavor` | A string identifying the messaging system kind used. Should be of the form `KIND.TECH`, e.g. `JMS.Web Sphere` or `AMQP.RabbitMQ`. | Yes | +| `msg.dst` | The message destination name. A deprecated alternative key for this attribute is [`message_bus.destination`][ot-msg]. | Yes | +| `msg.dst_kind` | The kind of message destination: Either `"queue"` or `"topic"`. | Yes | +| `msg.tmp_dst` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | +| `msg.id` | An integer or string used by the messaging system as an identifier for the message. | No | +| `msg.conversation_id` | An integer or string identifying the conversation to which the message belongs. Sometimes called "correlation ID". | No | + +It is strongly recommended to also set at least the [network attributes][] `net.peer.ip`, `net.peer.name` and `net.peer.port`. + +[network attributes]: data-span-general.md#general-network-connection-attributes +[ot-msg]: https://github.com/opentracing/specification/blob/master/semantic_conventions.md#message-bus + +For message consumers, the following additional attributes may be set: + +| Attribute name | Notes and examples | Required? | +| -------------- | ---------------------------------------------------------------------- | --------- | +| `msg.op` | A string identifying which part and kind of message consumption this span describes: `recv`, `peek` or `proc`. (If the operation is `send`, this attribute must not be set: the operation can be inferred from the span kind in that case.) | No | +| `msg.peeked_some` | A boolean indicating whether a `peek` operation resulted in a received message or not. Must only be added if `msg.op` is `peek`. I missing, assumed to be `true` if `msg.id` is set, otherwise assumed to be `false`. | No | + +Note that one or multiple Spans with `msg.op` = `proc` may often be the children of a Span with `msg.op` = `recv` or `peek`. +Even though in that case one might think that the span kind is `INTERNAL`, that kind MUST NOT be used. +Instead span kind should be set to either `CONSUMER` or `SERVER` according to the rules defined above. \ No newline at end of file From 6e59559a7632fce2eec56c02cf6ac4d07b192289 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Wed, 22 Jan 2020 16:23:10 +0100 Subject: [PATCH 02/34] Clarify message destination name --- specification/data-messaging.md | 3 +-- 1 file changed, 1 insertion(+), 2 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 8929cadf4e4..cd64ea9180d 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -58,7 +58,7 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | -------------- | ---------------------------------------------------------------------- | --------- | | `component` | Denotes the type of the span and needs to be `"msg"`. | Yes | | `msg.flavor` | A string identifying the messaging system kind used. Should be of the form `KIND.TECH`, e.g. `JMS.Web Sphere` or `AMQP.RabbitMQ`. | Yes | -| `msg.dst` | The message destination name. A deprecated alternative key for this attribute is [`message_bus.destination`][ot-msg]. | Yes | +| `msg.dst` | The message destination name, e.g. `"MyQueue"` or `"MyTopic"`. | Yes | | `msg.dst_kind` | The kind of message destination: Either `"queue"` or `"topic"`. | Yes | | `msg.tmp_dst` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | | `msg.id` | An integer or string used by the messaging system as an identifier for the message. | No | @@ -67,7 +67,6 @@ The processor of the message should set the kind to `CONSUMER`, unless it always It is strongly recommended to also set at least the [network attributes][] `net.peer.ip`, `net.peer.name` and `net.peer.port`. [network attributes]: data-span-general.md#general-network-connection-attributes -[ot-msg]: https://github.com/opentracing/specification/blob/master/semantic_conventions.md#message-bus For message consumers, the following additional attributes may be set: From 1f0e5d10034a8552a6d1aedb82e01a62e64f7815 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Wed, 22 Jan 2020 16:33:12 +0100 Subject: [PATCH 03/34] Rename msg to messaging and avoid other abbrevs --- specification/data-messaging.md | 22 +++++++++++----------- 1 file changed, 11 insertions(+), 11 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index cd64ea9180d..34a3968731f 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -56,13 +56,13 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | Attribute name | Notes and examples | Required? | | -------------- | ---------------------------------------------------------------------- | --------- | -| `component` | Denotes the type of the span and needs to be `"msg"`. | Yes | -| `msg.flavor` | A string identifying the messaging system kind used. Should be of the form `KIND.TECH`, e.g. `JMS.Web Sphere` or `AMQP.RabbitMQ`. | Yes | -| `msg.dst` | The message destination name, e.g. `"MyQueue"` or `"MyTopic"`. | Yes | -| `msg.dst_kind` | The kind of message destination: Either `"queue"` or `"topic"`. | Yes | -| `msg.tmp_dst` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | -| `msg.id` | An integer or string used by the messaging system as an identifier for the message. | No | -| `msg.conversation_id` | An integer or string identifying the conversation to which the message belongs. Sometimes called "correlation ID". | No | +| `component` | Denotes the type of the span and needs to be `"messaging"`. | Yes | +| `messaging.flavor` | A string identifying the messaging system kind used. Should be of the form `KIND.TECH`, e.g. `JMS.Web Sphere` or `AMQP.RabbitMQ`. | Yes | +| `messaging.dst` | The message destination name, e.g. `"MyQueue"` or `"MyTopic"`. | Yes | +| `messaging.dst_kind` | The kind of message destination: Either `"queue"` or `"topic"`. | Yes | +| `messaging.tmp_dst` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | +| `messaging.message_id` | An integer or string used by the messaging system as an identifier for the message. | No | +| `messaging.conversation_id` | An integer or string identifying the conversation to which the message belongs. Sometimes called "correlation ID". | No | It is strongly recommended to also set at least the [network attributes][] `net.peer.ip`, `net.peer.name` and `net.peer.port`. @@ -72,9 +72,9 @@ For message consumers, the following additional attributes may be set: | Attribute name | Notes and examples | Required? | | -------------- | ---------------------------------------------------------------------- | --------- | -| `msg.op` | A string identifying which part and kind of message consumption this span describes: `recv`, `peek` or `proc`. (If the operation is `send`, this attribute must not be set: the operation can be inferred from the span kind in that case.) | No | -| `msg.peeked_some` | A boolean indicating whether a `peek` operation resulted in a received message or not. Must only be added if `msg.op` is `peek`. I missing, assumed to be `true` if `msg.id` is set, otherwise assumed to be `false`. | No | +| `messaging.operation` | A string identifying which part and kind of message consumption this span describes: `receive`, `peek` or `process`. (If the operation is `send`, this attribute must not be set: the operation can be inferred from the span kind in that case.) | No | +| `messaging.peeked_some` | A boolean indicating whether a `peek` operation resulted in a received message or not. Must only be added if `messaging.operation` is `peek`. I missing, assumed to be `true` if `messaging.message_id` is set, otherwise assumed to be `false`. | No | -Note that one or multiple Spans with `msg.op` = `proc` may often be the children of a Span with `msg.op` = `recv` or `peek`. +Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.op` = `receive` or `peek`. Even though in that case one might think that the span kind is `INTERNAL`, that kind MUST NOT be used. -Instead span kind should be set to either `CONSUMER` or `SERVER` according to the rules defined above. \ No newline at end of file +Instead span kind should be set to either `CONSUMER` or `SERVER` according to the rules defined above. From 93bd69b31b8025aea54fe1d73ce262f822398901 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Wed, 22 Jan 2020 16:34:28 +0100 Subject: [PATCH 04/34] Remove WIP marker --- specification/data-messaging.md | 2 -- 1 file changed, 2 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 34a3968731f..5f67111ccf9 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -1,7 +1,5 @@ # Messaging systems -> 🚧 WORK IN PROGRESS - From 2a1791ab0b70d63c68666c1dca22fbf3b97fa6bd Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Thu, 23 Jan 2020 10:50:29 +0100 Subject: [PATCH 05/34] dst -> destination --- specification/data-messaging.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 5f67111ccf9..52308953994 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -56,9 +56,9 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | -------------- | ---------------------------------------------------------------------- | --------- | | `component` | Denotes the type of the span and needs to be `"messaging"`. | Yes | | `messaging.flavor` | A string identifying the messaging system kind used. Should be of the form `KIND.TECH`, e.g. `JMS.Web Sphere` or `AMQP.RabbitMQ`. | Yes | -| `messaging.dst` | The message destination name, e.g. `"MyQueue"` or `"MyTopic"`. | Yes | -| `messaging.dst_kind` | The kind of message destination: Either `"queue"` or `"topic"`. | Yes | -| `messaging.tmp_dst` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | +| `messaging.destination` | The message destination name, e.g. `"MyQueue"` or `"MyTopic"`. | Yes | +| `messaging.destination_kind` | The kind of message destination: Either `"queue"` or `"topic"`. | Yes | +| `messaging.temp_destination` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | | `messaging.message_id` | An integer or string used by the messaging system as an identifier for the message. | No | | `messaging.conversation_id` | An integer or string identifying the conversation to which the message belongs. Sometimes called "correlation ID". | No | From c8d2c9699ec6a9bb3f2671072e752f212ecb542d Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Thu, 23 Jan 2020 10:51:47 +0100 Subject: [PATCH 06/34] Require the messaging system's IP or hostname --- specification/data-messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 52308953994..7359492d099 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -62,7 +62,7 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | `messaging.message_id` | An integer or string used by the messaging system as an identifier for the message. | No | | `messaging.conversation_id` | An integer or string identifying the conversation to which the message belongs. Sometimes called "correlation ID". | No | -It is strongly recommended to also set at least the [network attributes][] `net.peer.ip`, `net.peer.name` and `net.peer.port`. +Additionally at least one of `net.peer.name` or `net.peer.ip` from the [network attributes][] is required and `net.peer.port` is recommended. [network attributes]: data-span-general.md#general-network-connection-attributes From 0975ab2251d9b63222aebaedb9443aa6c717de60 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Thu, 23 Jan 2020 11:21:27 +0100 Subject: [PATCH 07/34] typo --- specification/data-messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 7359492d099..66e7b8f0c4c 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -71,7 +71,7 @@ For message consumers, the following additional attributes may be set: | Attribute name | Notes and examples | Required? | | -------------- | ---------------------------------------------------------------------- | --------- | | `messaging.operation` | A string identifying which part and kind of message consumption this span describes: `receive`, `peek` or `process`. (If the operation is `send`, this attribute must not be set: the operation can be inferred from the span kind in that case.) | No | -| `messaging.peeked_some` | A boolean indicating whether a `peek` operation resulted in a received message or not. Must only be added if `messaging.operation` is `peek`. I missing, assumed to be `true` if `messaging.message_id` is set, otherwise assumed to be `false`. | No | +| `messaging.peeked_some` | A boolean indicating whether a `peek` operation resulted in a received message or not. Must only be added if `messaging.operation` is `peek`. If missing, assumed to be `true` if `messaging.message_id` is set, otherwise assumed to be `false`. | No | Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.op` = `receive` or `peek`. Even though in that case one might think that the span kind is `INTERNAL`, that kind MUST NOT be used. From 8d1788d2a37faf3311bf05fe3bd100f4b082dd7c Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Thu, 23 Jan 2020 11:22:10 +0100 Subject: [PATCH 08/34] Add messaging to list of span conventions --- specification/trace/semantic_conventions/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specification/trace/semantic_conventions/README.md b/specification/trace/semantic_conventions/README.md index 3ab706793cf..14bf5c06631 100644 --- a/specification/trace/semantic_conventions/README.md +++ b/specification/trace/semantic_conventions/README.md @@ -16,4 +16,5 @@ The following semantic conventions for spans are defined: * [HTTP](http.md): Spans for HTTP client and server. * [Database](database.md): Spans for SQL and NoSQL client calls. * [RPC/RMI](rpc.md): Spans for remote procedure calls (e.g., gRPC). +* [Messaging](../data-messaging.md): Spans for interaction with messaging systems (queues, publish/subscribe, etc.). * [General](span-general.md): General semantic attributes that may be used in describing different kinds of operations. From 8b41bc76affc53b6327440ee07b8dcaf37355abf Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Thu, 23 Jan 2020 12:22:08 +0100 Subject: [PATCH 09/34] Fix op -> operation --- specification/data-messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 66e7b8f0c4c..7e06f04c6da 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -73,6 +73,6 @@ For message consumers, the following additional attributes may be set: | `messaging.operation` | A string identifying which part and kind of message consumption this span describes: `receive`, `peek` or `process`. (If the operation is `send`, this attribute must not be set: the operation can be inferred from the span kind in that case.) | No | | `messaging.peeked_some` | A boolean indicating whether a `peek` operation resulted in a received message or not. Must only be added if `messaging.operation` is `peek`. If missing, assumed to be `true` if `messaging.message_id` is set, otherwise assumed to be `false`. | No | -Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.op` = `receive` or `peek`. +Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.operation` = `receive` or `peek`. Even though in that case one might think that the span kind is `INTERNAL`, that kind MUST NOT be used. Instead span kind should be set to either `CONSUMER` or `SERVER` according to the rules defined above. From b97bf6706d18a281b8ee6cdae3d826ed7925195f Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Thu, 23 Jan 2020 12:31:49 +0100 Subject: [PATCH 10/34] Remove peek operation --- specification/data-messaging.md | 8 ++------ 1 file changed, 2 insertions(+), 6 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 7e06f04c6da..48d4a1d3167 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -25,9 +25,6 @@ A message submitted to a queue is processed by a message *consumer* (usually exa The consumption of a message can happen in multiple steps: First, the lower-level receiving of a message at a consumer, and then the logical processing of the message. -For receiving, the consumer may have the choice between peeking a destination to get a message back only if one is currently available -(for the purpose of this document, it is not relevant whether the message is removed from the destination or not by the "peek" operation), -or waiting until a message is received. Often, the waiting for a message is not particularly interesting and hidden away in a framework that only invokes some handler function to process a message once one is received (in the same way that the listening on a TCP port for an incoming HTTP message is not particularly interesting). However, in a synchronous conversation, the wait time for a message is important. @@ -70,9 +67,8 @@ For message consumers, the following additional attributes may be set: | Attribute name | Notes and examples | Required? | | -------------- | ---------------------------------------------------------------------- | --------- | -| `messaging.operation` | A string identifying which part and kind of message consumption this span describes: `receive`, `peek` or `process`. (If the operation is `send`, this attribute must not be set: the operation can be inferred from the span kind in that case.) | No | -| `messaging.peeked_some` | A boolean indicating whether a `peek` operation resulted in a received message or not. Must only be added if `messaging.operation` is `peek`. If missing, assumed to be `true` if `messaging.message_id` is set, otherwise assumed to be `false`. | No | +| `messaging.operation` | A string identifying which part and kind of message consumption this span describes: either `receive` or `process`. (If the operation is `send`, this attribute must not be set: the operation can be inferred from the span kind in that case.) | No | -Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.operation` = `receive` or `peek`. +Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.operation` = `receive`. Even though in that case one might think that the span kind is `INTERNAL`, that kind MUST NOT be used. Instead span kind should be set to either `CONSUMER` or `SERVER` according to the rules defined above. From 9491f14726ff05d1732caa322cedd7f5bf3da91c Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Thu, 23 Jan 2020 12:53:44 +0100 Subject: [PATCH 11/34] Formatting --- specification/data-messaging.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 48d4a1d3167..aef232baaa8 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -51,10 +51,10 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | Attribute name | Notes and examples | Required? | | -------------- | ---------------------------------------------------------------------- | --------- | -| `component` | Denotes the type of the span and needs to be `"messaging"`. | Yes | +| `component` | Denotes the type of the span and needs to be `messaging`. | Yes | | `messaging.flavor` | A string identifying the messaging system kind used. Should be of the form `KIND.TECH`, e.g. `JMS.Web Sphere` or `AMQP.RabbitMQ`. | Yes | -| `messaging.destination` | The message destination name, e.g. `"MyQueue"` or `"MyTopic"`. | Yes | -| `messaging.destination_kind` | The kind of message destination: Either `"queue"` or `"topic"`. | Yes | +| `messaging.destination` | The message destination name, e.g. `MyQueue` or `MyTopic`. | Yes | +| `messaging.destination_kind` | The kind of message destination: Either `queue` or `topic`. | Yes | | `messaging.temp_destination` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | | `messaging.message_id` | An integer or string used by the messaging system as an identifier for the message. | No | | `messaging.conversation_id` | An integer or string identifying the conversation to which the message belongs. Sometimes called "correlation ID". | No | From 80727c4c8a092705d9bb3083b17c7914136e1de0 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Thu, 23 Jan 2020 12:55:15 +0100 Subject: [PATCH 12/34] Merge suggestions from #395 --- specification/data-messaging.md | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index aef232baaa8..ba5ddfa82be 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -52,10 +52,12 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | Attribute name | Notes and examples | Required? | | -------------- | ---------------------------------------------------------------------- | --------- | | `component` | Denotes the type of the span and needs to be `messaging`. | Yes | -| `messaging.flavor` | A string identifying the messaging system kind used. Should be of the form `KIND.TECH`, e.g. `JMS.Web Sphere` or `AMQP.RabbitMQ`. | Yes | +| `messaging.system` | A string identifying the messaging system vendor such as `kafka`, `rabbitmq` or `jms`. | Yes | | `messaging.destination` | The message destination name, e.g. `MyQueue` or `MyTopic`. | Yes | | `messaging.destination_kind` | The kind of message destination: Either `queue` or `topic`. | Yes | | `messaging.temp_destination` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | +| `messaging.protocol` | The transport protocol such as `AMQP` or `MQTT`. | No | +| `messaging.url` | Connection substring such as `tibjmsnaming://localhost:7222` or `https://queue.amazonaws.com/80398EXAMPLE/MyQueue`. | No | | `messaging.message_id` | An integer or string used by the messaging system as an identifier for the message. | No | | `messaging.conversation_id` | An integer or string identifying the conversation to which the message belongs. Sometimes called "correlation ID". | No | From d9764435f58d77a9b05bb75dfe9df3df847d532c Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Fri, 24 Jan 2020 15:29:16 +0100 Subject: [PATCH 13/34] Fix samples for messaging.system --- specification/data-messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index ba5ddfa82be..c6cfec18c5c 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -52,7 +52,7 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | Attribute name | Notes and examples | Required? | | -------------- | ---------------------------------------------------------------------- | --------- | | `component` | Denotes the type of the span and needs to be `messaging`. | Yes | -| `messaging.system` | A string identifying the messaging system vendor such as `kafka`, `rabbitmq` or `jms`. | Yes | +| `messaging.system` | A string identifying the messaging system vendor such as `kafka`, `rabbitmq` or `activemq`. | Yes | | `messaging.destination` | The message destination name, e.g. `MyQueue` or `MyTopic`. | Yes | | `messaging.destination_kind` | The kind of message destination: Either `queue` or `topic`. | Yes | | `messaging.temp_destination` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | From 9a7254abb8c47f34c5d8d0d42e3c7fa5d48639ce Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Wed, 29 Jan 2020 09:19:22 +0100 Subject: [PATCH 14/34] Remove `component` attribute as per #271 --- specification/data-messaging.md | 1 - 1 file changed, 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index c6cfec18c5c..c17f16ad109 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -51,7 +51,6 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | Attribute name | Notes and examples | Required? | | -------------- | ---------------------------------------------------------------------- | --------- | -| `component` | Denotes the type of the span and needs to be `messaging`. | Yes | | `messaging.system` | A string identifying the messaging system vendor such as `kafka`, `rabbitmq` or `activemq`. | Yes | | `messaging.destination` | The message destination name, e.g. `MyQueue` or `MyTopic`. | Yes | | `messaging.destination_kind` | The kind of message destination: Either `queue` or `topic`. | Yes | From d30a56d5cab0075d42a335b774ceccb8d99be25c Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 18 Feb 2020 14:35:33 +0100 Subject: [PATCH 15/34] Apply suggestions from code review Co-Authored-By: Sergey Kanzhelev --- specification/data-messaging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index c17f16ad109..052401faf5d 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -19,7 +19,7 @@ A *message* usually consists of headers (or properties, or meta information) and * Physically: some message *broker* (which can be e.g., a single server, or a cluster, or a local process reached via IPC). The broker handles the actual routing, delivery, re-delivery, persistence, etc. In some messaging systems the broker may be identical or co-located with (some) message consumers. * Logically: some particular message *destination*. -A destination is usually identified by some name unique within the messaging system instance, which might look like an URL or a simple one-word identifier. Two kinds of targets are are distinguished: *topic*s and *queue*s. +A destination is usually identified by some name unique within the messaging system instance, which might look like an URL or a simple one-word identifier. Two kinds of destinations are are distinguished: *topic*s and *queue*s. A message that is sent (the send-operation is often called "*publish*" in this context) to a *topic* is broadcasted to all *subscribers* of the topic. A message submitted to a queue is processed by a message *consumer* (usually exactly once although some message systems support a more performant at least once-mode for messages with [idempotent][] processing). @@ -29,7 +29,7 @@ Often, the waiting for a message is not particularly interesting and hidden away (in the same way that the listening on a TCP port for an incoming HTTP message is not particularly interesting). However, in a synchronous conversation, the wait time for a message is important. -In some messaging systems, a message can receive a reply message that answers a particular other message that was sent earlier. All messages that are grouped together by such a reply-relationship are called a *conversation*. The grouping usually happens through some sort of "In-Reply-To:" meta information or an explicit conversation ID. Sometimes a conversation can span multiple message targets (e.g. initiated via a topic, continued on a temporary one-to-one queue). +In some messaging systems, a message can receive a reply message that answers a particular other message that was sent earlier. All messages that are grouped together by such a reply-relationship are called a *conversation*. The grouping usually happens through some sort of "In-Reply-To:" meta information or an explicit conversation ID. Sometimes a conversation can span multiple message destinations (e.g. initiated via a topic, continued on a temporary one-to-one queue). Some messaging systems support the concept of *temporary destination* (often only temporary queues) that are established just for a particular set of communication partners (often one to one) or conversation. Often such destinations are unnamed or have an auto-generated name. From de0afba6e2834bac403d3d3b93d47b68f4a42448 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 18 Feb 2020 14:29:10 +0100 Subject: [PATCH 16/34] Define message_id as string --- specification/data-messaging.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 052401faf5d..5556f40c641 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -57,8 +57,8 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | `messaging.temp_destination` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | | `messaging.protocol` | The transport protocol such as `AMQP` or `MQTT`. | No | | `messaging.url` | Connection substring such as `tibjmsnaming://localhost:7222` or `https://queue.amazonaws.com/80398EXAMPLE/MyQueue`. | No | -| `messaging.message_id` | An integer or string used by the messaging system as an identifier for the message. | No | -| `messaging.conversation_id` | An integer or string identifying the conversation to which the message belongs. Sometimes called "correlation ID". | No | +| `messaging.message_id` | A value used by the messaging system as an identifier for the message, represented as a string. | No | +| `messaging.conversation_id` | A value identifying the conversation to which the message belongs, represented as a string. Sometimes called "Correlation ID". | No | Additionally at least one of `net.peer.name` or `net.peer.ip` from the [network attributes][] is required and `net.peer.port` is recommended. From a8b6e1b197baa7c5b693a0b874c1188ac5b059a2 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 18 Feb 2020 16:31:28 +0100 Subject: [PATCH 17/34] Typo --- specification/data-messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 5556f40c641..dfaa18bec89 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -21,7 +21,7 @@ A *message* usually consists of headers (or properties, or meta information) and A destination is usually identified by some name unique within the messaging system instance, which might look like an URL or a simple one-word identifier. Two kinds of destinations are are distinguished: *topic*s and *queue*s. A message that is sent (the send-operation is often called "*publish*" in this context) to a *topic* is broadcasted to all *subscribers* of the topic. -A message submitted to a queue is processed by a message *consumer* (usually exactly once although some message systems support a more performant at least once-mode for messages with [idempotent][] processing). +A message submitted to a queue is processed by a message *consumer* (usually exactly once although some message systems support a more performant at-least-once mode for messages with [idempotent][] processing). The consumption of a message can happen in multiple steps: First, the lower-level receiving of a message at a consumer, and then the logical processing of the message. From e8931def834d5aa13b1efc4b99bbcba35de969af Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 18 Feb 2020 16:33:29 +0100 Subject: [PATCH 18/34] Add guidance for span names for temp destinations --- specification/data-messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index dfaa18bec89..e310c82e6e9 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -41,7 +41,7 @@ Given these definitions, the remainder of this section describes the semantic co **Span name:** The span name should usually be set to the message destination name. The conversation ID should be used instead when it is expected to have lower cardinality. -In particular, the conversation ID must be used if the message destination is unnamed or temporary. +In particular, the conversation ID must be used if the message destination is unnamed or temporary unless multiple conversations can be combined to a logical destination of lower cardinality. **Span kind:** A producer of a message should set the span kind to `PRODUCER` unless it synchronously waits for a response: then it should use `CLIENT`. The processor of the message should set the kind to `CONSUMER`, unless it always sends back a reply that is directed to the producer of the message From 5da7c848ae7c24f2f4b300c770f6d242c230f370 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 18 Feb 2020 16:48:56 +0100 Subject: [PATCH 19/34] Add section on RabbitMQ --- specification/data-messaging.md | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index e310c82e6e9..f27eed7b583 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -73,3 +73,11 @@ For message consumers, the following additional attributes may be set: Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.operation` = `receive`. Even though in that case one might think that the span kind is `INTERNAL`, that kind MUST NOT be used. Instead span kind should be set to either `CONSUMER` or `SERVER` according to the rules defined above. + +### Attributes specific to certain messaging systems + +#### RabbitMQ + +In RabbitMQ, the destination is defined by an _exchange_ and a _routing key_. +`messaging.destination` MUST be set to the name of the exchange. This will be an empty string if the default exchange is used. +The routing key MUST be provided to the attribute `messaging.rabbitmq.routing_key`, unless it is empty. From f2ea69e80e2b41105ef30ecfa6e2408343b5f15b Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 18 Feb 2020 17:37:32 +0100 Subject: [PATCH 20/34] Add note that messaging.destination might be equal to span name --- specification/data-messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index f27eed7b583..838a1e9aa3f 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -52,7 +52,7 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | Attribute name | Notes and examples | Required? | | -------------- | ---------------------------------------------------------------------- | --------- | | `messaging.system` | A string identifying the messaging system vendor such as `kafka`, `rabbitmq` or `activemq`. | Yes | -| `messaging.destination` | The message destination name, e.g. `MyQueue` or `MyTopic`. | Yes | +| `messaging.destination` | The message destination name, e.g. `MyQueue` or `MyTopic`. This might be equal to the span name but is required nevertheless. | Yes | | `messaging.destination_kind` | The kind of message destination: Either `queue` or `topic`. | Yes | | `messaging.temp_destination` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | | `messaging.protocol` | The transport protocol such as `AMQP` or `MQTT`. | No | From cc4fb06aa2c41031554945fda4940c8f7d05771b Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 18 Feb 2020 17:43:12 +0100 Subject: [PATCH 21/34] Add note on receive/process timing --- specification/data-messaging.md | 1 + 1 file changed, 1 insertion(+) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 838a1e9aa3f..1d894fead6b 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -70,6 +70,7 @@ For message consumers, the following additional attributes may be set: | -------------- | ---------------------------------------------------------------------- | --------- | | `messaging.operation` | A string identifying which part and kind of message consumption this span describes: either `receive` or `process`. (If the operation is `send`, this attribute must not be set: the operation can be inferred from the span kind in that case.) | No | +The _receive_ span is be used to track the time used for receiving the message(s), whereas the _process_ span(s) track the time for processing the message(s). Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.operation` = `receive`. Even though in that case one might think that the span kind is `INTERNAL`, that kind MUST NOT be used. Instead span kind should be set to either `CONSUMER` or `SERVER` according to the rules defined above. From daf1874af3bf3622248d6ade373002e595abbbbf Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 18 Feb 2020 17:45:03 +0100 Subject: [PATCH 22/34] =?UTF-8?q?Remove=20colon=20=F0=9F=91=A8=E2=80=8D?= =?UTF-8?q?=E2=9A=95=EF=B8=8F?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit --- specification/data-messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 1d894fead6b..110f970df7b 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -23,7 +23,7 @@ A destination is usually identified by some name unique within the messaging sys A message that is sent (the send-operation is often called "*publish*" in this context) to a *topic* is broadcasted to all *subscribers* of the topic. A message submitted to a queue is processed by a message *consumer* (usually exactly once although some message systems support a more performant at-least-once mode for messages with [idempotent][] processing). -The consumption of a message can happen in multiple steps: +The consumption of a message can happen in multiple steps. First, the lower-level receiving of a message at a consumer, and then the logical processing of the message. Often, the waiting for a message is not particularly interesting and hidden away in a framework that only invokes some handler function to process a message once one is received (in the same way that the listening on a TCP port for an incoming HTTP message is not particularly interesting). From a9b82097106a7b35542876881a4b552c36d55084 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Wed, 19 Feb 2020 17:41:30 +0100 Subject: [PATCH 23/34] Add examples --- specification/data-messaging.md | 58 +++++++++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 110f970df7b..b3770acec01 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -7,6 +7,7 @@ - [Definitions](#definitions) - [Conventions](#conventions) - [Messaging attributes](#messaging-attributes) +- [Examples](#examples) @@ -82,3 +83,60 @@ Instead span kind should be set to either `CONSUMER` or `SERVER` according to th In RabbitMQ, the destination is defined by an _exchange_ and a _routing key_. `messaging.destination` MUST be set to the name of the exchange. This will be an empty string if the default exchange is used. The routing key MUST be provided to the attribute `messaging.rabbitmq.routing_key`, unless it is empty. + +## Examples + +### Topic with multiple consumers + +Given is a process P, that publishes a message to a topic T on messaging system MS, and two processes CA and CB, which both receive the message and process it. + +``` +Process P: | Span P1 | +-- +Process CA: | Span CA1 | +-- +Process CB: | Span CB1 | +``` + +| Field or Attribute | Span P1 | Span CA1 | Span CB1 | +|-|-|-|-|-|-| +| Name | `"T"` | `"T"` | `"T"` | +| Parent | | Span P1 | Span P1 | +| Links | | | | +| SpanKind | `PRODUCER` | `CONSUMER` | `CONSUMER` | +| Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | +| `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | +| `net.peer.port` | `1234` | `1234` | `1234` | `1234` | `1234` | +| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | +| `messaging.destination` | `"T"` | `"T"` | `"T"` | +| `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | +| `messaging.operation` | | `"process"` | `"process"` | +| `messaging.message_id` | `"a1"` | `"a1"`| `"a1"` | + +### Batch consumer + +Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them in one batch (Span C1) and processes each message separately (Spans C2 and C3). +Since a span can only have one parent and the propagated trace and span IDs are not known when the receiving span is started, the receiving span will have no parent and the processing spans are correlated with the producing spans using links. + +``` +Process P: | Span P1 | Span P2 | +-- +Process C: | Span C1 | + | Span C2 | + | Span C3 | +``` + +| Field or Attribute | Span P1 | Span P2 | Span C1 | Span C2 | Span C3 | +|-|-|-|-|-|-| +| Name | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | +| Parent | | | | Span C1 | Span C1 | +| Links | | | | Span P1 | Span P2 | +| SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | `CONSUMER` | `CONSUMER` | +| Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | +| `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | +| `net.peer.port` | `1234` | `1234` | `1234` | `1234` | `1234` | +| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | +| `messaging.destination` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | +| `messaging.destination_kind` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | +| `messaging.operation` | | | `"receive"` | `"process"` | `"process"` | +| `messaging.message_id` | `"a1"` | `"a2"` | | `"a1"` | `"a2"` | From bb968b74dfd2b4140d612b781c0f953dfec839c1 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Wed, 19 Feb 2020 17:43:21 +0100 Subject: [PATCH 24/34] Make markdownlint happy --- specification/data-messaging.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index b3770acec01..d505a291805 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -115,7 +115,8 @@ Process CB: | Span CB1 | ### Batch consumer -Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them in one batch (Span C1) and processes each message separately (Spans C2 and C3). +Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them in one batch (Span C1) and processes each message separately (Spans C2 and C3). + Since a span can only have one parent and the propagated trace and span IDs are not known when the receiving span is started, the receiving span will have no parent and the processing spans are correlated with the producing spans using links. ``` From 4613bd1c4132c93c3e9ea2ff009c76ba2a235017 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Thu, 20 Feb 2020 12:49:02 +0100 Subject: [PATCH 25/34] Clarify processing span kind --- specification/data-messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index d505a291805..3dd07e8b7f8 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -73,7 +73,7 @@ For message consumers, the following additional attributes may be set: The _receive_ span is be used to track the time used for receiving the message(s), whereas the _process_ span(s) track the time for processing the message(s). Note that one or multiple Spans with `messaging.operation` = `process` may often be the children of a Span with `messaging.operation` = `receive`. -Even though in that case one might think that the span kind is `INTERNAL`, that kind MUST NOT be used. +Even though in that case one might think that the processing span's kind should be `INTERNAL`, that kind MUST NOT be used. Instead span kind should be set to either `CONSUMER` or `SERVER` according to the rules defined above. ### Attributes specific to certain messaging systems From e00d1e888c25526c562b89873c505c5315e17563 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Mon, 24 Feb 2020 11:27:06 +0100 Subject: [PATCH 26/34] Fix table in example --- specification/data-messaging.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 3dd07e8b7f8..941790f6648 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -99,15 +99,15 @@ Process CB: | Span CB1 | ``` | Field or Attribute | Span P1 | Span CA1 | Span CB1 | -|-|-|-|-|-|-| +|-|-|-|-| | Name | `"T"` | `"T"` | `"T"` | | Parent | | Span P1 | Span P1 | | Links | | | | | SpanKind | `PRODUCER` | `CONSUMER` | `CONSUMER` | -| Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | -| `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | -| `net.peer.port` | `1234` | `1234` | `1234` | `1234` | `1234` | -| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | +| Status | `Ok` | `Ok` | `Ok` | +| `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | +| `net.peer.port` | `1234` | `1234` | `1234` | +| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | | `messaging.destination` | `"T"` | `"T"` | `"T"` | | `messaging.destination_kind` | `"topic"` | `"topic"` | `"topic"` | | `messaging.operation` | | `"process"` | `"process"` | From d73aa5ea0b71c9bbcfb3d54256e3d012c98dc648 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Mon, 24 Feb 2020 11:27:19 +0100 Subject: [PATCH 27/34] Make destination_kind optional --- specification/data-messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 941790f6648..0d77c3765ee 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -54,7 +54,7 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | -------------- | ---------------------------------------------------------------------- | --------- | | `messaging.system` | A string identifying the messaging system vendor such as `kafka`, `rabbitmq` or `activemq`. | Yes | | `messaging.destination` | The message destination name, e.g. `MyQueue` or `MyTopic`. This might be equal to the span name but is required nevertheless. | Yes | -| `messaging.destination_kind` | The kind of message destination: Either `queue` or `topic`. | Yes | +| `messaging.destination_kind` | The kind of message destination: Either `queue` or `topic`. | Yes, if either of them applies. | | `messaging.temp_destination` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | | `messaging.protocol` | The transport protocol such as `AMQP` or `MQTT`. | No | | `messaging.url` | Connection substring such as `tibjmsnaming://localhost:7222` or `https://queue.amazonaws.com/80398EXAMPLE/MyQueue`. | No | From 0be1d95f2866f9f0760f81f238fd5ea6d251d243 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Mon, 24 Feb 2020 18:09:49 +0100 Subject: [PATCH 28/34] Add batch processor example --- specification/data-messaging.md | 37 +++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 0d77c3765ee..291bb9e71bf 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -113,7 +113,7 @@ Process CB: | Span CB1 | | `messaging.operation` | | `"process"` | `"process"` | | `messaging.message_id` | `"a1"` | `"a1"`| `"a1"` | -### Batch consumer +### Batch receiving Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them in one batch (Span C1) and processes each message separately (Spans C2 and C3). @@ -124,7 +124,7 @@ Process P: | Span P1 | Span P2 | -- Process C: | Span C1 | | Span C2 | - | Span C3 | + | Span C3 | ``` | Field or Attribute | Span P1 | Span P2 | Span C1 | Span C2 | Span C3 | @@ -141,3 +141,36 @@ Process C: | Span C1 | | `messaging.destination_kind` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | | `messaging.operation` | | | `"receive"` | `"process"` | `"process"` | | `messaging.message_id` | `"a1"` | `"a2"` | | `"a1"` | `"a2"` | + +### Batch processing + +Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them separately (Span C1 and C2) and processes both messages in one batch (Span C3). + +Since each span can only have one parent, C3 should not choose a random parent out of C1 and C2, but rather rely on the implicitly selected parent as defined by the [tracing API spec](api-tracing.md). +Similarly, only one value can be set as `message_id`, so C3 cannot report both `a1` and `a2` and therefore attribute is left out. +Depending on the implementation, the producing spans might still be available in the meta data of the messages and should be added to C3 as links. +The client library or application could also add the receiver span's span context to the data structure it returns for each message. In this case, C3 could also add links to the receiver spans C1 and C2. + +The status of the batch processing span is selected by the application. Depending on the semantics of the operation. A span status `Ok` could, for example, be set only if all messages or if just at least one were properly processed. + +``` +Process P: | Span P1 | Span P2 | +-- +Process C: | Span C1 | Span C2 | + | Span C3 | +``` + +| Field or Attribute | Span P1 | Span P2 | Span C1 | Span C2 | Span C3 | +|-|-|-|-|-|-| +| Name | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | +| Parent | | | Span P1 | Span P2 | | +| Links | | | | | Span P1 + P2 | +| SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | `CONSUMER` | `CONSUMER` | +| Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | +| `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | +| `net.peer.port` | `1234` | `1234` | `1234` | `1234` | `1234` | +| `messaging.system` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | `"kafka"` | +| `messaging.destination` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | +| `messaging.destination_kind` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | `"queue"` | +| `messaging.operation` | | | `"receive"` | `"receive"` | `"process"` | +| `messaging.message_id` | `"a1"` | `"a2"` | `"a1"` | `"a2"` | | From 6355871affd86c1e8537f5a055d66e6345b0e5fc Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 25 Feb 2020 11:38:37 +0100 Subject: [PATCH 29/34] Apply suggestions from code review MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Christian Neumüller --- specification/data-messaging.md | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 291bb9e71bf..fc42e077bd4 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -20,7 +20,8 @@ A *message* usually consists of headers (or properties, or meta information) and * Physically: some message *broker* (which can be e.g., a single server, or a cluster, or a local process reached via IPC). The broker handles the actual routing, delivery, re-delivery, persistence, etc. In some messaging systems the broker may be identical or co-located with (some) message consumers. * Logically: some particular message *destination*. -A destination is usually identified by some name unique within the messaging system instance, which might look like an URL or a simple one-word identifier. Two kinds of destinations are are distinguished: *topic*s and *queue*s. +A destination is usually identified by some name unique within the messaging system instance, which might look like an URL or a simple one-word identifier. +Two kinds of destinations are distinguished: *topic*s and *queue*s. A message that is sent (the send-operation is often called "*publish*" in this context) to a *topic* is broadcasted to all *subscribers* of the topic. A message submitted to a queue is processed by a message *consumer* (usually exactly once although some message systems support a more performant at-least-once mode for messages with [idempotent][] processing). @@ -57,11 +58,12 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | `messaging.destination_kind` | The kind of message destination: Either `queue` or `topic`. | Yes, if either of them applies. | | `messaging.temp_destination` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | | `messaging.protocol` | The transport protocol such as `AMQP` or `MQTT`. | No | -| `messaging.url` | Connection substring such as `tibjmsnaming://localhost:7222` or `https://queue.amazonaws.com/80398EXAMPLE/MyQueue`. | No | +| `messaging.url` | Connection string such as `tibjmsnaming://localhost:7222` or `https://queue.amazonaws.com/80398EXAMPLE/MyQueue`. | No | | `messaging.message_id` | A value used by the messaging system as an identifier for the message, represented as a string. | No | | `messaging.conversation_id` | A value identifying the conversation to which the message belongs, represented as a string. Sometimes called "Correlation ID". | No | Additionally at least one of `net.peer.name` or `net.peer.ip` from the [network attributes][] is required and `net.peer.port` is recommended. +These attributes should be set to the broker to which the message is sent/from which it is received. [network attributes]: data-span-general.md#general-network-connection-attributes From 713b78c90b63f2a87df08cf7f35d74c6aaf1e1f3 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 25 Feb 2020 13:17:23 +0100 Subject: [PATCH 30/34] Rename spans in examples --- specification/data-messaging.md | 36 ++++++++++++++++----------------- 1 file changed, 18 insertions(+), 18 deletions(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index fc42e077bd4..0df8d3bcd6a 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -93,17 +93,17 @@ The routing key MUST be provided to the attribute `messaging.rabbitmq.routing_ke Given is a process P, that publishes a message to a topic T on messaging system MS, and two processes CA and CB, which both receive the message and process it. ``` -Process P: | Span P1 | +Process P: | Span Prod1 | -- Process CA: | Span CA1 | -- Process CB: | Span CB1 | ``` -| Field or Attribute | Span P1 | Span CA1 | Span CB1 | +| Field or Attribute | Span Prod1 | Span CA1 | Span CB1 | |-|-|-|-| | Name | `"T"` | `"T"` | `"T"` | -| Parent | | Span P1 | Span P1 | +| Parent | | Span Prod1 | Span Prod1 | | Links | | | | | SpanKind | `PRODUCER` | `CONSUMER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | @@ -117,23 +117,23 @@ Process CB: | Span CB1 | ### Batch receiving -Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them in one batch (Span C1) and processes each message separately (Spans C2 and C3). +Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them in one batch (Span Recv1) and processes each message separately (Spans Proc1 and Proc2). Since a span can only have one parent and the propagated trace and span IDs are not known when the receiving span is started, the receiving span will have no parent and the processing spans are correlated with the producing spans using links. ``` -Process P: | Span P1 | Span P2 | +Process P: | Span Prod1 | Span Prod2 | -- -Process C: | Span C1 | - | Span C2 | - | Span C3 | +Process C: | Span Recv1 | + | Span Proc1 | + | Span Proc2 | ``` -| Field or Attribute | Span P1 | Span P2 | Span C1 | Span C2 | Span C3 | +| Field or Attribute | Span Prod1 | Span Prod2 | Span Recv1 | Span Proc1 | Span Proc2 | |-|-|-|-|-|-| | Name | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | -| Parent | | | | Span C1 | Span C1 | -| Links | | | | Span P1 | Span P2 | +| Parent | | | | Span Recv1 | Span Recv1 | +| Links | | | | Span Prod1 | Span Prod2 | | SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | `CONSUMER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | | `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | @@ -146,7 +146,7 @@ Process C: | Span C1 | ### Batch processing -Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them separately (Span C1 and C2) and processes both messages in one batch (Span C3). +Given is a process P, that sends two messages to a queue Q on messaging system MS, and a process C, which receives both of them separately (Span Recv1 and Recv2) and processes both messages in one batch (Span Proc1). Since each span can only have one parent, C3 should not choose a random parent out of C1 and C2, but rather rely on the implicitly selected parent as defined by the [tracing API spec](api-tracing.md). Similarly, only one value can be set as `message_id`, so C3 cannot report both `a1` and `a2` and therefore attribute is left out. @@ -156,17 +156,17 @@ The client library or application could also add the receiver span's span contex The status of the batch processing span is selected by the application. Depending on the semantics of the operation. A span status `Ok` could, for example, be set only if all messages or if just at least one were properly processed. ``` -Process P: | Span P1 | Span P2 | +Process P: | Span Prod1 | Span Prod2 | -- -Process C: | Span C1 | Span C2 | - | Span C3 | +Process C: | Span Recv1 | Span Recv2 | + | Span Proc1 | ``` -| Field or Attribute | Span P1 | Span P2 | Span C1 | Span C2 | Span C3 | +| Field or Attribute | Span Prod1 | Span Prod2 | Span Recv1 | Span Recv2 | Span Proc1 | |-|-|-|-|-|-| | Name | `"Q"` | `"Q"` | `"Q"` | `"Q"` | `"Q"` | -| Parent | | | Span P1 | Span P2 | | -| Links | | | | | Span P1 + P2 | +| Parent | | | Span Prod1 | Span Prod2 | | +| Links | | | | | Span Prod1 + Prod2 | | SpanKind | `PRODUCER` | `PRODUCER` | `CONSUMER` | `CONSUMER` | `CONSUMER` | | Status | `Ok` | `Ok` | `Ok` | `Ok` | `Ok` | | `net.peer.name` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | `"ms"` | From feeb2c59c9024b602387e44dc3242af665648553 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 17 Mar 2020 15:14:32 +0100 Subject: [PATCH 31/34] Declare messaging.protocol to be without version --- specification/data-messaging.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 0df8d3bcd6a..7f21ade0ab3 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -57,7 +57,7 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | `messaging.destination` | The message destination name, e.g. `MyQueue` or `MyTopic`. This might be equal to the span name but is required nevertheless. | Yes | | `messaging.destination_kind` | The kind of message destination: Either `queue` or `topic`. | Yes, if either of them applies. | | `messaging.temp_destination` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | -| `messaging.protocol` | The transport protocol such as `AMQP` or `MQTT`. | No | +| `messaging.protocol` | The transport protocol name (without version), such as `AMQP` or `MQTT`. | No | | `messaging.url` | Connection string such as `tibjmsnaming://localhost:7222` or `https://queue.amazonaws.com/80398EXAMPLE/MyQueue`. | No | | `messaging.message_id` | A value used by the messaging system as an identifier for the message, represented as a string. | No | | `messaging.conversation_id` | A value identifying the conversation to which the message belongs, represented as a string. Sometimes called "Correlation ID". | No | From c63901234ebaed42d67839b629d90f788f164b9d Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 17 Mar 2020 15:18:50 +0100 Subject: [PATCH 32/34] Add `messaging.protocol_version` --- specification/data-messaging.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index 7f21ade0ab3..a7a52d7785a 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -57,7 +57,8 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | `messaging.destination` | The message destination name, e.g. `MyQueue` or `MyTopic`. This might be equal to the span name but is required nevertheless. | Yes | | `messaging.destination_kind` | The kind of message destination: Either `queue` or `topic`. | Yes, if either of them applies. | | `messaging.temp_destination` | A boolean that is `true` if the message destination is temporary. | If temporary (assumed to be `false` if missing). | -| `messaging.protocol` | The transport protocol name (without version), such as `AMQP` or `MQTT`. | No | +| `messaging.protocol` | The name of the transport protocol such as `AMQP` or `MQTT`. | No | +| `messaging.protocol_version` | The version of the transport protocol such as `0.9.1`. | No | | `messaging.url` | Connection string such as `tibjmsnaming://localhost:7222` or `https://queue.amazonaws.com/80398EXAMPLE/MyQueue`. | No | | `messaging.message_id` | A value used by the messaging system as an identifier for the message, represented as a string. | No | | `messaging.conversation_id` | A value identifying the conversation to which the message belongs, represented as a string. Sometimes called "Correlation ID". | No | From 92748abcd0c422889c0b660fe1bf2a864214473e Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Wed, 1 Apr 2020 18:22:56 +0200 Subject: [PATCH 33/34] Recommend setting `net.transport` --- specification/data-messaging.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/specification/data-messaging.md b/specification/data-messaging.md index a7a52d7785a..88a7494947c 100644 --- a/specification/data-messaging.md +++ b/specification/data-messaging.md @@ -64,9 +64,12 @@ The processor of the message should set the kind to `CONSUMER`, unless it always | `messaging.conversation_id` | A value identifying the conversation to which the message belongs, represented as a string. Sometimes called "Correlation ID". | No | Additionally at least one of `net.peer.name` or `net.peer.ip` from the [network attributes][] is required and `net.peer.port` is recommended. +Furthermore, it is strongly recommended to add the [`net.transport`][] attribute and follow its guidelines, especially for in-process queueing systems (like [Hangfire][], for example). These attributes should be set to the broker to which the message is sent/from which it is received. [network attributes]: data-span-general.md#general-network-connection-attributes +[`net.transport`]: data-span-general.md#nettransport-attribute +[Hangfire]: https://www.hangfire.io/ For message consumers, the following additional attributes may be set: From 1cf72e59134ed82f9a74b331a17f7fcbfe797c44 Mon Sep 17 00:00:00 2001 From: Armin Ruech Date: Tue, 7 Apr 2020 09:01:34 +0200 Subject: [PATCH 34/34] Move file after directories were restructured upstream --- specification/trace/semantic_conventions/README.md | 2 +- .../semantic_conventions/messaging.md} | 0 2 files changed, 1 insertion(+), 1 deletion(-) rename specification/{data-messaging.md => trace/semantic_conventions/messaging.md} (100%) diff --git a/specification/trace/semantic_conventions/README.md b/specification/trace/semantic_conventions/README.md index 14bf5c06631..312b0f008e4 100644 --- a/specification/trace/semantic_conventions/README.md +++ b/specification/trace/semantic_conventions/README.md @@ -16,5 +16,5 @@ The following semantic conventions for spans are defined: * [HTTP](http.md): Spans for HTTP client and server. * [Database](database.md): Spans for SQL and NoSQL client calls. * [RPC/RMI](rpc.md): Spans for remote procedure calls (e.g., gRPC). -* [Messaging](../data-messaging.md): Spans for interaction with messaging systems (queues, publish/subscribe, etc.). +* [Messaging](messaging.md): Spans for interaction with messaging systems (queues, publish/subscribe, etc.). * [General](span-general.md): General semantic attributes that may be used in describing different kinds of operations. diff --git a/specification/data-messaging.md b/specification/trace/semantic_conventions/messaging.md similarity index 100% rename from specification/data-messaging.md rename to specification/trace/semantic_conventions/messaging.md