From 7c80a1f55b27e6e344ebcd528bc2c5aae6b7a00f Mon Sep 17 00:00:00 2001 From: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> Date: Fri, 13 Dec 2024 14:37:34 +0100 Subject: [PATCH 01/12] ipc: first process test Issue-ref: see #69 --- docs/features/communication/ipc/index.rst | 2 ++ 1 file changed, 2 insertions(+) diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index ce7b68d1..c45d9358 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -14,3 +14,5 @@ Inter-process Communication ########################### + + From 3956575945226bc0b7166b33727e0a46886f7ced Mon Sep 17 00:00:00 2001 From: Nico Hartmann Date: Fri, 13 Dec 2024 18:01:11 +0100 Subject: [PATCH 02/12] ipc: First thoughts on data types, raw structure Issue-ref: see #69 --- docs/features/communication/ipc/index.rst | 169 ++++++++++++++++++++++ 1 file changed, 169 insertions(+) diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index c45d9358..fd732506 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -15,4 +15,173 @@ Inter-process Communication ########################### +Inter-process Communication (IPC) Framework handles the safe, secure and performant exchange of information between software and/or hardware components of the Score stack. +The IPC framework is independent of the application context it is used in. + +There are two fundamental information exchange concepts defined + +* Topics: A topic is a data entity identifyed by a name, being structured by a *data type* and containing zero to multiple instances of that type (a *value*). +* Remote Procedures: A remote procedure is an invocable entity identifyed by a name, being structured by parameters of a *data type* and being activated by an invokation passing arguments, i.e. a value for each parameter. + +Data Types +---------- + +Data types describe the inner structure of data entities known as values. + +Primitive Types +``````````````` + +Primitive data types consist of a single element with no further inherent structure. +The following primitive data types and their Rust and C++ representation are defined: + +========= ========== ========= ========================== ====== ============================ +Data Type Class Rust C++ Size Description +========= ========== ========= ========================== ====== ============================ +Bool Boolean ``bool`` ``bool`` 1 A boolean value, true or false +Int8 Integer ``i8`` ``int8_t`` 1 An 8 bit signed integer +UInt8 Integer ``u8`` ``uint8_t`` 1 An 8 bit unsigned integer +UInt128 Integer ``u128`` ``uint128_t`` 16 An 128 bit unsigned integer +Float16 Floating ``f16`` ``float16_t`` 2 An IEEE 756 32 bit floating point +Float32 Floating ``f32`` ``float``, ``float32_t`` 4 An IEEE 756 32 bit floating point +Float64 Floating ``f64`` ``double``, ``float64_t`` 8 An IEEE 756 64 bit floating point +BFloat16 Floating ``bf16`` ``bfloat16_t`` 2 A Google brain float 16 floating point +Char String ``char`` ``char32_t`` 4 A unicode codepoint (32 bit) +String String ``str`` - n/a A UTF-8 encoded text +========= ========== ========= ========================== ====== ============================ + +The type ``Byte`` may be used as alias for ``UInt8``. + +| Pointers are no recognized primitive types. +| References are no recognized primitive types. + +Tuples +`````` + +Tuples are ordered collections of arbitrary data types. A tuple shall be expressed by parentheses: +``(Int8, Float32)``. + +Structs +``````` + +A struct is an ordered collection of named arbitrary data types called fields: + +:: + + struct MyStruct { + field1: Int8, + field2: Float32 + } + +Arrays +`````` + +Arrays are ordered collections of data elements of the same type with a fixed length. A single element is addressed by an *index*: + +:: + + Char[32] + + +Tensors +``````` + +A Tensor is a multi-dimensional array of numerical values that generalizes scalars, vectors, and matrices to higher dimensions, commonly used in mathematics, physics, and machine learning. + +The number of dimensions is called the *rank* or *order* of the Tensor. +The vector of dimensions with the Tensor's rank is called the *shape* of the Tensor. + +:: + + Tensor + +The 5x5 kernel of a CNN layer with 128 features. + +List +```` + +A List is an ordered collection of data elements of the same type with a variable length. A single element is addressed by an *index*: + +:: + + List + +HashMap +``````` + +A HashMap is an unordered collection of data element of the same type with variable length. A single element is addressed by a *key* of a specific data type. + +:: + + HashMap + + +Topic Reference +``````````````` + +A topic reference is a data type that provides a resolvable reference to a Topic. + +:: + + TopicRef + +The underlying implementation shall not use more than 64 bits for the representation of TopicRef. + +All properties of the referenced topics shall be retrievable through the TopicRef. + +Data access to the content of the Topic shall be possible through the TopicRef. + + +RP Reference +````````````` + +An RP Reference is a data type that provides a resolvable reference to a Remote Procedure. + +:: + + RPRef + +The underlying implementation shall not use more than 64 bits for the representation of RPRef. + +All properties of the referenced Remote Procedure shall be retrievable through the RPRef. + +Invocation of the referenced Remote Procedure shall be possible through the RPRef. + + +Namespaces +---------- + + +Topics +------ + +* Name +* Data Type +* Queue Depth +* Initialization +* Publisher +* Subscriber + + +Remote Procedures +----------------- + +* Name +* Signature, Parameter Pack +* Publishing +* Discovery +* Invocation +* Sync/Async + + +Zero Copy +--------- + +* Definition +* Shared Memory +* Memory Management +* DMA + + +Safety +------ From f87b6902d9a5c71e7e592f5121ffb64e09e536f6 Mon Sep 17 00:00:00 2001 From: Nico Hartmann Date: Sun, 15 Dec 2024 09:17:15 +0100 Subject: [PATCH 03/12] ipc: added Event --- .env | 2 + .vscode/settings.json | 14 ++++ docs/conf.py | 78 ++++++++++++++++++++++- docs/doc8.ini | 2 + docs/features/communication/ipc/index.rst | 15 ++++- 5 files changed, 107 insertions(+), 4 deletions(-) create mode 100644 .env create mode 100644 docs/doc8.ini diff --git a/.env b/.env new file mode 100644 index 00000000..a98ee355 --- /dev/null +++ b/.env @@ -0,0 +1,2 @@ +PYTHONPATH=${workspaceFolder}/_extensions + diff --git a/.vscode/settings.json b/.vscode/settings.json index 92c92ecf..abba7d2c 100644 --- a/.vscode/settings.json +++ b/.vscode/settings.json @@ -15,4 +15,18 @@ "[restructuredtext]": { "editor.tabSize": 3, }, + + // Python emnvironment + "python.envFile": "${workspaceFolder}/.env", + + // Restructured Text + "restructuredtext.linter.doc8.extraArgs": [ + "--config", + "${workspaceFolder}/docs/doc8.ini" + ], + + // Sphinx-Needs + "ubcode.srcDir": "${workspaceFolder}/docs", + "ubcode.needsJson": "${workspaceFolder}/docs/needs.json", + } diff --git a/docs/conf.py b/docs/conf.py index 4da35231..6f9a7cbd 100644 --- a/docs/conf.py +++ b/docs/conf.py @@ -17,7 +17,83 @@ # https://www.sphinx-doc.org/en/master/usage/configuration.html # from process.process_model_configuration import * -from _extensions import metamodel, layouts +# from _extensions import metamodel, layouts + +class metamodel: + needs_types = [ + # Requirements + dict( + directive="stkh_req", + title="Stakeholder requirements", + prefix="STKH_REQ__", + color="#BFD8D2", + style="node", + ), + dict( + directive="tool_req", + title="Tool Requirements", + prefix="TOOL_REQ__", + color="#BFD8D2", + style="node", + ), + ] + + # Define extra options for needs object + needs_extra_options = [ + "security", + "safety", + "level", + "rationale", + "mitigated_by", + "reqtype", + "codelink", + "testlink", + "reqcovered", + "testcovered", + ] + + needs_extra_links = [ + # TODO: Refer process document for the usage of links + { + "option": "satisfies", + "incoming": "is satisfied by", + "outgoing": "satisfies", + "style_start": "-up", + "style_end": "->", + }, + {"option": "implements", "incoming": "implements by", "outgoing": "implements"}, + ] + +class layouts: + needs_layouts = { + "score": { + "grid": "complex", + "layout": { + "head_left": [ + '<>', + ], + "head": [ + 'status: **<>**', + 'security: **<>**', + 'safety: **<>**', + ], + "head_right": [ + '<> ' + ], + "meta_left": [ + '<>', + "<>", + ], + "meta_right": [], + "footer_left": ["<>"], + "footer": ['<>'], + "footer_right": [], + }, + }, + } + + needs_global_options = {"layout": "score"} + # -- Project information ----------------------------------------------------- diff --git a/docs/doc8.ini b/docs/doc8.ini new file mode 100644 index 00000000..8fb0e5c0 --- /dev/null +++ b/docs/doc8.ini @@ -0,0 +1,2 @@ +[doc8] +ignore-path-errors=D004 \ No newline at end of file diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index fd732506..8a2c545f 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -19,10 +19,11 @@ Inter-process Communication (IPC) Framework handles the safe, secure and perform The IPC framework is independent of the application context it is used in. -There are two fundamental information exchange concepts defined +There are three fundamental information exchange concepts defined -* Topics: A topic is a data entity identifyed by a name, being structured by a *data type* and containing zero to multiple instances of that type (a *value*). -* Remote Procedures: A remote procedure is an invocable entity identifyed by a name, being structured by parameters of a *data type* and being activated by an invokation passing arguments, i.e. a value for each parameter. +* **Topics**: A topic is a data entity identifyed by a name, being structured by a *data type* and containing zero to multiple instances of that type (a *value*). +* **Remote Procedures**: A remote procedure is an invocable entity identifyed by a name, being structured by parameters of a *data type* and being activated by an invokation passing arguments, i.e. a value for each parameter. +* **Events**: An event signals the change of the state of something. An event carries no data, only the information "the related state change occurred". Data Types ---------- @@ -173,6 +174,14 @@ Remote Procedures * Invocation * Sync/Async +Events +------ + +* Name +* Publisher +* Subscriber +* Chains / Buffering + Zero Copy --------- From b50eef7ae88512674a977c6cc4887f4c9ca57dfa Mon Sep 17 00:00:00 2001 From: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> Date: Mon, 16 Dec 2024 10:58:25 +0530 Subject: [PATCH 04/12] ipc: added names and namespaces --- docs/features/communication/ipc/index.rst | 233 +++++++++++++++++++++- 1 file changed, 222 insertions(+), 11 deletions(-) diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index 8a2c545f..e0124dcd 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -11,24 +11,211 @@ # # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* + # + # Authors + # @hartmannnico, + + +Communication Framework +####################### + +The Communication Framework handles the safe, secure and performant exchange of information between software and/or hardware components of the Score stack. + +Communiction covers information exchange between endpoints residing in +* the same or different process, running on +* the same or different operating systems, deployed on +* the same or different compute devices, built into +* the same or different devices. + +In current communication designs (SomeIP, Protobuf, Zenoh) communication on networks is at the centerpoint of considerations. We believe this to be the wrong approach for software defined machines. Network based communication circles around protocols based on a wire based communication paradigm, requiring serialization and segmentation of data. + +Instead, we promote a memory centered core paradigm and put Inter-Process Communication (IPC) into our conceptional focus. This allows for true zero copy design, does not segment or serialize data and fosters significantly higher performance in algorithms. As a bonus, it provides easy gateway mechanisms into serial communication like Ethernet and CAN. + +The communication framework is independent of the domain of the application it is used in. + +General Architecture +-------------------- + +The communication framework defines two central groups of elements: + +* Information Elements +* Infrastructure Elements + +While the former carry the information exchanged the latter provide the means through which the information is exchanged. + +Information Elements +```````````````````` + +There are three fundamental concepts of information exchange defined. One communication element represents each concept: + +* **Topics**: A topic is the information carrier for **data**. A unique Id identifies a topic while a *data type* defines it's memory layout. The topic carries zero or multiple *values*. A value represents a single instances of the data type. See _`Topics`, _`Names`, _`Data Types`. +* **Remote Procedures**: A remote procedure is the information carrier for **execution** progress. A Id handle identifies the remote procedure together with an ordered set of named *parameters*. Each parameter defined by a data type. A caller of a remote procedure can cause it's activation by invoking the remote procedure with passed *arguments*. An argument is a single value instance for a parameter. See _`Remote Procedures`, _`Names`, _`Data Types`. +* **Events**: An event is the information carrier for runtime **synchronization**. A unique Id identifies the event. It signals the change of state. There is no data conveyed with the event. See _`Events`, _`Names`. + +While the Id uniquely identifies an information element within the communication framework, it can also have a *name* as alias to conveniently identify the element. While the Id may not be publicly known, the *name* allows for public lookup. + + +Infrastructure Elements +``````````````````````` + +Infrastructure elements provide the means through which the information exchange executes. +We define three fundamental building elements: + +* **Endpoints**: Endpoints are both the source and the target of every information exchange in teh communication framework. An endpoint providing information is consequently called a *provider*. With the same logic a *consumer* is an endpoint consuming information. Endpoints have an *Id* that uniquely identifies the endpoint within a node. +* **Nodes**: A node is an entity in the communication system that hosts several endpoints. It is the central element of the communication fabric by connecting endpoints and routing data. Nodes have an *Id* that uniquely identifies the node within a fabric. A node itself is also an endpoint. +* **Links**: A link is the fundamental abstraction of a connection between any two nodes. A link conveys information between nodes. + +The combination of NodeId and EndpointId we also refer to as *address*. As nodes are also endpoints, they implicitly have an address. + +Nodes and endpoints may also be identified by a *name* that resolves into references to these elements. See _`Names`, _`References`. + +Connecting nodes though links creates a mesh of nodes that can mutually exchange information utilizing the above concepts. The boundary of the mesh is at the sole discretion of the deployment and may span from a single application into a connected cloud environment. + +The entirety of connected nodes within a mesh we call *fabric*. + +**Design Note** -Inter-process Communication -########################### +From a perspective of safety, a node also encapsulates a single safety domain. Links provide the means for separating safety domains and thus allow for mixed criticality applications. -Inter-process Communication (IPC) Framework handles the safe, secure and performant exchange of information between software and/or hardware components of the Score stack. -The IPC framework is independent of the application context it is used in. +Names +----- -There are three fundamental information exchange concepts defined +A name is an UTF-8 tag of an element. The underlying Unicode standard is release 16.0.0 https://www.unicode.org/versions/Unicode16.0.0/. + +All codepoints are valid in a name with the exception of the following codepoints: + +* the *control* codepoints ``�`` to ``F`` +* the *dot* separator, codepoint ``FULL STOP``: ``.`` ``E`` +* the *path separator*, codepoint ``SOLIDUS``: ``/`` ``F`` +* the *wildcard* marker, codepoint ``ASTERISK``: ``*`` ``A`` +* the *any* marker, codepoint ``QUESTION MARK``: ``?`` ``F`` + +Forbidden as first character in a name is + +* the *reference* marker codepoint ``AMPERSAND``: ``&`` ````. It's use is discouraged in names in general. + +Further discouraged is the use of the *whitespace* codepoint ``SPACE``: `` `` ````. + +Element names prefixed with an underscore ``LOW LINE``: ``_`` ``F`` are regarded to have *private* visibility within the scope they are defined in. While references to private elements are possible, name resolution ony works from within the namespace they are defined in. + +**Design Note** + +A name is not a property of an element itself. +Instead, a name acts as an *alias* to obtain an element *reference*. +See _`References`. + + +Namespaces +---------- + +A namespace is a named scope in which the definition of the elements topic, remote procedures, event and recursively namespace is valid. + +General rules of names apply to namespace names. + +Namespaces can be nested. +The path separator between the names is the unicode codepoint ``SOLIDUS``: ``/`` ``F``. + +:: + + This/is/a/nested/namespace + + +The namespace name ``/`` is reserved and refers to the global namespace. +*Global* here means visible with respect to a certain realm that is not further defined. A realm can be a vehicle with it's attached cloud environment or just an application context. It is up to deployment to define the scope of the global namespace. + +The namespace name ``super`` is reserved and refers to the parent namespace of the namespace where ``super`` is used. The use of ``super`` in the global namespace is an error. + +The namespace name ``package`` is reserved for future use. It must appear as first name in a path. + + +Scoping Rules +````````````` -* **Topics**: A topic is a data entity identifyed by a name, being structured by a *data type* and containing zero to multiple instances of that type (a *value*). -* **Remote Procedures**: A remote procedure is an invocable entity identifyed by a name, being structured by parameters of a *data type* and being activated by an invokation passing arguments, i.e. a value for each parameter. -* **Events**: An event signals the change of the state of something. An event carries no data, only the information "the related state change occurred". +Namespaces isolate a naming scope from another. Within a namespace element names must be unique. + +Elements in one namespace are by default not visible to elements in other namespaces. + +:: + + namespace A + topic T: Int8 // This is T in A + + namespace B + topic T: Float64 // This is T in B + +The topics ``A/T`` and ``B/T`` are different. + +Namespaces can be nested, i.e. within a namespace another namespace can exist. + +:: + + namespace A + topic T: Int8 // This resides in namespace A + + namespace Aa + topic T: Int8 // This T resides in namespace A/Aa + + +A namespace can *use* elements of another namespace thus making it visible under a given or implied local name. + +:: + + namespace A + topic Important: Int8 + + namespace Aa + topic Nested: Int8 + + namespace B + use A/Important // makes topic "Important" visible in B + use A/Aa // makes namespace "Aa" visible in B + + algorithm + access Important + access Aa/Nested + +A ``use`` clause may end with the wildcard ``ASTERISK``: ``*`` ``A``. This indicates the mapping of all elements of the given namespace into the current scope. + +:: + + namespace A + topic One: Int8 + topic Two: Int8 + + namespace B + use A/* // makes One and Two visible in B + + algorithm + access One // valid, One is visible in B + access Two // valid, Two is visible in B + + +Within a namespace elements from another namespace are visible without an explicit use when a resolving path is given. + +:: + + namespace A + topic Important: Int8 + + namespace B + algorithm + access A/Important + + +Handles +------- + +A handle is a numeric value that uniquely refers to an individual element in the communication system. + +A specific element in the communication system Data Types ---------- Data types describe the inner structure of data entities known as values. +A specific data type will always have the identical memory layout, independent from compiler, operating system and controller architecture. + Primitive Types ``````````````` @@ -49,12 +236,11 @@ Float64 Floating ``f64`` ``double``, ``float64_t`` 8 An IEEE 756 64 BFloat16 Floating ``bf16`` ``bfloat16_t`` 2 A Google brain float 16 floating point Char String ``char`` ``char32_t`` 4 A unicode codepoint (32 bit) String String ``str`` - n/a A UTF-8 encoded text +Handle Reference - - 8 A 64 bit unsigned integer handle ========= ========== ========= ========================== ====== ============================ The type ``Byte`` may be used as alias for ``UInt8``. - -| Pointers are no recognized primitive types. -| References are no recognized primitive types. +The type ``Handle`` may be used as alias for ``UInt64``. Tuples `````` @@ -62,6 +248,11 @@ Tuples Tuples are ordered collections of arbitrary data types. A tuple shall be expressed by parentheses: ``(Int8, Float32)``. +The empty Tuple `()` is called the *unit* type, identifying the type with no data. The unit type can carry the one and only one value `()`. + +There is no explicit Data Type for neither `Empty`, `Void` or `None`. + + Structs ``````` @@ -117,6 +308,26 @@ A HashMap is an unordered collection of data element of the same type with varia HashMap +Pointers +```````` + +There are no data types defined for pointers to other data. + +References +`````````` + +| Generic references are not allowed as data types. Instead, there are distinct reference types defined: + +* Topic References +* Remote Procedure References +* Event References + +The reference marker to an element shall be `AMPERSAND`: `&` ``. + +Values of reference + + + Topic Reference ``````````````` From caa6411d61f00ba3bc3b5aa5d56cf680a9ec59ae Mon Sep 17 00:00:00 2001 From: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> Date: Fri, 20 Dec 2024 15:25:12 +0530 Subject: [PATCH 05/12] ipc: revised references --- docs/features/communication/ipc/index.rst | 78 +++++++++++------------ 1 file changed, 37 insertions(+), 41 deletions(-) diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index e0124dcd..b35cc9d9 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -13,7 +13,7 @@ # ******************************************************************************* # # Authors - # @hartmannnico, + # @hartmannnico, Communication Framework @@ -21,7 +21,8 @@ Communication Framework The Communication Framework handles the safe, secure and performant exchange of information between software and/or hardware components of the Score stack. -Communiction covers information exchange between endpoints residing in +Communication covers information exchange between endpoints residing in + * the same or different process, running on * the same or different operating systems, deployed on * the same or different compute devices, built into @@ -75,13 +76,13 @@ The entirety of connected nodes within a mesh we call *fabric*. **Design Note** -From a perspective of safety, a node also encapsulates a single safety domain. Links provide the means for separating safety domains and thus allow for mixed criticality applications. +From a perspective of safety, a node also encapsulates a single safety domain. Links provide the means for separating safety domains and thus allow for mixed criticality applications. Names ----- -A name is an UTF-8 tag of an element. The underlying Unicode standard is release 16.0.0 https://www.unicode.org/versions/Unicode16.0.0/. +A name is an UTF-8 tag of an element. The underlying Unicode standard is release 16.0.0 https://www.unicode.org/versions/Unicode16.0.0/. All codepoints are valid in a name with the exception of the following codepoints: @@ -91,7 +92,7 @@ All codepoints are valid in a name with the exception of the following codepoint * the *wildcard* marker, codepoint ``ASTERISK``: ``*`` ``A`` * the *any* marker, codepoint ``QUESTION MARK``: ``?`` ``F`` -Forbidden as first character in a name is +Forbidden as first character in a name is * the *reference* marker codepoint ``AMPERSAND``: ``&`` ````. It's use is discouraged in names in general. @@ -101,8 +102,8 @@ Element names prefixed with an underscore ``LOW LINE``: ``_`` ``F`` are reg **Design Note** -A name is not a property of an element itself. -Instead, a name acts as an *alias* to obtain an element *reference*. +A name is not a property of an element itself. +Instead, a name acts as an *alias* to obtain an element *reference*. See _`References`. @@ -113,7 +114,7 @@ A namespace is a named scope in which the definition of the elements topic, remo General rules of names apply to namespace names. -Namespaces can be nested. +Namespaces can be nested. The path separator between the names is the unicode codepoint ``SOLIDUS``: ``/`` ``F``. :: @@ -121,7 +122,7 @@ The path separator between the names is the unicode codepoint ``SOLIDUS``: ``/`` This/is/a/nested/namespace -The namespace name ``/`` is reserved and refers to the global namespace. +The namespace name ``/`` is reserved and refers to the global namespace. *Global* here means visible with respect to a certain realm that is not further defined. A realm can be a vehicle with it's attached cloud environment or just an application context. It is up to deployment to define the scope of the global namespace. The namespace name ``super`` is reserved and refers to the parent namespace of the namespace where ``super`` is used. The use of ``super`` in the global namespace is an error. @@ -206,9 +207,9 @@ Within a namespace elements from another namespace are visible without an explic Handles ------- -A handle is a numeric value that uniquely refers to an individual element in the communication system. +A handle is a numeric value that uniquely refers to an individual element in the communication system. -A specific element in the communication system +A specific element in the communication system Data Types ---------- @@ -236,7 +237,7 @@ Float64 Floating ``f64`` ``double``, ``float64_t`` 8 An IEEE 756 64 BFloat16 Floating ``bf16`` ``bfloat16_t`` 2 A Google brain float 16 floating point Char String ``char`` ``char32_t`` 4 A unicode codepoint (32 bit) String String ``str`` - n/a A UTF-8 encoded text -Handle Reference - - 8 A 64 bit unsigned integer handle +Handle Reference - - 8 A 64 bit unsigned integer handle ========= ========== ========= ========================== ====== ============================ The type ``Byte`` may be used as alias for ``UInt8``. @@ -311,57 +312,51 @@ A HashMap is an unordered collection of data element of the same type with varia Pointers ```````` -There are no data types defined for pointers to other data. +There are no data types defined for pointers to data types. References `````````` -| Generic references are not allowed as data types. Instead, there are distinct reference types defined: - -* Topic References -* Remote Procedure References -* Event References +The communication framework allows for three potential classes of references: -The reference marker to an element shall be `AMPERSAND`: `&` ``. +* References to data types + There are no data types defined for references to data types. -Values of reference +* References to information elements + * Topic References, data type ``TopicRef`` + * Remote Procedure References, data type ``FnRef`` + * Event References, data type ``EventRef`` +* References to infrastructure elements + Currently we do not define references to infrastructure elements. + However, for conceptual symmetry reasons and application value they + might come up in future versions. -Topic Reference -``````````````` +The `AMPERSAND`: `&` `` as first character in a path is defined as the marker for references to information elements. -A topic reference is a data type that provides a resolvable reference to a Topic. +In names and paths the reference marker to an information element of a path. :: - TopicRef - -The underlying implementation shall not use more than 64 bits for the representation of TopicRef. - -All properties of the referenced topics shall be retrievable through the TopicRef. - -Data access to the content of the Topic shall be possible through the TopicRef. + &/Body/Doors/Windows/LeftFront/Position +References the topic of the top left windows's position. -RP Reference -````````````` - -An RP Reference is a data type that provides a resolvable reference to a Remote Procedure. +A Reference shall have a corresponding unique handle. The communication framework shall be able to dispatch handles of references like any other value of a data type. The underlying value type for handles should be ``UInt64`` and must have lockfree atomic read and write operations available. -:: +The application should not have access to handles directly, but only to the references themselves. We call the conversion of a handle into a reference “resolution of the handle”. - RPRef +The operations granted through a reference to an information item shall be identical to the operations of the information item itself. -The underlying implementation shall not use more than 64 bits for the representation of RPRef. -All properties of the referenced Remote Procedure shall be retrievable through the RPRef. +**Implementation Note** -Invocation of the referenced Remote Procedure shall be possible through the RPRef. +Internally, the communication framework may actually only pass TopicRef’s to the application. From a semantic view it makes no difference to hold a TopicRef or a Topic directly. +**Implementation Note** -Namespaces ----------- +A ``TopicRef`` is *not* the same as ``&Topic`` as it may require additional validity checks. Topics @@ -405,3 +400,4 @@ Zero Copy Safety ------ + From 8b1e99bbae2be328c7da2e2af15823781c25d991 Mon Sep 17 00:00:00 2001 From: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> Date: Fri, 13 Dec 2024 14:37:34 +0100 Subject: [PATCH 06/12] ipc: initial feature request Issue-ref: see #69 --- docs/features/communication/ipc/index.rst | 530 +++++++++++++++++++++- 1 file changed, 528 insertions(+), 2 deletions(-) diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index ce7b68d1..a0635342 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -11,6 +11,532 @@ # # SPDX-License-Identifier: Apache-2.0 # ******************************************************************************* + # + # Authors + # @hartmannnico, + + +Communication Framework +####################### + +The Communication Framework handles the safe, secure and performant exchange of information between software and/or hardware components of the Score stack. + +Communication covers information exchange between endpoints residing in + +* the same or different process, running on +* the same or different operating systems, deployed on +* the same or different compute devices, built into +* the same or different devices. + +In current communication designs (SomeIP, Protobuf, Zenoh) communication on networks is at the centerpoint of considerations. We believe this to be the wrong approach for software defined machines. Network based communication circles around protocols based on a wire based communication paradigm, requiring serialization and segmentation of data. + +Instead, we promote a memory centered core paradigm and put Inter-Process Communication (IPC) into our conceptional focus. This allows for true zero copy design, does not segment or serialize data and fosters significantly higher performance in algorithms. As a bonus, it provides easy gateway mechanisms into serial communication like Ethernet and CAN. + +The communication framework is independent of the domain of the application it is used in. + +General Architecture +-------------------- + +The communication framework defines two central groups of elements: + +* Information Elements +* Infrastructure Elements + +While the former carry the information exchanged the latter provide the means through which the information is exchanged. + +Information Elements +```````````````````` + +There are three fundamental concepts of information exchange defined. One communication element represents each concept: + +* **Topics**: A topic is the information carrier for **data**. A unique Id identifies a topic while a *data type* defines it's memory layout. The topic carries zero or multiple *values*. A value represents a single instances of the data type. See `Topics`_, `Names`_, `Data Types`_. +* **Remote Procedures**: A remote procedure is the information carrier for **execution** progress. A Id handle identifies the remote procedure together with an ordered set of named *parameters*. Each parameter defined by a data type. A caller of a remote procedure can cause it's activation by invoking the remote procedure with passed *arguments*. An argument is a single value instance for a parameter. See `Remote Procedures`_, `Names`_, `Data Types`_. +* **Events**: An event is the information carrier for runtime **synchronization**. A unique Id identifies the event. It signals the change of state. There is no data conveyed with the event. See `Events`_, `Names`_. + +While the Id uniquely identifies an information element within the communication framework, it can also have a *name* as alias to conveniently identify the element. While the Id may not be publicly known, the *name* allows for public lookup. + + +Infrastructure Elements +``````````````````````` + +Infrastructure elements provide the means through which the information exchange executes. +We define three fundamental building elements: + +* **Endpoints**: Endpoints are both the source and the target of every information exchange in teh communication framework. An endpoint providing information is consequently called a *provider*. With the same logic a *consumer* is an endpoint consuming information. Endpoints have an *Id* that uniquely identifies the endpoint within a node. +* **Nodes**: A node is an entity in the communication system that hosts several endpoints. It is the central element of the communication fabric by connecting endpoints and routing data. Nodes have an *Id* that uniquely identifies the node within a fabric. A node itself is also an endpoint. +* **Links**: A link is the fundamental abstraction of a connection between any two nodes. A link conveys information between nodes. + +The combination of NodeId and EndpointId we also refer to as *address*. As nodes are also endpoints, they implicitly have an address. + +Nodes and endpoints may also be identified by a *name* that resolves into references to these elements. See `Names`_, `References`_. + +Connecting nodes though links creates a mesh of nodes that can mutually exchange information utilizing the above concepts. The boundary of the mesh is at the sole discretion of the deployment and may span from a single application into a connected cloud environment. + +The entirety of connected nodes within a mesh we call *fabric*. + +.. admonition:: Design Note + + From a perspective of safety, a node also encapsulates a single safety domain. Links provide the means for separating safety domains and thus allow for mixed criticality applications. + + +Names +----- + +A name is an UTF-8 tag of an element. The underlying Unicode standard is `release 16.0.0 `_. + +All codepoints are valid in a name with the exception of the following codepoints: + +* the *control* codepoints ``�`` to ``F`` +* the *dot* separator, codepoint ``FULL STOP``: ``.`` ``E`` +* the *path separator*, codepoint ``SOLIDUS``: ``/`` ``F`` +* the *wildcard* marker, codepoint ``ASTERISK``: ``*`` ``A`` +* the *any* marker, codepoint ``QUESTION MARK``: ``?`` ``F`` + +Forbidden as first character in a name is + +* the *reference* marker codepoint ``AMPERSAND``: ``&`` ````. It's use is discouraged in names in general. + +Further discouraged is the use of the *whitespace* codepoint ``SPACE``: `` `` ````. + +Element names prefixed with an underscore ``LOW LINE``: ``_`` ``F`` are regarded to have *private* visibility within the scope they are defined in. While references to private elements are possible, name resolution ony works from within the namespace they are defined in. + +.. admonition:: Design Note + + A name is not a property of an element itself. + Instead, a name acts as an *alias* to obtain an element *reference*. + See `References`_. + + +Namespaces +---------- + +A namespace is a named scope in which the definition of the elements topic, remote procedures, event and recursively namespace is valid. + +General rules of names apply to namespace names. + +Namespaces can be nested. +The path separator between the names is the unicode codepoint ``SOLIDUS``: ``/`` ``F``. + +:: + + This/is/a/nested/namespace + + +The namespace name ``/`` is reserved and refers to the global namespace. +*Global* here means visible with respect to a certain realm that is not further defined. A realm can be a vehicle with it's attached cloud environment or just an application context. It is up to deployment to define the scope of the global namespace. + +The namespace name ``super`` is reserved and refers to the parent namespace of the namespace where ``super`` is used. The use of ``super`` in the global namespace is an error. + +The namespace name ``package`` is reserved for future use. It must appear as first name in a path. + + +Scoping Rules +````````````` + +Namespaces isolate a naming scope from another. Within a namespace element names must be unique. + +Elements in one namespace are by default not visible to elements in other namespaces. + +:: + + namespace A + topic T: Int8 // This is T in A + + namespace B + topic T: Float64 // This is T in B + +The topics ``A/T`` and ``B/T`` are different. + +Namespaces can be nested, i.e. within a namespace another namespace can exist. + +:: + + namespace A + topic T: Int8 // This resides in namespace A + + namespace Aa + topic T: Int8 // This T resides in namespace A/Aa + + +A namespace can *use* elements of another namespace thus making it visible under a given or implied local name. + +:: + + namespace A + topic Important: Int8 + + namespace Aa + topic Nested: Int8 + + namespace B + use A/Important // makes topic "Important" visible in B + use A/Aa // makes namespace "Aa" visible in B + + algorithm + access Important + access Aa/Nested + +A ``use`` clause may end with the wildcard ``ASTERISK``: ``*`` ``A``. This indicates the mapping of all elements of the given namespace into the current scope. + +:: + + namespace A + topic One: Int8 + topic Two: Int8 + + namespace B + use A/* // makes One and Two visible in B + + algorithm + access One // valid, One is visible in B + access Two // valid, Two is visible in B + + +Within a namespace elements from another namespace are visible without an explicit use when a resolving path is given. + +:: + + namespace A + topic Important: Int8 + + namespace B + algorithm + access A/Important + + +Handles +------- + +A handle is a numeric value that uniquely refers to an individual element in the communication system. + +A specific element in the communication system + +Data Types +---------- + +Data types describe the inner structure of data entities known as values. +A specific data type will always have the identical memory layout, independent from compiler, operating system and controller architecture. + + +Primitive Types +``````````````` + +Primitive data types consist of a single element with no further inherent structure. +The following primitive data types and their Rust and C++ representation are defined: + +========= ========== ========= ========================== ====== ============================ +Data Type Class Rust C++ Size Description +========= ========== ========= ========================== ====== ============================ +Bool Boolean ``bool`` ``bool`` 1 A boolean value, true or false +Int8 Integer ``i8`` ``int8_t`` 1 An 8 bit signed integer +UInt8 Integer ``u8`` ``uint8_t`` 1 An 8 bit unsigned integer +UInt128 Integer ``u128`` ``uint128_t`` 16 An 128 bit unsigned integer +Float16 Floating ``f16`` ``float16_t`` 2 An IEEE 756 32 bit floating point +Float32 Floating ``f32`` ``float``, ``float32_t`` 4 An IEEE 756 32 bit floating point +Float64 Floating ``f64`` ``double``, ``float64_t`` 8 An IEEE 756 64 bit floating point +BFloat16 Floating ``bf16`` ``bfloat16_t`` 2 A Google brain float 16 floating point +Char String ``char`` ``char32_t`` 4 A unicode codepoint (32 bit) +String String ``str`` - n/a A UTF-8 encoded text +Handle Reference - - 8 A 64 bit unsigned integer handle +========= ========== ========= ========================== ====== ============================ + +The type ``Byte`` may be used as alias for ``UInt8``. +The type ``Handle`` may be used as alias for ``UInt64``. + +Tuples +`````` + +Tuples are ordered collections of arbitrary data types. A tuple shall be expressed by parentheses: +``(Int8, Float32)``. + +The empty Tuple `()` is called the *unit* type, identifying the type with no data. The unit type can carry the one and only one value `()`. + +There is no explicit Data Type for neither `Empty`, `Void` or `None`. + + +Structs +``````` + +A struct is an ordered collection of named arbitrary data types called fields: + +:: + + struct MyStruct { + field1: Int8, + field2: Float32 + } + +Arrays +`````` + +Arrays are ordered collections of data elements of the same type with a fixed length. A single element is addressed by an *index*: + +:: + + Char[32] + + +Tensors +``````` + +A Tensor is a multi-dimensional array of numerical values that generalizes scalars, vectors, and matrices to higher dimensions, commonly used in mathematics, physics, and machine learning. + +The number of dimensions is called the *rank* or *order* of the Tensor. +The vector of dimensions with the Tensor's rank is called the *shape* of the Tensor. + +:: + + Tensor + +The 5x5 kernel of a CNN layer with 128 features. + +List +```` + +A List is an ordered collection of data elements of the same type with a variable length. A single element is addressed by an *index*: + +:: + + List + +HashMap +``````` + +A HashMap is an unordered collection of data element of the same type with variable length. A single element is addressed by a *key* of a specific data type. + +:: + + HashMap + + +Pointers +```````` + +There are no data types defined for pointers to data types. + +References +`````````` + +The communication framework allows for three potential classes of references: + +* References to data types + + There are no data types defined for references to data types. + +* References to information elements + + * Topic References, data type ``TopicRef`` + * Remote Procedure References, data type ``FnRef`` + * Event References, data type ``EventRef`` + +* References to infrastructure elements + + Currently we do not define references to infrastructure elements. + However, for conceptual symmetry reasons and application value they + might come up in future versions. + +The `AMPERSAND`: `&` `` as first character in a path is defined as the marker for references to information elements. + +In names and paths the reference marker to an information element of a path. + +:: + + &/Body/Doors/Windows/LeftFront/Position + +References the topic of the top left windows's position. + +A Reference shall have a corresponding unique handle. The communication framework shall be able to dispatch handles of references like any other value of a data type. The underlying value type for handles should be ``UInt64`` and must have lockfree atomic read and write operations available. + +The application should not have access to handles directly, but only to the references themselves. We call the conversion of a handle into a reference “resolution of the handle”. + +The operations granted through a reference to an information item shall be identical to the operations of the information item itself. + + +.. admonition:: Implementation Note + + Internally, the communication framework may actually only pass TopicRef's to the application. From a semantic view it makes no difference to hold a TopicRef or a Topic directly. + +.. admonition:: Implementation Note + + A ``TopicRef`` is *not* the same as ``&Topic`` as it may require additional validity checks. + + +Topics +------ + +A topic is an information carrier for data elements. Data elements have a data type and zero to multiple values, following the format and layout defined by the type. + +Publisher & Subscriber +`````````````````````` + +A topic follows the publisher/subscriber pattern. This means + +- A topic exists on it's own. The framework's data communication system owns the topic. +- A topic can have zero or one publishers. The publisher updates ('publishes') new data into the topic. +- A topic can have zero or multiple subscribers. A subscriber consumes the data published into the topic. + +Queue +````` + +A topic stores it's values in a queue with a given depth. The queue has a policy that defines the behavior of new data published. There are four policies when the queue is full: + +- Ignore: New data is ignored. +- Overwrite oldest: New data overwrites the oldest element. This makes the topic a ring buffer. +- Replace latest: New data replaces the latest element. This makes the latest update always available. +- Error: Writing new data to the topic raises an error at the subscriber (and potentially in the topic) + +Namespace and Name +`````````````````` + +A topic can have a name that exists within a namespace. The name is not an attribute of the topic itself. Instead, it is an alias to the topic's reference that allows access to the data of the topic. + +Lifetime +```````` + +Once created, the topic belongs to the communication framework which determines it's lifetime. + +A topic must live while it is in use, i.e. it has a publisher and/or subscribers. + + +Remote Procedures +----------------- + +A remote procedure is an invokable information element that receives a set of values ('arguments') following a given signature ('parameters') individual to each invocation. + +The invocation happens asyncronously. I.e. the invocation of a remote procedure returns immediately to the invoker. + +Synchronous behavior is achievable by two intertwined remote procedures. See Result. + +**Discussion**: An alternative to intertwined RPCs is the use of a Future mechanism. The advantage of the RPC reference based approach is the symmetry in invocation and result transmission. One implementation achieves both. + +Result +`````` + +A remote procedure may produce a result that is returned to the caller. The result also has a data type and consists of a single value. + +.. note:: + + Instead of passing back the result from the procedure the caller may pass a result-return reference that is a remote procedure itself. This way the framework may have a straight-forward way of implementing a Future mechanism that completes upon reception of the response call. + +Name & Namespace +```````````````` + +A remote procedure may have a name that exists in a namespace. The name is not a property of the remote procedure, but acts as an alias to a unique RPC reference associated with the remote procedure. + +Publishing & Discovery +`````````````````````` + +Attaching a name to a remote procedure means to publish the remote procedure. +The communication framework owns both the name and the namespace. + +Thus, publishing a remote procedure under a name also provides the means to discover it. + +Interfaces & Services +````````````````````` + +There is no specific structure defined as an 'Interface' or 'Service'. Instead, an interface as collection of remote procedures can be seen as a collection of remote procedures within a specific namespace that represents the interface. + +Lifetime +```````` + +Once created, the remote procedure belongs to the communication framework which determines it's lifetime. + +It must exist as long as references to it exist. + +Events +------ + +An event is an information element that communicates the change of a state. An event has no value. +The main purpose of an event is to support runtime orchestration. + +.. note:: An Event is not the same as a topic with no data. Topic mechanisms are designed to convey values. Events convey occurrences of state changes. + +Publisher & Subscriber +`````````````````````` + +An event follows the publisher/subscriber pattern. This means + +- An event exists on it's own. The framework's data communication system owns the event. +- An event can have zero or one publishers. The publisher updates ('publishes') new occurrences of associated state changes into the event. This is called 'triggering the event'. +- An event can have zero or multiple subscribers, also called 'listeners'. A subscriber consumes the state change notifications. + +Immediate Events & Queued Events +```````````````````````````````` + +An event is designed to convey an immediate notification of the associated state change. However for cases where a subscriber cannot react immediately an event occurrence may be latched in a queue for deferred processing. This is called an 'Event Queue'. The framework may opt to offer event queues on top of immediate event propagation. + +Namespace and Name +`````````````````` + +An event can have a name that exists within a namespace. The name is not an attribute of the event itself. Instead, it is an alias to the event's reference that allows triggering and listening to the event. + +Lifetime +```````` + +Once created, the event belongs to the communication framework which determines it's lifetime. + +An event must live while it is in use, i.e. it has a publisher and/or subscribers. + + +Zero Copy +--------- + +Zero-Copy data exchange is defined as concurrent access to data by a sender and a receiver without alternation of the memory location or layout of the data in the process of the exchange. + +This includes: + +- No serialization of data +- No deserialization of data +- No moving of data in memory + +Data Properties +``````````````` + +To meet zero-copy requirements data require to be: + +- Coherent - All data belonging to a data item must occupy adjacent memory locations. +- Relocatable - The correct interpretation of a data item is independent from it's address in memory. +- Atomic - Access to the data item is an atomic operation. To achieve this the data requires one of two access modes: + + - Lockfree Access - No thread lock is required to read or write the data. This is the preferred property of zero-copy data. + - Mutually Exclusive Access - A thread lock (mutex) is required to access the data. + +Buffers +``````` + +In general zero-copy requires that data locations and layout are owned by the communication framework. Obviously these locations must be placed in shared memory to allow access from both producer and consumer side, should these lie in different processes or operating systems or even compute devices. + +A data storage memory location is called a 'buffer'. The communication framework executes buffer allocation and deallocation. It shall provide references to these buffers that can be shared between the communication partners. + +From a data producer side as well as from a data consumer side this means that the data is accessed directly through a buffer reference to the data. + +DMA +``` + +As large amount of data are often produced or consumed by hardware the communication framework shall be able to provide raw access to buffers for direct memory access (DMA) capabilities of the underlying platforms. + +Safety +------ + +We base this document on the ISO 26262-1:2018 released in December 2018. + +Exchange of information through information elements always involves an information producer and one or many information consumers. As these can be part of different functions or partitions in the context of the application purpose. As such these partitions are likely to differ in their safety requirements and are thus domains of different safety levels. + +The communication framework shall support safety integrity level ASIL-B. The communication framework shall also provide means for information exchange in mixed-criticallity applications where sender and receiver reside in domains of different safety classification. + + +Security +-------- + +We base this document on the ISO/SAE 21434 released in August 2021. + +Communication and data exchange at the boundaries within the communication framework is subject of security considerations. + +The communication framework shall support the following principal security capabilities: + +- Authenticaion: Unambigous identification of the communication elements, especially producers and consumers of data. +- Authorization: A set of rules granting or denying access to communication operations based on the Authentication of participants in the communication framework. +- Protection: Means to protect the integrity of data received by consumers. +- Encryption: Means to protext the content of data in transit. -Inter-process Communication -########################### +The definition of requirements for appropriate cryptographic hashing and encryption algorithms is not part of this document. From 3edb2c8699de4503fe800fe01674b00df52fb090 Mon Sep 17 00:00:00 2001 From: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> Date: Wed, 15 Jan 2025 12:13:41 +0100 Subject: [PATCH 07/12] Added TF32 format in data types Signed-off-by: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> --- docs/features/communication/ipc/index.rst | 1 + 1 file changed, 1 insertion(+) diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index a0635342..dc79c462 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -235,6 +235,7 @@ Float16 Floating ``f16`` ``float16_t`` 2 An IEEE 756 32 Float32 Floating ``f32`` ``float``, ``float32_t`` 4 An IEEE 756 32 bit floating point Float64 Floating ``f64`` ``double``, ``float64_t`` 8 An IEEE 756 64 bit floating point BFloat16 Floating ``bf16`` ``bfloat16_t`` 2 A Google brain float 16 floating point +TFloat32 Floating ``tf32`` ``tfloat32_t`` 2 An NVIDIA tensor float 32 floating point Char String ``char`` ``char32_t`` 4 A unicode codepoint (32 bit) String String ``str`` - n/a A UTF-8 encoded text Handle Reference - - 8 A 64 bit unsigned integer handle From aa3ff3cb66b41f09a69a14983991036bd0a4c409 Mon Sep 17 00:00:00 2001 From: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> Date: Wed, 15 Jan 2025 12:16:48 +0100 Subject: [PATCH 08/12] ipc: Fixed tf32 size Signed-off-by: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> --- docs/features/communication/ipc/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index dc79c462..45ae068b 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -235,7 +235,7 @@ Float16 Floating ``f16`` ``float16_t`` 2 An IEEE 756 32 Float32 Floating ``f32`` ``float``, ``float32_t`` 4 An IEEE 756 32 bit floating point Float64 Floating ``f64`` ``double``, ``float64_t`` 8 An IEEE 756 64 bit floating point BFloat16 Floating ``bf16`` ``bfloat16_t`` 2 A Google brain float 16 floating point -TFloat32 Floating ``tf32`` ``tfloat32_t`` 2 An NVIDIA tensor float 32 floating point +TFloat32 Floating ``tf32`` ``tfloat32_t`` 4 An NVIDIA tensor float 32 floating point Char String ``char`` ``char32_t`` 4 A unicode codepoint (32 bit) String String ``str`` - n/a A UTF-8 encoded text Handle Reference - - 8 A 64 bit unsigned integer handle From d5819e85a9d78c5449a99709cd0b741eab8af6ed Mon Sep 17 00:00:00 2001 From: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> Date: Wed, 15 Jan 2025 14:40:51 +0100 Subject: [PATCH 09/12] Minor fixes Signed-off-by: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> --- docs/features/communication/ipc/index.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index 45ae068b..25e7a0c2 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -209,7 +209,7 @@ Handles A handle is a numeric value that uniquely refers to an individual element in the communication system. -A specific element in the communication system +The numeric value can safely be transported through the communication system and shall be resolvable into a reference of the original element. Data Types ---------- @@ -538,6 +538,6 @@ The communication framework shall support the following principal security capab - Authenticaion: Unambigous identification of the communication elements, especially producers and consumers of data. - Authorization: A set of rules granting or denying access to communication operations based on the Authentication of participants in the communication framework. - Protection: Means to protect the integrity of data received by consumers. -- Encryption: Means to protext the content of data in transit. +- Encryption: Means to protect the content of data in transit. The definition of requirements for appropriate cryptographic hashing and encryption algorithms is not part of this document. From 5d31d5b5b9ed6c3dbb5defb950ac6ee6943a38e2 Mon Sep 17 00:00:00 2001 From: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> Date: Mon, 20 Jan 2025 16:07:42 +0100 Subject: [PATCH 10/12] ipc: Added open points for CFT CFT to comment on the topics in the initial list Signed-off-by: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> --- docs/features/communication/ipc/index.rst | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index 25e7a0c2..4ab7c322 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -19,6 +19,13 @@ Communication Framework ####################### +* Purpose - why and what do we need this for +* Scope and context diagram +* QoS requirements +* Error handling +* ASIL requirements (hashing, integrity, ...) +* Interoperability + The Communication Framework handles the safe, secure and performant exchange of information between software and/or hardware components of the Score stack. Communication covers information exchange between endpoints residing in @@ -78,6 +85,7 @@ The entirety of connected nodes within a mesh we call *fabric*. From a perspective of safety, a node also encapsulates a single safety domain. Links provide the means for separating safety domains and thus allow for mixed criticality applications. +^^^^ End of Big Picture ^^^^ Names ----- From b4136629089825a4bedd3271e826b296341c4363 Mon Sep 17 00:00:00 2001 From: Nico Hartmann Date: Fri, 24 Jan 2025 15:00:12 +0100 Subject: [PATCH 11/12] com: fixed typos, minor updates --- docs/features/communication/ipc/index.rst | 18 +++++++++--------- 1 file changed, 9 insertions(+), 9 deletions(-) diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index b35cc9d9..69869a68 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -19,7 +19,7 @@ Communication Framework ####################### -The Communication Framework handles the safe, secure and performant exchange of information between software and/or hardware components of the Score stack. +The Communication Framework handles the safe, secure and performant exchange of information between software and/or hardware components of the S-CORE stack. Communication covers information exchange between endpoints residing in @@ -28,7 +28,7 @@ Communication covers information exchange between endpoints residing in * the same or different compute devices, built into * the same or different devices. -In current communication designs (SomeIP, Protobuf, Zenoh) communication on networks is at the centerpoint of considerations. We believe this to be the wrong approach for software defined machines. Network based communication circles around protocols based on a wire based communication paradigm, requiring serialization and segmentation of data. +In current communication designs (SOME/IP, Protobuf, zenoh) communication on networks is at the centerpoint of considerations. We believe this to be the wrong approach for software defined machines. Network based communication circles around protocols based on a wire based communication paradigm, requiring serialization and segmentation of data. Instead, we promote a memory centered core paradigm and put Inter-Process Communication (IPC) into our conceptional focus. This allows for true zero copy design, does not segment or serialize data and fosters significantly higher performance in algorithms. As a bonus, it provides easy gateway mechanisms into serial communication like Ethernet and CAN. @@ -42,15 +42,15 @@ The communication framework defines two central groups of elements: * Information Elements * Infrastructure Elements -While the former carry the information exchanged the latter provide the means through which the information is exchanged. +While the former carry the information exchanged, the latter provide the means through which the information is exchanged. Information Elements ```````````````````` There are three fundamental concepts of information exchange defined. One communication element represents each concept: -* **Topics**: A topic is the information carrier for **data**. A unique Id identifies a topic while a *data type* defines it's memory layout. The topic carries zero or multiple *values*. A value represents a single instances of the data type. See _`Topics`, _`Names`, _`Data Types`. -* **Remote Procedures**: A remote procedure is the information carrier for **execution** progress. A Id handle identifies the remote procedure together with an ordered set of named *parameters*. Each parameter defined by a data type. A caller of a remote procedure can cause it's activation by invoking the remote procedure with passed *arguments*. An argument is a single value instance for a parameter. See _`Remote Procedures`, _`Names`, _`Data Types`. +* **Topics**: A topic is the information carrier for **data**. A unique Id identifies a topic while a *data type* defines its memory layout. The topic carries zero or multiple *values*. A value represents a single instances of the data type. See _`Topics`, _`Names`, _`Data Types`. +* **Remote Procedures**: A remote procedure is the information carrier for **execution** progress. An Id handle identifies the remote procedure together with an ordered set of named *parameters*. Each parameter defined by a data type. A caller of a remote procedure can cause its activation by invoking the remote procedure with passed *arguments*. An argument is a single value instance for a parameter. See _`Remote Procedures`, _`Names`, _`Data Types`. * **Events**: An event is the information carrier for runtime **synchronization**. A unique Id identifies the event. It signals the change of state. There is no data conveyed with the event. See _`Events`, _`Names`. While the Id uniquely identifies an information element within the communication framework, it can also have a *name* as alias to conveniently identify the element. While the Id may not be publicly known, the *name* allows for public lookup. @@ -62,7 +62,7 @@ Infrastructure Elements Infrastructure elements provide the means through which the information exchange executes. We define three fundamental building elements: -* **Endpoints**: Endpoints are both the source and the target of every information exchange in teh communication framework. An endpoint providing information is consequently called a *provider*. With the same logic a *consumer* is an endpoint consuming information. Endpoints have an *Id* that uniquely identifies the endpoint within a node. +* **Endpoints**: Endpoints are both the source and the target of every information exchange in the communication framework. An endpoint providing information is consequently called a *provider*. With the same logic a *consumer* is an endpoint consuming information. Endpoints have an *Id* that uniquely identifies the endpoint with regard to the node it is attached to. * **Nodes**: A node is an entity in the communication system that hosts several endpoints. It is the central element of the communication fabric by connecting endpoints and routing data. Nodes have an *Id* that uniquely identifies the node within a fabric. A node itself is also an endpoint. * **Links**: A link is the fundamental abstraction of a connection between any two nodes. A link conveys information between nodes. @@ -70,7 +70,7 @@ The combination of NodeId and EndpointId we also refer to as *address*. As nodes Nodes and endpoints may also be identified by a *name* that resolves into references to these elements. See _`Names`, _`References`. -Connecting nodes though links creates a mesh of nodes that can mutually exchange information utilizing the above concepts. The boundary of the mesh is at the sole discretion of the deployment and may span from a single application into a connected cloud environment. +Connecting nodes through links creates a mesh of nodes that can mutually exchange information utilizing the above concepts. The boundary of the mesh is at the sole discretion of the deployment and may span from a single application into a connected cloud environment. The entirety of connected nodes within a mesh we call *fabric*. @@ -98,7 +98,7 @@ Forbidden as first character in a name is Further discouraged is the use of the *whitespace* codepoint ``SPACE``: `` `` ````. -Element names prefixed with an underscore ``LOW LINE``: ``_`` ``F`` are regarded to have *private* visibility within the scope they are defined in. While references to private elements are possible, name resolution ony works from within the namespace they are defined in. +Element names prefixed with an underS-CORE ``LOW LINE``: ``_`` ``F`` are regarded to have *private* visibility within the scope they are defined in. While references to private elements are possible, name resolution ony works from within the namespace they are defined in. **Design Note** @@ -192,7 +192,7 @@ A ``use`` clause may end with the wildcard ``ASTERISK``: ``*`` ``A``. This access Two // valid, Two is visible in B -Within a namespace elements from another namespace are visible without an explicit use when a resolving path is given. +Within a namespace, elements from another namespace are visible without an explicit use when a resolving path is given. :: From 9df16a496ff79c90390a8eb7d9365e2f12dc4dcb Mon Sep 17 00:00:00 2001 From: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> Date: Mon, 27 Jan 2025 14:21:47 +0100 Subject: [PATCH 12/12] com: Update docs/features/communication/ipc/index.rst Co-authored-by: Lars Bauhofer <155632781+qor-lb@users.noreply.github.com> Signed-off-by: Nico Hartmann <14351007+HartmannNico@users.noreply.github.com> --- docs/features/communication/ipc/index.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/features/communication/ipc/index.rst b/docs/features/communication/ipc/index.rst index b98dc192..86b1dd85 100644 --- a/docs/features/communication/ipc/index.rst +++ b/docs/features/communication/ipc/index.rst @@ -389,7 +389,7 @@ A topic stores it's values in a queue with a given depth. The queue has a policy - Ignore: New data is ignored. - Overwrite oldest: New data overwrites the oldest element. This makes the topic a ring buffer. -- Replace latest: New data replaces the latest element. This makes the latest update always available. +- Replace latest: New data replaces the latest element, ensuring that the most recent update is always available. A typical use case is when you need to have the newest value at all times while still keeping track of the oldest value that has not been processed. - Error: Writing new data to the topic raises an error at the subscriber (and potentially in the topic) Namespace and Name