SCORE Initial Architecture Workshop

[HartmannNico]

SCORE Initial Architecture Workshop

Hi Team, as discussed this discussion is in preparation of the upcoming architecture workshop.
Please name up to two participants (and their github names) from your organization. The nominees should not only have architecture knowledge but also implementation background.

Schedule

Start: 03.12.2024, 10:00
End: 05.12.2024, 15:00

Location

ESRLabs: Balanstr. 73 Haus 11, 4. OG, 81541 München

Participants

BMW: Markus Bechter @ramceb , Jaakko Aho
ESRLabs: Felix Obenhuber, Martin Thiede, Gerd Schäfer, Johannes Glamsch
ETAS: Arvid Sievert, Zoran Cutura
Mercedes: Fabian Frie, Alexander Warg, Thilo Schmitt
Qorix: Nico Hartmann @HartmannNico, Lars Bauhofer @qor-lb

Agenda

• Day 1

  • Agreement on agenda

  • Introduction (1000-1130)

    • Setting the Scene
    • Requirements
    • Targets

  • Core Architecture (1230-1800)

    • Principles & Paradigms
    • Building Blocks
    • Scalability

• Day 2

  • Architecture Details Core (0830-1130)
    • Core
    • Memory
    • Data
    • Communication
    • Runtime
  • ADAS application framework
    • Requirements and architecture
    • Breakdown of requirements towards the platform
  • Architecture Details Core Services (1230-1800)
    • Log & Trace
    • Persistency
    • State & Execution
    • Diagnostics
    • Health
    • ...

• Day 3

  • Implementation (830-1130)
    • Guidelines
    • Programming Languages
    • Existing Contributions
  • Roadmap & Resources (1230-1500)
  • Wrap-Up & Next Steps

Topics to cover

• Multiple OS: Linux, QNX
• Multi Language: C++, Rust
• Performance Communication: Zero Copy IPC
• Assumptions on OS
• HW Accelerators
• HW Peripherals
• Resources & Namespaces
• API & Access Management
• Use-case scoping (ADAS vs. Connectivity)

[qor-lb]

Day 1

Participants

• Markus Bechter @ramceb
• Nico Hartmann @HartmannNico
• Fabian Frie
• Lars Bauhofer @qor-lb
• Zoran Cutura
• Arvid Sievert
• Thilo Schmitt @thilo-schmitt
• Gunnar Piel
• Armin Bartsch @armin-acn
• Felix Obenhuber @flxo
• Gerd Schäfer
• Martin Thiede (online) @mthiede-acn2
• Alexander Warg

Meeting Minutes

Introduction (11:00 - 14:00)

• The relationship between the Core Stack and IVI frameworks was discussed. While their coexistence is recognized, it falls outside the current scope of the SCORE project.
• The Minimum Viable Product (MVP) will focus on Advanced Driver Assistance Systems (ADAS), including the required application framework.
• Although the MVP targets ADAS, it does not explicitly exclude integration with Telematic Control Units (TCUs) and other specialized frameworks in the future.

Core Architecture (14:00 - 17:30)

• Development Approach:

  • Application development should not rely on a code generator. Eventually, higher-level abstractions and IDLs (Interface Definition Languages) may be introduced for automated interface generation.

• Fabric Capabilities:

  • Both applications and system services should leverage the same foundational communication primitives (e.g., Topics, RPC, Notifications, Events/Virtual Interrupts).
  • The solution should clearly separate interface definitions, their implementation, and deployment aspects.

• Reusability:

  • Core elements of the application framework are expected to be largely reusable when developing and implementing system services.

Referenced Material

• 20241203.SCORE.Architecture.Workshop

Revised Agenda Day 2

• ADAS application framework (0830 - ...)
  • Requirements and architecture
  • Breakdown of requirements towards the platform
• Architecture Details Core (... - 1130)
  • Core
  • Memory
  • Data
  • Communication
  • Runtime
• Definition Breakout Sessions Day 2 (1230 - 1300)
  • Zero Copy IPC
  • HW Accelerators: NPU, TPU, GPU
  • Deployment & Integration - Resources & Namespaces (Container): Isolation, Resource allocation - FS, CPU, ..., API & Access Management
• Architecture Details Core Services (1300 - 1800)
  • Log & Trace
  • Persistency
  • State & Execution
  • Diagnostics
  • Health
  • ...

[qor-lb]

Day 2

Participants

• Markus Bechter @ramceb
• Nico Hartmann @HartmannNico
• Fabian Frie
• Lars Bauhofer @qor-lb
• Zoran Cutura
• Arvid Sievert
• Thilo Schmitt @thilo-schmitt
• Armin Bartsch @armin-acn
• Felix Obenhuber @flxo
• Gerd Schäfer
• Alexander Warg

Meeting Minutes

ADAS Application Framework (08:30 - 11:00)

• Terminology & Concepts:

  • A glossary is needed to define key terms such as Application, Function, Activity, and Task Chain clearly.

• Thread Management & Concurrency:

  • Consideration of using a “Grand Central Dispatch” or thread pool model to manage activities.
  • Discussion of activity thread pinning to ensure predictable execution environments.

• Activity Dependencies & Error Handling:

  • Support for AND/OR logic in activity dependencies was explored to enable flexible execution paths.
  • Local error handling within the Fixed Execution Order (FEO) Framework was discussed, including the possibility of an error task chain to manage fault scenarios.
  • Monitoring timing conditions within a task chain is desirable for ensuring performance and reliability.

• Language Interoperability & Data Sharing:

  • Mixing multiple programming languages (e.g., Rust and C++) in a single task chain was considered.
  • Activities in different languages must agree on a well-defined data type structure.
  • Activities should be fully implemented in their chosen language rather than partially wrapped APIs.
  • Activities should not share state directly; state management should be explicit and isolated.

• Task Chain Structure & Data Passing:

  • Task chains may share data or trigger each other. This raises the question of whether multiple entry points in a single chain effectively create one larger chain or multiple related chains.
  • Data sharing across task chains should be enabled, potentially modeled as a Sender-Receiver pattern.
  • Consider improving the naming for Sender/Receiver Activities (e.g., Entry/Exit Activity).

• Tooling & Configuration Support:

  • A desire for tooling to define and configure task chains was expressed, including validation of configurations, detection of critical paths, and potentially static code analysis for a “config-as-code” approach.

• State Management & Replay Use Cases:

  • The FEO should support the definition of states for replay scenarios. Activities can leverage this but are not mandated to do so.

• Performance & Profiling:

  • Enhanced capabilities for tracing and profiling performance are desired.

• Definition of “Container” in SCORE:

  • The concept of a “container” was discussed, potentially as a deployable unit or a resource/namespace organization element. This requires further definition. Proposal to be provided by Alexander Warg.

Architecture Details Core (13:00 - 18:00)

• Application Framework & Orchestration:

  • The architecture should support the previously discussed orchestration concepts, integrating application frameworks with clear execution and communication patterns.

• Communication (Com) Layer & IPC Considerations:

  • Communication primitives should include topics (data-centric, decoupled from service instances) and RPC (instance-centric and groupable by objects/services).
  • Network-level details are out of scope but should be considered in context for future integration.

• Memory Management & Shared Memory (SHM):

  • Multi-language support and shared memory allocation strategies were discussed, including OS-level procurement and preparation of SHM for communication.
  • Potential differentiation between setting up memory (QM - Quality Management) and validating the setup (ASIL - Automotive Safety Integrity Level) before starting communication.
  • Ideally, separate shared memory regions per topic and possibly for control channels to avoid interference and false cache sharing.
  • Support for dynamic payloads in topics was considered, referencing virtio’s single-producer/single-consumer model as a good fit for RPC.
  • Decouple topic definitions from memory regions, allowing multiple topics to share one SHM segment or assign dedicated SHM segments as needed.

• Data Types & Language Interoperability:

  • Zero-copy support requires well-defined data types (primitive, struct, array, vector, hash map, blob).
  • Descriptions of data types and memory layouts should facilitate code generation for multiple languages, ensuring native “look and feel.”
  • Use of an Interface Definition Language (IDL) for type description was suggested.
  • Type generation tooling is needed to create consistent memory layouts across languages.

Overall, the meeting established foundational concepts for the ADAS application framework and its orchestration, clarified communication and memory management strategies, and identified key requirements for interoperability, tool support, and performance profiling within the envisioned SCORE architecture.

Referenced Material

• 20241203.SCORE.Architecture.Workshop
• Whiteboard Topic
• Whiteboard RPC
• Whiteboard SHM Queue
• Whiteboard Topic Notation & Data Representation

Proposal Agenda Day 3

• Implementation (830-1130)
  • Guidelines
  • Programming Languages
  • Existing Contributions
• Roadmap & Resources (1230-1400)
• Wrap-Up & Next Steps (1400-1500)

Breakouts for the future (potentially outside of the workshop)

• Record/Replay/States
• HW Accelerators: NPU, TPU, GPU
• Deployment & Integration - Resources & Namespaces (Container): Isolation, Resource allocation - FS, CPU, ..., API & Access Management

[qor-lb]

Day 3

Participants

• Markus Bechter @ramceb
• Nico Hartmann @HartmannNico
• Fabian Frie
• Lars Bauhofer @qor-lb
• Arvid Sievert
• Armin Bartsch @armin-acn
• Felix Obenhuber @flxo
• Gerd Schäfer
• Alexander Warg

Meeting Minutes

Summary of Meeting Outcomes

Implementation (08:30 - 11:30)

• Team Setup & Collaboration:

  • The current workshop participants will form the SCORE Architecture Community.
  • A regular 1-hour weekly sync meeting will be held on Mondays. @ramceb will send invitations.

• Process & Contribution Guidelines:

  • Coding guidelines for C++ and Rust will be established, including adherence to standards like MISRA for safety-critical code.
  • There should be a “playground” environment for proof-of-concept code before it is subjected to the strict ASIL-compliant development process.

• Language & Build Infrastructure:

  • Both C++17 and Rust will be used. Interoperability between processes written in these languages is required.
  • Other language bindings or integrations (e.g., with other languages) will be addressed on a case-by-case basis.
  • Bazel will be used as the primary build infrastructure, with Rust support needing evaluation and integration.
  • The native development environment is Linux.
  • Consideration is being given to allowing local developer workflows with tools like Cargo (for Rust).

• Proof of Concept & Decision-Making:

  • Existing proposals will be profiled to lower the barrier to entry (e.g., a native build system).
  • Architectural decisions should be measured against their conceptual benefits rather than implementation detail.

• Components & Frameworks Under Evaluation:

  • FEO (Fixed Execution Order):

    • Existing frameworks like Flux and Nautilus will be considered and tested to identify missing application use-cases.
    • FEO acts as an orchestrator and integrates with stack components. Over time, components can be refactored or moved as the architecture matures.
    • Next steps include code-based comparison of Nautilus and Flux in the context of FEO.

  • Orchestration (FEO Activities): Runtime, Nautilus, and Flux will be evaluated.

  • IPC & Communication:

    • Topics, shared memory coordination, and zero-copy mechanisms will be examined via solutions like LoLa and IceOryx.
    • Data type representations and shared memory layouts will need to be defined.

  • Logging:

    • Logging guidelines (potentially leveraging ESR Guidelines) must be established.

  • Persistency:

    • Non-volatile storage, key-value stores, and file system abstractions will be considered, but this is not a top priority at the moment.

  • Network Communication:

    • Not a priority for immediate specification. The focus is on ensuring integration capability rather than defining detailed network protocols now.

Roadmap & Resources (12:30 - 14:00)

• MVP Timeline:

  • By March 2025 (03/2025): Code implementations and concept decisions should be completed, independent of formal process and infrastructure.
  • By June 2025 (06/2025): Partial contributions should be available, with requirements and architecture documentation consistent with the code. Safety certification is not mandatory at this stage.

• Community & Next Steps:

  • The architecture community will track progress towards the MVP and discuss developments regularly.
  • The next in-person workshop is planned for around January/February.

• Infrastructure & Repositories:

  • Preparation of demonstration/incubation repositories, one repository per topic, will be led by Johannes.

• Topic Responsibility & Coordination:

  • Topics and their respective participants will be defined before the next Architecture community meeting.
  • Operational start date is targeted for January 13, 2025, with Product Leads/Facilitators nominated by December 9, 2024.

Overall, the meeting clarified the technical direction for implementation, set guidelines for contributions and interoperability, outlined the evaluation of existing frameworks and tools, established a tentative roadmap, and initiated a community-driven approach to maintain momentum towards the SCORE MVP.

Referenced Material

• 20241203.SCORE.Architecture.Workshop