Use VSS as mapping format for dbcfeeder

.github/workflows/kuksa_dbc_feeder.yml

@@ -68,9 +68,24 @@ jobs:
           tags: ${{ github.sha }}
           labels: |
             org.opencontainers.image.source=https://github.com/${{steps.repository-name-adjusted.outputs.lowercase}}
+
       - name: Temporarily save Docker image
         uses: actions/upload-artifact@v3
         with:
           name: Container image
           path: ${{github.workspace}}/dbc2val.tar
           retention-days: 1
+
+      - name: Run dbc tests
+        run: |
+          cd dbc2val
+          pip3 install --no-cache-dir -r requirements-dev.txt
+          python -m pytest
+
+      - name: Run pylint (but accept errors for now)
+        run: |
+          cd dbc2val
+          # First just show, never fail
+          pylint --exit-zero dbcfeeder.py dbcfeederlib test
+          # Fail on errors and above
+          pylint -E dbcfeeder.py dbcfeederlib test

dbc2val/CHANGELOG.md

@@ -0,0 +1,8 @@
+# Changelog for dbc2val
+
+This file lists important changes to dbc2cal
+
+## Refactoring and changed configuration format (2023-02)
+
+Feeder refactored and new mapping format based on VSS introduced, see [documentation](mapping.md).
+Old mapping format no longer supported.

dbc2val/Dockerfile

@@ -11,6 +11,7 @@
 # * SPDX-License-Identifier: Apache-2.0
 # ********************************************************************************/
 
+
 # Build stage, to create a Virtual Environent
 FROM --platform=$TARGETPLATFORM python:3.9-slim-bullseye as builder
 
@@ -32,15 +33,16 @@ RUN /opt/venv/bin/python3 -m pip install --upgrade pip \
 
 RUN pip3 install wheel scons && pip3 install pyinstaller patchelf==0.17.0.0 staticx
 
-RUN pyinstaller --clean -F -s dbcfeeder.py
+# By default we use certificates and tokens from kuksa_certificates, so they must be included
+RUN pyinstaller --collect-data kuksa_certificates --clean -F -s dbcfeeder.py
 #   --debug=imports
 
 WORKDIR /dist
 RUN staticx dbcfeeder dbcfeeder-exe
 
 WORKDIR /data
 COPY ./config/* ./config/
-COPY ./*.yml ./*.dbc ./candump*.log ./
+COPY ./*.json ./*.dbc ./candump*.log ./
 
 # Runner stage, to copy in the virtual environment and the app
 FROM scratch

dbc2val/Readme.md

@@ -1,8 +1,8 @@
 # Feeder CAN for real CAN interface or dumpfile replay
 
 This is an a DBC CAN feeder for the [KUKSA.val](https://github.com/eclipse/kuksa.val) server and databroker. The basic operation is as follows:
-The feeder connects to a socket CAN interface and reads raw CAN data, that will be parsed based on a DBC file. The mapping file describes which DBC signal
-should be matched to which part in the VSS tree. The respective data point can then be sent to the kuksa.val databroker or kuksa.val server.
+The feeder connects to a socket CAN interface and reads raw CAN data, that will be parsed based on a DBC file.
+The mapping file (called `vss_dbc.json` in the picture) describes mappings between VSS signals and DBC signals. The respective data point can then be sent to the kuksa.val databroker or kuksa.val server.
 It is also possible to replay CAN dumpfiles without the SocketCAN interface being available, e.g. in a CI test environment.
 See "Steps for a local test with replaying a can dump file"
 
@@ -22,9 +22,9 @@ See "Steps for a local test with replaying a can dump file"
 |                 |         +--------------+       |
 +-----------------+                 |              |    +-------------+
                                     |              |    |             |
-                             +-------------+       |--->|  Databroker |
-                             | mapping.yml |            |             |
-                             +-------------+            +-------------+
+                            +--------------+       |--->|  Databroker |
+                            | vss_dbc.json |            |             |
+                            +--------------+            +-------------+
 
 ```
 
@@ -55,6 +55,8 @@ $ pip install -r requirements.txt
 
 2. Start the can player
 
+_This is only needed if you want feed data from a dumpfile!_
+
 ```console
 $ ./createvcan.sh vcan0
 $ canplayer vcan0=elmcan -v -I candump.log -l i -g 1
@@ -108,7 +110,7 @@ A smaller excerpt from the above sample, with less signals.
 | canport                       | -               | [can].port           | CAN_PORT                      | `--canport`           | Read from this CAN interface                                                                            |
 | use-j1939                     | False           | [can].j1939          | USE_J1939                     | `--use-j1939`         | Use J1939                                                                                               |
 | use-socketcan                 | False           | -                    | -                             | `--use-socketcan`     | Use SocketCAN (overriding any use of --dumpfile)                                                        |
-| mapping                       | mapping.yml     | [general].mapping    | MAPPING_FILE                  | `--mapping`           | Mapping file used to map CAN signals to databroker datapoints. Take a look on usage of the mapping file |
+| mapping                       | vss_dbc.json    | [general].mapping    | MAPPING_FILE                  | `--mapping`           | Mapping file used to map CAN signals to databroker datapoints. Take a look on usage of the mapping file |
 | server-type                   | kuksa_val_server | [general].server_type | SERVER_TYPE                 | `--server-type`       | Which type of server the feeder should connect to (kuksa_val_server or kuksa_databroker |
 | DAPR_GRPC_PORT                | -               | -                    | DAPR_GRPC_PORT                | -                     | Override broker address & connect to DAPR sidecar @ 127.0.0.1:DAPR_GRPC_PORT                            |
 | VEHICLEDATABROKER_DAPR_APP_ID | -               | -                    | VEHICLEDATABROKER_DAPR_APP_ID | -                     | Add dapr-app-id metadata                                                                                |
@@ -197,76 +199,11 @@ Vehicle.OBD.Speed: 12.00
 > **Note**
 > To end the subscription currently you have to stop the client cli via 'quit' and Enter.
 
-## Using mapping.yml
-
-Please replace the values xxx with our content for a new signal template:
-
-```yaml
-xxx: # CAN signal name taken from the used dbc file
-  minupdatedelay: xxx # update interval of the signal in ms, if no delay given default is 1000ms
-  targets:
-    xxx: {} # Name of the VSS signal
-      vss: # vss definition
-        datatype: xxx # type of the data
-        type: xxx # type of the value
-        unit: xxx # unit of the value
-        description: # description of the value
-      transform: {}  # which (math) transformations to apply to the signal
-```
-
-example:
-
-```yaml
-UIspeed_signed257: # CAN signal name taken from the used dbc file
-  minupdatedelay: 100 # 100ms update interval of the signal
-  targets:
-    Vehicle.OBD.Speed:  # Name of the VSS signal
-      vss: # vss definition
-        datatype: float # type of the data
-        type: sensor # type of the value
-        unit: km/h # unit of the value
-        description: vehicle speed # description of the value
-```
-
-Please note, the minimal set to map a signal for KUKSA.val server, where all data model knwoledge is in the data server itself,  is just a CAN signal name and at least one target.
-For KUKSA.val databroker, an architecture that requires Clients to know somehting about the VSS model, a CAN signal name, a target and at least a datatype in the vss section is required. In most cases, you would probably require a `transform`, unless the DBC already describes the exact semantics and/or scaling of a VSS signal.
-
-### Mapping examples
-
-
-```yaml
-VCFRONT_brakeFluidLevel:
-  minupdatedelay: 1000
-  targets:
-    Vehicle.Chassis.Axle.Row1.Wheel.Left.Brake.FluidLevelLow:
-      transform:
-        fullmapping:
-          LOW: "true"
-          NORMAL: "false"
-    Vehicle.Chassis.Axle.Row1.Wheel.Right.Brake.FluidLevelLow:
-      transform:
-        fullmapping:
-          LOW: "true"
-          NORMAL: "false"
-```
-Here the same DBC signal `VCFRONT_brakeFluidLevel`is mapped to different VSS path. Also _fullmapping_ transform is applied. The value `LOW` from the DBC is mapped to the value `true` in the VSS path and `NORMAL`is mapped to `false`. In the _fullmapping_ transform, if no match is found, the value will be ignored. There also exists a _partialmapping_ transform, which works similarly, with the difference, that if no match is found the value from the DBC will be written as-is to KUKSA.val.
-
-Another transform is the _math_ transform
-
-```yaml
-VCLEFT_mirrorTiltXPosition:
-   minupdatedelay: 100
-   targets:
-    Vehicle.Body.Mirrors.Left.Pan:
-      transform: #scale 0..5 to -100..100
-        math: floor((x*40)-100)
-```
-
-This can be used if the scale of a signal as described in the DBC is not compatible with the VSS model.  The value - with all transforms described in the DBC -  is used as `x` on the formula given by the _math_ transform. For available operators, functions and constants supported by the _math_ transform check https://pypi.org/project/py-expression-eval/.
-
-
-Take a look at the [mapping.yml](./mapping.yml) to see more examples.
+## Mapping file
 
+The mapping file describes mapping between VSS signals and DBC signals.
+It shall be a JSON file with VSS syntax with metadata for dbc information.
+Please see [mapping documentation](mapping.md) for more information.
 
 ## Logging
 

dbc2val/config/dbc_feeder.ini

@@ -14,7 +14,7 @@
 # default kuksa_val_server
 server_type = kuksa_val_server
 # VSS mapping file
-mapping = mapping.yml
+mapping = vss_dbc.json
 
 [kuksa_val_server]
 # kuksa_val_server IP address or host name