Discrepancies in the behavior of test runners for Mapping and Service

[akphi]

Mapping Test Runner vs Service Test Runner

Mapping Test

Mapping tests are currently run within Studio and SDLC pipeline. We haven't examined the pipeline mapping test runner, but for Studio, we do it by matching the hash of the actual data (created by building a test runtime and execute using the execute API) vs. the expected data.

Service Test

Service tests are currently run using doTest api call method in engine with test data and an assert lambda returning a boolean. This assert function uses the equalJsonStrings function in Pure.

Discrepancies

Discrepancies arise when the same protocol definition of the service expected data and the mapping expected data produces different test results. The service test fails with the below while the mapping test passes. Notice both have been escaped to \\'DATE_COL\\' in the protocol JSON. Studio is trying to make sure for either mapping test or service test, the assertion data strings are the same. However, as stated, this same string causes different response in the test runners. QUESTION: Is this is the intended behavior, should we fix either to match the other?

Now, one problem with finos/legend-engine is related to the general strategy we handle STRING fields in our grammar parsers. In certain places, the STRING fields mean JSON strings and we handle the escaping differently (see the pair of methods fromGrammarString() and convertString()). For MappingTest, the ExpectedOutputMappingTestAssert.expectedOutput is understood to be JSON string and for service assertion lambda, this is not the case, because the string value is wrapped in a CString and would not be interpreted as a JSON string. If we use the same string for both, we would produce different grammars!

So the question is, should we do anything about this behavior? This deserves its own Github issue, but I'm starting to think we can use base64 for this.

Model Grammar

###FlatData
FlatData model::FlatDataStore
{
  section default: DelimitedWithHeadings
  {
    scope.untilEof;
    delimiter: ',';
    quoteChar: '"';
    nullString: '';

    Record
    {
      DATE_COL: DATE(format='MM/dd/yyyy', optional);
    }
  }
}


###Service
Service model::PersonService
{
  pattern: '/a4a67e91-1f4f-4226-9949-43f872b8c8a8';
  documentation: '';
  autoActivateUpdates: true;
  execution: Single
  {
    query: |model::Person.all()->graphFetchChecked(#{model::Person{DOB}}#)->serialize(#{model::Person{DOB}}#, ^meta::pure::graphFetch::execution::AlloySerializationConfig(typeKeyName='@type' , removePropertiesWithNullValues=true));
    mapping: model::PersonMapping;
    runtime:
    #{
      mappings:
      [
        model::PersonMapping
      ];
      connections:
      [
        model::FlatDataStore:
        [
          connection_1:
          #{
            FlatDataConnection
            {
              store: model::FlatDataStore;
              url: 'executor:default';
            }
          }#
        ]
      ];
    }#;
  }
  test: Single
  {
    data: 'DATE_COL\r\n ,0001-01-01';
    asserts:
    [
      { [], res: meta::pure::mapping::Result[1]|$res.values->toOne()->toString()->equalJsonStrings('{"defects":[],"source":{"defects":[{"message":"Failed to read \'DATE_COL\' with value:  ,0001-01-01, error: ParseException Unparseable date: \\" ,0001-01-01\\"","enforcementLevel":"Error","ruleType":"InvalidInput","ruleDefinerPath":"model::FlatDataStore","path":[]}],"source":{"number":1,"lineNumber":2,"record":" ,0001-01-01","recordValues":[{"address":"DATE_COL","rawValue":" ,0001-01-01"}]},"value":{"typeName":"model::FlatDataStore.default.default","values":[]}},"value":{}}') }
    ];
  }
}


###Pure
Class model::Person
{
  DOB: StrictDate[0..1];
}


###Mapping
Mapping model::PersonMapping
(
  *model::Person[accounts_classA]: FlatData
  {
    ~src model::FlatDataStore.default
    DOB: $src['DATE_COL']
  }

  MappingTests
  [
    test_1
    (
      query: |model::Person.all()->graphFetchChecked(#{model::Person{DOB}}#)->serialize(#{model::Person{DOB}}#);
      data:
      [
        <FlatData, model::FlatDataStore, 'DATE_COL\r\n ,0001-01-01'>
      ];
      assert: '{"defects":[],"source":{"defects":[{"id":null,"externalId":null,"message":"Failed to read \'DATE_COL\' with value:  ,0001-01-01, error: ParseException Unparseable date: \" ,0001-01-01\"","enforcementLevel":"Error","ruleType":"InvalidInput","ruleDefinerPath":"model::FlatDataStore","path":[]}],"source":{"number":1,"lineNumber":2,"record":" ,0001-01-01","recordValues":[{"address":"DATE_COL","rawValue":" ,0001-01-01"}]},"value":{"typeName":"model::FlatDataStore.default.default","values":[]}},"value":{"DOB":null}}';
    )
  ]
)

Service protocol JSON vs. Mapping protocol JSON

{
  "_type": "service",
  "autoActivateUpdates": true,
  "documentation": "",
  "execution": {
    "_type": "pureSingleExecution",
    "func": {
      "_type": "lambda",
      "body": [
        {
          "_type": "func",
          "function": "serialize",
          "parameters": [
            {
              "_type": "func",
              "function": "graphFetchChecked",
              "parameters": [
                {
                  "_type": "func",
                  "function": "getAll",
                  "parameters": [
                    {
                      "_type": "packageableElementPtr",
                      "fullPath": "model::Person"
                    }
                  ]
                },
                {
                  "_type": "rootGraphFetchTree",
                  "class": "model::Person",
                  "subTrees": [
                    {
                      "_type": "propertyGraphFetchTree",
                      "parameters": [],
                      "property": "DOB",
                      "subTrees": []
                    }
                  ]
                }
              ]
            },
            {
              "_type": "rootGraphFetchTree",
              "class": "model::Person",
              "subTrees": [
                {
                  "_type": "propertyGraphFetchTree",
                  "parameters": [],
                  "property": "DOB",
                  "subTrees": []
                }
              ]
            },
            {
              "_type": "func",
              "function": "new",
              "parameters": [
                {
                  "_type": "packageableElementPtr",
                  "fullPath": "meta::pure::graphFetch::execution::AlloySerializationConfig"
                },
                {
                  "_type": "string",
                  "multiplicity": {
                    "lowerBound": 1,
                    "upperBound": 1
                  },
                  "values": []
                },
                {
                  "_type": "collection",
                  "multiplicity": {
                    "lowerBound": 1,
                    "upperBound": 1
                  },
                  "values": [
                    {
                      "_type": "keyExpression",
                      "add": false,
                      "expression": {
                        "_type": "string",
                        "multiplicity": {
                          "lowerBound": 1,
                          "upperBound": 1
                        },
                        "values": ["@type"]
                      },
                      "key": {
                        "_type": "string",
                        "multiplicity": {
                          "lowerBound": 1,
                          "upperBound": 1
                        },
                        "values": ["typeKeyName"]
                      }
                    },
                    {
                      "_type": "keyExpression",
                      "add": false,
                      "expression": {
                        "_type": "boolean",
                        "multiplicity": {
                          "lowerBound": 1,
                          "upperBound": 1
                        },
                        "values": [true]
                      },
                      "key": {
                        "_type": "string",
                        "multiplicity": {
                          "lowerBound": 1,
                          "upperBound": 1
                        },
                        "values": ["removePropertiesWithNullValues"]
                      }
                    }
                  ]
                }
              ]
            }
          ]
        }
      ],
      "parameters": []
    },
    "mapping": "model::PersonMapping",
    "runtime": {
      "_type": "engineRuntime",
      "connections": [
        {
          "store": {
            "path": "model::FlatDataStore",
            "type": "STORE"
          },
          "storeConnections": [
            {
              "connection": {
                "_type": "FlatDataConnection",
                "element": "model::FlatDataStore",
                "url": "executor:default"
              },
              "id": "connection_1"
            }
          ]
        }
      ],
      "mappings": [
        {
          "path": "model::PersonMapping",
          "type": "MAPPING"
        }
      ]
    }
  },
  "name": "PersonService",
  "owners": [],
  "package": "model",
  "pattern": "/a4a67e91-1f4f-4226-9949-43f872b8c8a8",
  "test": {
    "_type": "singleExecutionTest",
    "asserts": [
      {
        "assert": {
          "_type": "lambda",
          "body": [
            {
              "_type": "func",
              "function": "equalJsonStrings",
              "parameters": [
                {
                  "_type": "func",
                  "function": "toString",
                  "parameters": [
                    {
                      "_type": "func",
                      "function": "toOne",
                      "parameters": [
                        {
                          "_type": "property",
                          "parameters": [
                            {
                              "_type": "var",
                              "name": "res"
                            }
                          ],
                          "property": "values"
                        }
                      ]
                    }
                  ]
                },
                {
                  "_type": "string",
                  "multiplicity": {
                    "lowerBound": 1,
                    "upperBound": 1
                  },
                  "values": [
                    "{\"defects\":[],\"source\":{\"defects\":[{\"message\":\"Failed to read 'DATE_COL' with value:  ,0001-01-01, error: ParseException Unparseable date: \\\" ,0001-01-01\\\"\",\"enforcementLevel\":\"Error\",\"ruleType\":\"InvalidInput\",\"ruleDefinerPath\":\"model::FlatDataStore\",\"path\":[]}],\"source\":{\"number\":1,\"lineNumber\":2,\"record\":\" ,0001-01-01\",\"recordValues\":[{\"address\":\"DATE_COL\",\"rawValue\":\" ,0001-01-01\"}]},\"value\":{\"typeName\":\"model::FlatDataStore.default.default\",\"values\":[]}},\"value\":{}}"
                  ]
                }
              ]
            }
          ],
          "parameters": [
            {
              "_type": "var",
              "class": "meta::pure::mapping::Result",
              "multiplicity": {
                "lowerBound": 1,
                "upperBound": 1
              },
              "name": "res"
            }
          ]
        }
      }
    ],
    "data": "DATE_COL\r\n ,0001-01-01"
  }
}

{
  "_type": "mapping",
  "classMappings": [
    {
      "_type": "flatData",
      "class": "model::Person",
      "flatData": "model::FlatDataStore",
      "id": "accounts_classA",
      "propertyMappings": [
        {
          "_type": "flatDataPropertyMapping",
          "property": {
            "class": "model::Person",
            "property": "DOB"
          },
          "source": "accounts_classA",
          "transform": {
            "_type": "lambda",
            "body": [
              {
                "_type": "property",
                "parameters": [
                  {
                    "_type": "var",
                    "name": "src"
                  },
                  {
                    "_type": "string",
                    "multiplicity": {
                      "lowerBound": 1,
                      "upperBound": 1
                    },
                    "values": ["DATE_COL"]
                  }
                ],
                "property": "oneString"
              }
            ],
            "parameters": []
          }
        }
      ],
      "root": true,
      "sectionName": "default"
    }
  ],
  "enumerationMappings": [],
  "includedMappings": [],
  "name": "PersonMapping",
  "package": "model",
  "tests": [
    {
      "assert": {
        "_type": "expectedOutputMappingTestAssert",
        "expectedOutput": "{\"defects\":[],\"source\":{\"defects\":[{\"id\":null,\"externalId\":null,\"message\":\"Failed to read \\'DATE_COL\\' with value:  ,0001-01-01, error: ParseException Unparseable date: \\\" ,0001-01-01\\\"\",\"enforcementLevel\":\"Error\",\"ruleType\":\"InvalidInput\",\"ruleDefinerPath\":\"model::FlatDataStore\",\"path\":[]}],\"source\":{\"number\":1,\"lineNumber\":2,\"record\":\" ,0001-01-01\",\"recordValues\":[{\"address\":\"DATE_COL\",\"rawValue\":\" ,0001-01-01\"}]},\"value\":{\"typeName\":\"model::FlatDataStore.default.default\",\"values\":[]}},\"value\":{\"DOB\":null}}"
      },
      "inputData": [
        {
          "_type": "flatData",
          "data": "DATE_COL\r\n ,0001-01-01",
          "sourceFlatData": {
            "path": "model::FlatDataStore",
            "type": "STORE"
          }
        }
      ],
      "name": "test_1",
      "query": {
        "_type": "lambda",
        "body": [
          {
            "_type": "func",
            "function": "serialize",
            "parameters": [
              {
                "_type": "func",
                "function": "graphFetchChecked",
                "parameters": [
                  {
                    "_type": "func",
                    "function": "getAll",
                    "parameters": [
                      {
                        "_type": "packageableElementPtr",
                        "fullPath": "model::Person"
                      }
                    ]
                  },
                  {
                    "_type": "rootGraphFetchTree",
                    "class": "model::Person",
                    "subTrees": [
                      {
                        "_type": "propertyGraphFetchTree",
                        "parameters": [],
                        "property": "DOB",
                        "subTrees": []
                      }
                    ]
                  }
                ]
              },
              {
                "_type": "rootGraphFetchTree",
                "class": "model::Person",
                "subTrees": [
                  {
                    "_type": "propertyGraphFetchTree",
                    "parameters": [],
                    "property": "DOB",
                    "subTrees": []
                  }
                ]
              }
            ]
          }
        ],
        "parameters": []
      }
    }
  ]
}

[stale]

This issue has been automatically marked as stale. If this issue is still affecting you, please leave any comment (for example, "bump"), and we'll keep it open. We are sorry that we haven't been able to prioritize it yet. If you have any new additional information, please include it with your comment!

[finos-admin]

This issue is stale because it has been open for 30 days with no activity. Please remove stale label or add any comment to keep this open. Otherwise this will be closed in 5 days.

[hardikmaheshwari]

I believe this should be handled once we mapping is integrated into a testable mechanism - #866.
Service is already using testable mechanism

[akphi]

@hardikmaheshwari Honestly, I don't recall much the context around this. It has to do with A worksaround we had to add in Studio as well. But I think what you said makes sense. Now that we use test dataand store data as true strings and nolonger have a test runner within Studio, I believe this problem should go away naturally.

[finos-admin]

This issue is stale because it has been open for 30 days with no activity. Please remove stale label or add any comment to keep this open. Otherwise this will be closed in 5 days.

[finos-admin]

This issue was closed because it has been inactive for 35 days. Please re-open if this issue is still relevant.