More Precise Error Validation for allOf, anyOf, oneOf

[erichulburd]

Given the following schema:

{
  "type": "object",
  "required": [
    "numberOfModules"
  ],
  "properties": {
    "numberOfModules": {
      "allOf": [
        {
          "not": {
            "type": "integer",
            "minimum": 38
          }
        },
        {
          "not": {
            "type": "integer",
            "maximum": 37,
            "minimum": 25
          }
        },
        {
          "not": {
            "type": "integer",
            "maximum": 24,
            "minimum": 12
          }
        }
      ]
    }
  }
}

When I validate, I get a high level error description saying allOf failed:

2.3.3 :009 > schemer.validate({ 'numberOfModules' => 32 }).to_a
 => [{"data"=>21, "schema"=>{"x-notification"=>"Reduce system size - Deselect modules to reduce the cost of a main service panel upgrade.", "allOf"=>[{"not"=>{"x-cost"=>7500, "type"=>"integer", "minimum"=>38}}, {"not"=>{"x-cost"=>5000, "type"=>"integer", "maximum"=>37, "minimum"=>25}}, {"not"=>{"x-cost"=>2500, "type"=>"integer", "maximum"=>24, "minimum"=>12}}, {"x-error"=>"Number of modules cannot be negative.", "type"=>"integer", "minimum"=>0}]}, "pointer"=>"/numberOfModules", "type"=>"allOf"}]

https://ajv.js.org/ gives a more specific error path:

> ajv.validate(s.design, { numberOfModules: 32 })
false
> ajv.errors
[ { keyword: 'maximum',
    dataPath: '.numberOfModules',
    schemaPath: '#/properties/numberOfModules/then/then/else/maximum',
    params: { comparison: '<=', limit: 24, exclusive: false },
    message: 'should be <= 24' } ]

It's nice to have the schema path for the lowest level of failure. I'm potentially open to working on a PR, but first wanted to know your thoughts on whether:

• You are open to moving in this direction.
• Moving in this direction would open up a whole can of worms that would require a large effort. For instance, if we made error paths more precise of allOf - would we then need to work on that for anyOf, oneOf, if, then, etc - or could we add these piecemeal?

[davishmcclurg]

@erichulburd sorry, I didn't see this until today. I'll take a look this evening and get back to you.

[davishmcclurg]

I pushed a branch (08b781f) with an attempt at addressing this. It adds a subschemas key to the error hashes that contains errors for nested schemas in allOf, anyOf, oneOf, and contains. I didn't implement them as top-level errors because it seemed strange to return all errors for things like anyOf, which only require one of the subschemas to be valid.

There are a couple issues with my approach:

• It's awkward to get errors for subschemas (https://github.com/davishmcclurg/json_schemer/compare/subschemas#diff-05e8376a74a8d91443c8d001e944703cR84)
• There's currently no pointer for the subschema path

I might be able to clean things up by wrapping the error hashes into an object with some helpers.

I'm going to think about this some more and look closer at what ajv does. Let me know if you've got any feedback/suggestions.

[erichulburd]

@davishmcclurg Thanks - I think AJV takes a bubble up approach. Basically it seems to only return errors for low level validations on primitives, includes the path for the violation, and does not return errors for schema objects.

Let me take a closer look at your branch. I may have time in the next couple of weeks to put a PR together.

[davishmcclurg]

I added another branch that I think behaves more like ajv: e5ec222
I used your example in one of the tests so you can see what the output would be.

[davishmcclurg]

json-schema-org/json-schema-spec#396
json-schema-org/json-schema-spec#643

[bilby91]

Was this implemented ? I'm using this awesome gem for https://github.com/bilby91/ddsl and one of the main pain points right now is the lack of clear error messages on referenced schemas using the of familily

[bilby91]

@davishmcclurg ,

I spend some time yesterday integrating ddsl with the commits under the branch more-errors. I think this is definitely an improvement over the previous implementation.

Is there any plan on getting that merged ?

[davishmcclurg]

Hi @bilby91! Thanks for taking the time to try that branch. I put that work on hold until the json schema organization figured out what they wanted to do about output formatting. They merged something recently (json-schema-org/json-schema-spec#679) and I took a stab at implementing it, but haven't finished it due to some confusion about the spec. Hopefully I can get back to it soon.

Would the basic output structure described in the spec be good enough for you?

Basic - Provides validation information in a flat list structure.

This is their example:

{
  "valid": false,
  "errors": [
    {
      "keywordLocation": "#",
      "instanceLocation": "#",
      "error": "A subschema had errors."
    },
    {
      "keywordLocation": "#/items/$ref",
      "absoluteKeywordLocation":
        "http://example.com/polygon#/definitions/point",
      "instanceLocation": "#/1",
      "error": "A subschema had errors."
    },
    {
      "keywordLocation": "#/items/$ref/required",
      "absoluteKeywordLocation":
        "http://example.com/polygon#/definitions/point/required",
      "instanceLocation": "#/1",
      "error": "Required property 'y' not found."
    },
    {
      "keywordLocation": "#/items/$ref/additionalProperties",
      "absoluteKeywordLocation":
        "http://example.com/polygon#/definitions/point/additionalProperties",
      "instanceLocation": "#/1/z",
      "error": "Additional property 'z' found but was invalid."
    },
    {
      "keywordLocation": "#/minItems",
      "instanceLocation": "#",
      "error": "Expected at least 3 items but found 2"
    }
  ]
}

[davishmcclurg]

@bilby91 I released something similar to more-errors in 0.2.0