Improve pretty error messages when schema composition constraints are violated

[babelfish]

I have a schema where I'm enforcing rules like:

allOf:
  - anyOf:
      - required: [a]
      - required: [b]
  - oneOf:
      - required: [c]
        not:
          required: [d]
      - required: [d]
        not:
            required: [c]

When writing tests I noticed I get really unhelpful error messages when violating these constraints. If I provide data that includes both c and d, for example, I get the following "pretty" error: root is invalid: error_type=not.

[davishmcclurg]

Hi @babelfish, thanks for taking the time to open an issue. What error message would you like to see in this case? Do you mind providing a pared down schema and example data?

not error messaging is tricky because there isn't a lot to report: the schema validated successfully and it shouldn't have. The new output formats (released in 2.0.0) might be more helpful.

Here's your example using output_format: basic:

{"valid"=>false,
 "keywordLocation"=>"",
 "absoluteKeywordLocation"=>"json-schemer://schema#",
 "instanceLocation"=>"",
 "error"=>"value at root does not match schema",
 "errors"=>
  [{"valid"=>false,
    "keywordLocation"=>"/allOf/1/oneOf",
    "absoluteKeywordLocation"=>"json-schemer://schema#/allOf/1/oneOf",
    "instanceLocation"=>"",
    "error"=>"value at root does not match exactly one `oneOf` schema"},
   {"valid"=>false,
    "keywordLocation"=>"/allOf/1/oneOf/0/not",
    "absoluteKeywordLocation"=>"json-schemer://schema#/allOf/1/oneOf/0/not",
    "instanceLocation"=>"",
    "error"=>"value at root matches `not` schema"},
   {"valid"=>false,
    "keywordLocation"=>"/allOf/1/oneOf/1/not",
    "absoluteKeywordLocation"=>"json-schemer://schema#/allOf/1/oneOf/1/not",
    "instanceLocation"=>"",
    "error"=>"value at root matches `not` schema"}]}

And output_format: 'detailed' (nested errors):

{"valid"=>false,
 "keywordLocation"=>"/allOf/1/oneOf",
 "absoluteKeywordLocation"=>"json-schemer://schema#/allOf/1/oneOf",
 "instanceLocation"=>"",
 "error"=>"value at root does not match exactly one `oneOf` schema",
 "errors"=>
  [{"valid"=>false,
    "keywordLocation"=>"/allOf/1/oneOf/0/not",
    "absoluteKeywordLocation"=>"json-schemer://schema#/allOf/1/oneOf/0/not",
    "instanceLocation"=>"",
    "error"=>"value at root matches `not` schema"},
   {"valid"=>false,
    "keywordLocation"=>"/allOf/1/oneOf/1/not",
    "absoluteKeywordLocation"=>"json-schemer://schema#/allOf/1/oneOf/1/not",
    "instanceLocation"=>"",
    "error"=>"value at root matches `not` schema"}]}

[babelfish]

With the simpler example:

properties:
  a:
    type: string
  b:
    type: string
oneOf:
  - required: [a]
    not:
      required: [b]
  - required: [b]
    not:
      required: [a]

a: foo
b: bar

I'm not entirely sure what the best way to handle it would be. Generating helpful, human-readable error messages when dealing with root-level composition keywords does seem like a daunting task. Ideally I'd be able to get an error message along the lines of "properties a and b were provided, however only one or the other may be specified".

I've been using JSONSchemer::Errors.pretty for generating the message for 422 responses in an API, the nature of which means that the people looking at request failures may not have any knowledge of JSON Schema. I can look at the more detailed error format and see where the problem is, but I can't guarantee that our users could.

Currently it looks like I'll have to try to programmatically determine when json_schemer will be unable to provide a good error message and override it with one created specifically for our schema, but it's an unsatisfying and brittle solution.

[davishmcclurg]

I'm not entirely sure what the best way to handle it would be. Generating helpful, human-readable error messages when dealing with root-level composition keywords does seem like a daunting task. Ideally I'd be able to get an error message along the lines of "properties a and b were provided, however only one or the other may be specified".

Yeah, it's a tough problem to solve in a generic way. It comes up enough, though, that I would like to have a better solution to offer.

One approach I've thought about is a custom keyword that would override the error message, eg:

?> JSONSchemer.schema({
?>   "oneOf": [
?>     {
?>       "x-error": "properties a and b were provided, however only one or the other may be specified",
?>       "required": ["a"],
?>       "not": { "required": ["b"] }
?>     },
?>     {
?>       "x-error": "properties a and b were provided, however only one or the other may be specified",
?>       "required": ["b"],
?>       "not": { "required": ["a"] }
?>     }
?>   ]
?> }).validate({
?>   "a": "foo",
?>   "b": "bar"
>> }).map { _1.fetch('error') }
=> ["properties a and b were provided, however only one or the other may be specified", "properties a and b were provided, however only one or the other may be specified"]

And you could also provide keyword-specific errors by defining x-error as a hash with keys matching the target keywords. I think this is similar to ajv-errors.

The other approach I've considered is somehow using i18n to lookup error messages. That has the advantage of being locale-specific. I haven't thought this one through completely yet.

Do either of those approaches seem like they would work for your use case?

[babelfish]

The first does seem reasonable, though I would have to experiment to make sure it doesn't cause problems. I'm using OAS 3.1 with API docs hosted by redocly and I don't know offhand how it handles unknown keywords.

[babelfish]

Yeah, I think the x-error solution would work, and seems like it would be relatively easy to implement.

Redocly apparently doesn't handle these keywords well in the first place, but that's another problem for me to deal with.

[babelfish]

Is that something you think you could implement in the short term? I'm needing to decide whether to wait for a solution here or implement a custom one.

[davishmcclurg]

@babelfish I opened a PR with x-error support here: #149
If you have a chance, do you mind trying it out to see if it works for you?

[babelfish]

I switched to that branch and started writing some test cases, and I think this'll meet my needs perfectly. Thanks so much. I am leaving one potential suggestion on the PR.

[davishmcclurg]

Thanks for trying it out, @babelfish—glad it's working for you. Let me know if anything comes up. I'll probably merge and release next week.

I'll look into your suggestion as well. Seems reasonable to add "x-error": true to the output hashes. Need to figure out if that will work for i18n too.

[davishmcclurg]

Custom error options were just released in 2.1.0.