From cc8957f57cc5fe3732552df0fab5cc28eaa4c32d Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Mon, 19 Nov 2018 11:28:58 +1300 Subject: [PATCH 01/13] Add 'output formatting' markdown composition as working medium --- standardized-output-composition.md | 291 +++++++++++++++++++++++++++++ 1 file changed, 291 insertions(+) create mode 100644 standardized-output-composition.md diff --git a/standardized-output-composition.md b/standardized-output-composition.md new file mode 100644 index 00000000..f90c2733 --- /dev/null +++ b/standardized-output-composition.md @@ -0,0 +1,291 @@ +# Output formatting + +## Summary + +JSON Schema is defined to be platform-independent. As such, to increase compatibility across platforms, implementations SHOULD conform to a standard validation output format. This section describes the minimum requirements that consumers will need to properly interpret validation results. + +## Format + +JSON Schema output is defined using JSON. Implementations MAY deviate from this as supported by their specific languages and platforms, however it is RECOMMENDED that the output be convertible to the JSON format defined herein via serialization or other means. + +## Output Levels **(is "levels" the right word?)** + +This specification defines four levels of output. See the *Output Structure* section for the requirements of each level. + +- **Pass/Fail** - A boolean which simply indicates whether the schema was validated. +- **Basic** - Provides validation information in a flat list structure. +- **Detailed** - Provides validation information in a condensed hierarchical structure based on the structure of the schema. +- **Verbose** - Provides validation information in an uncondensed hierarchical structure that matches the exact structure of the schema. + +Implementations SHOULD specify which levels of validation they support. + +## Minimum Information + +Beyond a simplistic pass/fail, additional information is useful to aid in debugging a schema or instance. Each sub-result SHOULD contain the information contained within this section at a minimum. + +A single object that contains all of these components is considered an output unit. + +Implementations MAY elect to provide additional information. + +### Keyword relative location + +The relative location of the validating keyword that follows the validation path. The value MUST be expressed as a JSON Pointer, and it MUST include `$ref` or `$recursiveRef` segments for any reference jumps **(wording?)**. + +``` +#/properties/minLength/$ref/minimum +``` + +The JSON key for this information is `keywordLocation`. + +### Keyword absolute location + +The absolute, dereferenced location of the validating keyword. The value MUST be expressed as an absolute URI, and it MUST NOT include `$ref` or `$recursiveRef` segments. + +``` +http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum +``` + +This information MAY be omitted if either the relative location contains no references or if the schema does not declare an absolute URI as its `$id`. + +The JSON key for this information is `absoluteKeywordLocation`. + +### Instance location + +The location of the JSON value within the instance being validated. The value MUST be expressed as a JSON Pointer. + +The JSON key for this information is `instanceLocation`. + +### Error/Annotation + +The error or annotation that is produced by the validation. + +For errors, the specific wording for the message is not defined by this specification. Implementations will need to provide this. + +The JSON key for failed validations is `error`; for successful validations it is `annotation`. + +### Nested results + +For the two hierarchical structures, this property will hold nested errors and annotations. + +The JSON key for nested results in failed validations is `errors`; for successful validations it is `annotations`. + +## Output Structure + +The output MUST be an object containing two properties + +- `valid` - a boolean value indicating the overall validation success or failure +- `errors` - the collection of errors or annotations produced by a failed validation +- `annotations` - the collection of errors or annotations produced by a successful validation + +For these examples, the following schema and instance will be used. + +```json +{ + "$id": "http://example.com/polygon#", + "$schema": "http://json-schema.org/draft-08/schema#", + "$defs": { + "point": { + "type": "object", + "properties": { + "x": { "type": "number" }, + "y": { "type": "number" } + }, + "additionalProperties": false, + "required": [ "x", "y" ] + } + }, + "type": "array", + "items": { "$ref": "#/$defs/point" }, + "minItems": 3 +} + +[ + { + "x": 2.5, + "y": 1.3, + }, + { + "x": 1, + "z": 6.7 + } +] +``` + +This instance will fail validation and produce errors, but it's trivial to deduce examples for passing schemas that produce annotations. + +Specifically, the errors it will produce are: + +1. The second element in the `vertices` property is missing a `y` property. +2. The second element in the `vertices` property has a disallowed `z` property. +3. There are only two vertices, but three are required. + +Note that the wording in the error messages depicted in these examples is not a requirement of this specification. Implementations SHOULD craft error messages tailored for their audience. + +### Pass/Fail + +In the simplest case, merely the boolean result for the `valid` valid property needs to be fulfilled. + +```json +{ + "valid": false +} +``` + +Because no errors or annotations are returned with this format, it is RECOMMENDED that implementations use short-circuiting logic to return failure or success as soon as the outcome can be determined. For example, if an `anyOf` keyword contains five sub-schemas, and the second one passes, there is no need to check the other three. The logic can simply return with success. + +### Basic + +The **Basic** structure is a flat list of output units. + +```json +{ + "valid": false, + "errors": [ + { + "keywordLocation": "#", + "instanceLocation": "#", + "error": "A subschema had errors." + }, + { + "keywordLocation": "#/items/$ref", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point", + "instanceLocation": "#/1", + "error": "A subschema had errors." + }, + { + "keywordLocation": "#/items/$ref/required", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/required", + "instanceLocation": "#/1", + "keyword": "required", + "error": "Required property 'y' not found." + }, + { + "keywordLocation": "#/items/$ref/additionalProperties", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/additionalProperties", + "instanceLocation": "#/1/z", + "error": "Additional property 'z' found but was invalid." + }, + { + "keywordLocation": "#/minItems", + "instanceLocation": "#", + "keyword": "minItems", + "error": "Expected at least 3 items but found 2" + } + ] +} +``` + +### Detailed + +The **Detailed** structure is based on the schema and can be more readable for both humans and machines. Having the structure organized this way makes associations between the errors more apparent. For example, the fact that the missing `y` property and the extra `z` property both stem from the same location in the instance is not immediately obvious in the **Basic** structure. In a hierarchy, the correllation is more easily identified. + +The following rules govern the construction of the results object: + +1. All applicator keywords (`*Of`, `$ref`, `if`/`then`/`else`, etc.) require a node. +2. Nodes that have no children are removed. +3. Nodes that have a single child are replaced by the child. + +Branch nodes do not require an error message or an annotation. + +```json +{ + "valid": false, + "keywordLocation": "#", + "instanceLocation": "#", + "errors": [ + { + "valid": false, + "keywordLocation": "#/items/$ref", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point", + "instanceLocation": "#/1", + "errors": [ + { + "valid": false, + "keywordLocation": "#/items/$ref/required", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/required", + "instanceLocation": "#/1", + "error": "Required property 'y' not found." + }, + { + "valid": false, + "keywordLocation": "#/items/$ref/additionalProperties", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/additionalProperties", + "instanceLocation": "#/1/z", + "error": "Additional property 'z' found but was invalid." + } + ] + }, + { + "valid": false, + "keywordLocation": "#/minItems", + "instanceLocation": "#", + "error": "Expected at least 3 items but found 2" + } + ] +} +``` + +### Verbose + +The **Verbose** structure is a fully realized hierarchy that exactly matches that of the schema. This structure has applications in form generation and validation where the error's location is important. + +The primary difference between this and the **Detailed** structure is that *all* results are returned. This includes sub-schema validation results that would otherwise be removed (e.g. annotations for failed validations, successful validations inside a `not` keyword, etc.). Because of this, it is RECOMMENDED that each node also carry a `valid` property to indicate the validation result for that node. + +Because this output structure can be quite large, a smaller example is given here for brevity. The full output structure of the example above can be found [here](standardized-output-verbose.json) + +```json +// schema +{ + "$id": "http://example.com/polygon#", + "$schema": "http://json-schema.org/draft-08/schema#", + "type": "object", + "properties": { + "validProp": true, + }, + "additionalProperties": false +} + +// instance +{ + "validProp": 5, + "disallowedProp": "value" +} + +// result +{ + "valid": false, + "keywordLocation": "#", + "instanceLocation": "#", + "errors": [ + { + "valid": true, + "keywordLocation": "#/type", + "instanceLocation": "#" + }, + { + "valid": true, + "keywordLocation": "#/properties", + "instanceLocation": "#" + }, + { + "valid": false, + "keywordLocation": "#/additionalProperties", + "instanceLocation": "#", + "errors": [ + { + "valid": false, + "keywordLocation": "#/additionalProperties", + "instanceLocation": "#/disallowedProp", + "error": "Additional property 'disallowedProp' found but was invalid." + } + ] + } + ] +} +``` From 2a4b28b160df59d7bba37cfc9e8aa66b97150ed8 Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Mon, 19 Nov 2018 11:30:01 +1300 Subject: [PATCH 02/13] Add exceedingly large verbose output as separate file --- standardized-output-verbose.json | 153 +++++++++++++++++++++++++++++++ 1 file changed, 153 insertions(+) create mode 100644 standardized-output-verbose.json diff --git a/standardized-output-verbose.json b/standardized-output-verbose.json new file mode 100644 index 00000000..a5254f65 --- /dev/null +++ b/standardized-output-verbose.json @@ -0,0 +1,153 @@ +{ + "valid": false, + "keywordLocation": "#", + "instanceLocation": "#", + "errors": [ + { + "valid": true, + "keywordLocation": "#/definitions", + "instanceLocation": "#", + "keyword": "definitions" + }, + { + "valid": true, + "keywordLocation": "#/type", + "instanceLocation": "#", + "keyword": "type" + }, + { + "valid": false, + "keywordLocation": "#/items", + "instanceLocation": "#", + "keyword": "items", + "errors": [ + { + "valid": true, + "keywordLocation": "#/items/$ref", + "absoluteKeywordLocation": + "http://example.com/polygon#/items/$ref", + "instanceLocation": "#/0", + "keyword": "$ref", + "annotations": [ + { + "valid": true, + "keywordLocation": "#/items/$ref", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point", + "instanceLocation": "#/0", + "annotations": [ + { + "valid": true, + "keywordLocation": "#/items/$ref/type", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/type", + "instanceLocation": "#/0", + "keyword": "type" + }, + { + "valid": true, + "keywordLocation": "#/items/$ref/properties", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/properties", + "instanceLocation": "#/0", + "keyword": "properties" + }, + { + "valid": true, + "keywordLocation": "#/items/$ref/required", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/required", + "instanceLocation": "#/0", + "keyword": "required" + }, + { + "valid": true, + "keywordLocation": "#/items/$ref/additionalProperties", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/additionalProperties", + "instanceLocation": "#/0", + "keyword": "additionalProperties" + } + ] + } + ] + }, + { + "valid": false, + "keywordLocation": "#/items/$ref", + "absoluteKeywordLocation": + "http://example.com/polygon#/items/$ref", + "instanceLocation": "#/1", + "keyword": "$ref", + "errors": [ + { + "valid": false, + "keywordLocation": "#/items/$ref", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point", + "instanceLocation": "#/1", + "errors": [ + { + "valid": true, + "keywordLocation": "#/items/$ref/type", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/type", + "instanceLocation": "#/1", + "keyword": "type" + }, + { + "valid": true, + "keywordLocation": "#/items/$ref/properties", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/properties", + "instanceLocation": "#/1", + "keyword": "properties" + }, + { + "valid": false, + "keywordLocation": "#/items/$ref/required", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/required", + "instanceLocation": "#/1", + "keyword": "required", + "additionalInfo": { + "missing": [ + "y" + ] + } + }, + { + "valid": false, + "keywordLocation": "#/items/$ref/additionalProperties", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/additionalProperties", + "instanceLocation": "#/1", + "keyword": "additionalProperties", + "errors": [ + { + "valid": false, + "keywordLocation": "#/items/$ref/additionalProperties", + "absoluteKeywordLocation": + "http://example.com/polygon#/definitions/point/additionalProperties", + "instanceLocation": "#/1/z" + } + ] + } + ] + } + ] + } + ] + }, + { + "valid": false, + "keywordLocation": "#/minItems", + "instanceLocation": "#", + "keyword": "minItems", + "additionalInfo": { + "expected": 3, + "actual": 2 + } + } + ] +} From 6571b0672290dbe3a65d398938394c4344a999a7 Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Mon, 19 Nov 2018 11:42:53 +1300 Subject: [PATCH 03/13] Add composition to spec XML --- jsonschema-core.xml | 469 ++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 455 insertions(+), 14 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index 567d304e..e44c51e4 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -41,6 +41,18 @@ + +
+ + + Auckland + + NZ + + gregsdennis@yahoo.com +
+ + Internet Engineering Task Force JSON @@ -86,7 +98,8 @@ This specification defines JSON Schema core terminology and mechanisms, including pointing to another JSON Schema by reference, dereferencing a JSON Schema reference, - and specifying the vocabulary being used. + specifying the vocabulary being used, + and defining the expected output. Other specifications define the vocabularies that perform assertions about validation, @@ -1411,14 +1424,6 @@
- - - The exact structure and format of the information collected is TBD, - but will be defined before the next draft. Some details of this - section may change as a result, but the overall process is expected - to remain the same. See GitHub issue #396 to track progress. - - Annotations are collected by keywords that explicitly define annotation-collecting behavior. Note that boolean schemas cannot @@ -1434,12 +1439,13 @@ The instance location to which it is attached, as a JSON Pointer - The absolute schema location of the attaching keyword, as a URI + The schema location path, indicating how reference keywords + such as "$ref" were followed to reach the absolute schema location. - The schema location path, indicating how reference keywords - such as "$ref" were followed to reach the absolute schema location - The exact format of this path is TBD, again see issue #396 + The absolute schema location of the attaching keyword, as a URI. + This MAY be omitted if it is the same as the schema location path + from above. The attached value(s) @@ -2095,6 +2101,440 @@
+
+ + JSON Schema is defined to be platform-independent. As such, to increase compatibility + across platforms, implementations SHOULD conform to a standard validation output + format. This section describes the minimum requirements that consumers will need to + properly interpret validation results. + + +
+ + JSON Schema output is defined using JSON. Implementations MAY deviate from this + as supported by their specific languages and platforms, however it is RECOMMENDED + that the output be convertible to the JSON format defined herein via serialization + or other means. + +
+ +
+ + This specification defines four levels of output. See the *Output Structure* + section for the requirements of each level. + + + Pass/Fail - A boolean which simply indicates whether the schema was + validated. + + + Basic - Provides validation information in a flat list structure. + + + Detailed - Provides validation information in a condensed hierarchical + structure based on the structure of the schema. + + + Verbose - Provides validation information in an uncondensed hierarchical + structure that matches the exact structure of the schema. + + + Implementations SHOULD specify which levels of validation they support. + + +
+ +
+ + Beyond a simplistic pass/fail, additional information is useful to aid in + debugging a schema or instance. Each sub-result SHOULD contain the information + contained within this section at a minimum. + + + A single object that contains all of these components is considered an + output unit. + + + Implementations MAY elect to provide additional information. + + +
+ + The relative location of the validating keyword that follows the validation + path. The value MUST be expressed as a JSON Pointer, and it MUST include + "$ref" or "$recursiveRef" segments for any reference jumps. + +
+ + + +
+ + The JSON key for this information is "keywordLocation". + +
+ +
+ + The absolute, dereferenced location of the validating keyword. The value MUST + be expressed as an absolute URI, and it MUST NOT include "$ref" or + "$recursiveRef" segments. + +
+ + + +
+ + This information MAY be omitted if either the relative location contains no + references or if the schema does not declare an absolute URI as its "$id". + + + The JSON key for this information is "absoluteKeywordLocation". + +
+ +
+ + The location of the JSON value within the instance being validated. The + value MUST be expressed as a JSON Pointer. + + + The JSON key for this information is "instanceLocation". + +
+ +
+ + The error or annotation that is produced by the validation. + + + For errors, the specific wording for the message is not defined by this + specification. Implementations will need to provide this. + + + The JSON key for failed validations is "error"; for successful validations + it is "annotation". + +
+ +
+ + For the two hierarchical structures, this property will hold nested errors + and annotations. + + + The JSON key for nested results in failed validations is "errors"; for + successful validations it is "annotations". + +
+ +
+ +
+ + The output MUST be an object containing two properties + + + "valid" - a boolean value indicating the overall validation success or + failure + + + "errors" - the collection of errors or annotations produced by a failed + validation + + + "annotations" - the collection of errors or annotations produced by a + successful validation + + + For these examples, the following schema and instance will be used. + +
+ + + +
+ + This instance will fail validation and produce errors, but it's trivial to deduce + examples for passing schemas that produce annotations. + + + Specifically, the errors it will produce are: + + + The second element in the "vertices" property is missing a "y" property. + + + The second element in the "vertices" property has a disallowed "z" property. + + + There are only two vertices, but three are required. + + + Note that the wording in the error messages depicted in these examples is not a + requirement of this specification. Implementations SHOULD craft error messages + tailored for their audience. + + +
+ + In the simplest case, merely the boolean result for the "valid" valid property + needs to be fulfilled. + +
+ + + +
+ + Because no errors or annotations are returned with this format, it is + RECOMMENDED that implementations use short-circuiting logic to return + failure or success as soon as the outcome can be determined. For example, + if an "anyOf" keyword contains five sub-schemas, and the second one + passes, there is no need to check the other three. The logic can simply + return with success. + +
+ +
+ + The "Basic" structure is a flat list of output units. + +
+ + + +
+
+ +
+ + The "Detailed" structure is based on the schema and can be more readable + for both humans and machines. Having the structure organized this way makes + associations between the errors more apparent. For example, the fact that + the missing "y" property and the extra "z" property both stem from the same + location in the instance is not immediately obvious in the "Basic" structure. + In a hierarchy, the correllation is more easily identified. + + + The following rules govern the construction of the results object: + + + All applicator keywords ("*Of", "$ref", "if"/"then"/"else", etc.) require + a node. + + + Nodes that have no children are removed. + + + Nodes that have a single child are replaced by the child. + + + Branch nodes do not require an error message or an annotation. + +
+ + + +
+
+ +
+ + The **Verbose** structure is a fully realized hierarchy that exactly matches + that of the schema. This structure has applications in form generation and + validation where the error's location is important. + + + The primary difference between this and the **Detailed** structure is that + *all* results are returned. This includes sub-schema validation results that + would otherwise be removed (e.g. annotations for failed validations, + successful validations inside a `not` keyword, etc.). Because of this, it + is RECOMMENDED that each node also carry a `valid` property to indicate the + validation result for that node. + + + Because this output structure can be quite large, a smaller example is given + here for brevity. The full output structure of the example above can be found + [here](standardized-output-verbose.json) + +
+ + + +
+
+ +
+ +
+
@@ -2416,7 +2856,8 @@ User-Agent: product-name/5.4.1 so-cool-json-schema/1.0.2 curl/7.43.0 Brad Bowman, Gowry Sankar, Donald Pipowitch, - and Dave Finlay + Dave Finlay, + and Greg Dennis for their submissions and patches to the document.
From 59f9ff0ea2afd267ab938eb6e4f7e49036d78dea Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Fri, 30 Nov 2018 21:05:11 +1300 Subject: [PATCH 04/13] updated per @Anthropic's suggestions :+1: --- jsonschema-core.xml | 10 ++++++---- 1 file changed, 6 insertions(+), 4 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index e44c51e4..b56282da 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -2124,7 +2124,7 @@ section for the requirements of each level. - Pass/Fail - A boolean which simply indicates whether the schema was + Flag - A boolean which simply indicates whether the schema was validated. @@ -2146,7 +2146,7 @@
- Beyond a simplistic pass/fail, additional information is useful to aid in + Beyond the simplistic "Flag" output, additional information is useful to aid in debugging a schema or instance. Each sub-result SHOULD contain the information contained within this section at a minimum. @@ -2237,7 +2237,9 @@ http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum
- The output MUST be an object containing two properties + The output MUST be an object containing a boolean property named "valid". When + additional information about the result is required, the output MUST also contain + "errors" or "annotations" as described below. "valid" - a boolean value indicating the overall validation success or @@ -2311,7 +2313,7 @@ http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum tailored for their audience. -
+
In the simplest case, merely the boolean result for the "valid" valid property needs to be fulfilled. From cf7a6e0e349e12db23020256f8b18dce17e2e24e Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Thu, 6 Dec 2018 12:56:56 +1300 Subject: [PATCH 05/13] addressed most of @handrews's comments --- jsonschema-core.xml | 24 +++++++++++++----------- 1 file changed, 13 insertions(+), 11 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index b56282da..d596a093 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -41,7 +41,7 @@ - +
@@ -616,6 +616,9 @@ exists during the evaluation of a schema, typically together with an instance document. The outermost dynamic scope is the root schema of the schema document in which processing begins. + The path from this root schema to any particular keyword (that + includes any "$ref" and "$recursiveRef" keywords that may have + been resolved) is considered the keyword's "validation path." Or should this be the schema object at which processing begins, even if it is not a root? This has some implications @@ -634,8 +637,8 @@ dynamic parent, rather than examining the local lexically enclosing parent. - The concept of dynamic scope is primarily used with "$recursiveRef" - and "$recursiveAnchor", and should be considered and advanced feature + The concept of dynamic scope is primarily used with "$recursiveRef", + "$recursiveAnchor", and should be considered an advanced feature and used with caution when defining additional keywords.
@@ -2111,10 +2114,10 @@
- JSON Schema output is defined using JSON. Implementations MAY deviate from this - as supported by their specific languages and platforms, however it is RECOMMENDED - that the output be convertible to the JSON format defined herein via serialization - or other means. + JSON Schema output is defined using the JSON Schema data instance model as described + in section 4.2.1. Implementations MAY deviate from this as supported by their + specific languages and platforms, however it is RECOMMENDED that the output be + convertible to the JSON format defined herein via serialization or other means.
@@ -2179,8 +2182,8 @@
The absolute, dereferenced location of the validating keyword. The value MUST - be expressed as an absolute URI, and it MUST NOT include "$ref" or - "$recursiveRef" segments. + be expressed as an absolute URI, and it MUST NOT include by-reference applicators + such as "$ref" or "$recursiveRef".
@@ -2858,8 +2861,7 @@ User-Agent: product-name/5.4.1 so-cool-json-schema/1.0.2 curl/7.43.0 Brad Bowman, Gowry Sankar, Donald Pipowitch, - Dave Finlay, - and Greg Dennis + and Dave Finlay for their submissions and patches to the document.
From fd1b00d0091564ee0e2dfeee43849fa791d74056 Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Fri, 7 Dec 2018 09:40:19 +1300 Subject: [PATCH 06/13] removed composition file; added output schema --- jsonschema-core.xml | 15 +- schema-output.json | 85 +++++++++ standardized-output-composition.md | 291 ----------------------------- standardized-output-verbose.json | 43 +---- 4 files changed, 105 insertions(+), 329 deletions(-) create mode 100644 schema-output.json delete mode 100644 standardized-output-composition.md diff --git a/jsonschema-core.xml b/jsonschema-core.xml index d596a093..9eff41c9 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -2311,9 +2311,9 @@ http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum There are only two vertices, but three are required. - Note that the wording in the error messages depicted in these examples is not a - requirement of this specification. Implementations SHOULD craft error messages - tailored for their audience. + Note that neither the error message property nor its wording as depicted in these + examples is not a requirement of this specification. Implementations SHOULD craft + error messages tailored for their audience.
@@ -2367,7 +2367,6 @@ http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum "absoluteKeywordLocation": "http://example.com/polygon#/definitions/point/required", "instanceLocation": "#/1", - "keyword": "required", "error": "Required property 'y' not found." }, { @@ -2380,7 +2379,6 @@ http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum { "keywordLocation": "#/minItems", "instanceLocation": "#", - "keyword": "minItems", "error": "Expected at least 3 items but found 2" } ] @@ -2536,6 +2534,13 @@ http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum
+
+ + For convenience, a JSON Schema has been provided to validate output generated + by implementations. It can be found [here](schema-output.json). + +
+
diff --git a/schema-output.json b/schema-output.json new file mode 100644 index 00000000..579ef8bb --- /dev/null +++ b/schema-output.json @@ -0,0 +1,85 @@ +{ + "$schema": "http://json-schema.org/draft-08/schema#", + "$id": "http://json-schema.org/draft-08/schema-output#", + "description": "A schema that validates the minimum requirements for validation output", + "$defs": { + "outputUnit":{ + "properties": { + "valid": { "type": "boolean" }, + "keywordLocation": { + "type": "string", + "format": "uri-reference" + }, + "absoluteKeywordLocation": { + "type": "string", + "format": "uri" + }, + "instanceLocation": { + "type": "string", + "format": "uri-reference" + }, + "errors": { + "$ref": "#/$defs/outputUnitArray" + }, + "annotations": { + "$ref": "#/$defs/outputUnitArray" + } + }, + "required": [ "valid", "keywordLocation", "instanceLocation" ], + "allOf": [ + { + "if": { + "properties": { + "valid": { "const": false } + } + }, + "then": { + "required": [ "errors" ] + } + }, + { + "if": { + "oneOf": [ + { + "properties": { + "keywordLocation": { + "pattern": ".*/$ref/.*" + } + } + }, + { + "properties": { + "keywordLocation": { + "pattern": ".*/$recursiveRef/.*" + } + } + } + ] + }, + "then": { + "required": [ "absoluteKeywordLocation" ] + } + } + ] + }, + "outputUnitArray": { + "type": "array", + "items": { "$ref": "#/$defs/outputUnit" } + }, + "flag": { + "properties": { + "valid": { "type": "boolean" } + }, + "required": [ "valid" ] + }, + "basic": { "$ref": "#/outputUnit" }, + "detailed": { "$ref": "#/outputUnit" }, + "verbose": { "$ref": "#/outputUnit" } + }, + "oneOf": [ + { "$ref": "#/$defs/flag" }, + { "$ref": "#/$defs/basic" }, + { "$ref": "#/$defs/detailed" }, + { "$ref": "#/$defs/verbose" } + ] +} \ No newline at end of file diff --git a/standardized-output-composition.md b/standardized-output-composition.md deleted file mode 100644 index f90c2733..00000000 --- a/standardized-output-composition.md +++ /dev/null @@ -1,291 +0,0 @@ -# Output formatting - -## Summary - -JSON Schema is defined to be platform-independent. As such, to increase compatibility across platforms, implementations SHOULD conform to a standard validation output format. This section describes the minimum requirements that consumers will need to properly interpret validation results. - -## Format - -JSON Schema output is defined using JSON. Implementations MAY deviate from this as supported by their specific languages and platforms, however it is RECOMMENDED that the output be convertible to the JSON format defined herein via serialization or other means. - -## Output Levels **(is "levels" the right word?)** - -This specification defines four levels of output. See the *Output Structure* section for the requirements of each level. - -- **Pass/Fail** - A boolean which simply indicates whether the schema was validated. -- **Basic** - Provides validation information in a flat list structure. -- **Detailed** - Provides validation information in a condensed hierarchical structure based on the structure of the schema. -- **Verbose** - Provides validation information in an uncondensed hierarchical structure that matches the exact structure of the schema. - -Implementations SHOULD specify which levels of validation they support. - -## Minimum Information - -Beyond a simplistic pass/fail, additional information is useful to aid in debugging a schema or instance. Each sub-result SHOULD contain the information contained within this section at a minimum. - -A single object that contains all of these components is considered an output unit. - -Implementations MAY elect to provide additional information. - -### Keyword relative location - -The relative location of the validating keyword that follows the validation path. The value MUST be expressed as a JSON Pointer, and it MUST include `$ref` or `$recursiveRef` segments for any reference jumps **(wording?)**. - -``` -#/properties/minLength/$ref/minimum -``` - -The JSON key for this information is `keywordLocation`. - -### Keyword absolute location - -The absolute, dereferenced location of the validating keyword. The value MUST be expressed as an absolute URI, and it MUST NOT include `$ref` or `$recursiveRef` segments. - -``` -http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum -``` - -This information MAY be omitted if either the relative location contains no references or if the schema does not declare an absolute URI as its `$id`. - -The JSON key for this information is `absoluteKeywordLocation`. - -### Instance location - -The location of the JSON value within the instance being validated. The value MUST be expressed as a JSON Pointer. - -The JSON key for this information is `instanceLocation`. - -### Error/Annotation - -The error or annotation that is produced by the validation. - -For errors, the specific wording for the message is not defined by this specification. Implementations will need to provide this. - -The JSON key for failed validations is `error`; for successful validations it is `annotation`. - -### Nested results - -For the two hierarchical structures, this property will hold nested errors and annotations. - -The JSON key for nested results in failed validations is `errors`; for successful validations it is `annotations`. - -## Output Structure - -The output MUST be an object containing two properties - -- `valid` - a boolean value indicating the overall validation success or failure -- `errors` - the collection of errors or annotations produced by a failed validation -- `annotations` - the collection of errors or annotations produced by a successful validation - -For these examples, the following schema and instance will be used. - -```json -{ - "$id": "http://example.com/polygon#", - "$schema": "http://json-schema.org/draft-08/schema#", - "$defs": { - "point": { - "type": "object", - "properties": { - "x": { "type": "number" }, - "y": { "type": "number" } - }, - "additionalProperties": false, - "required": [ "x", "y" ] - } - }, - "type": "array", - "items": { "$ref": "#/$defs/point" }, - "minItems": 3 -} - -[ - { - "x": 2.5, - "y": 1.3, - }, - { - "x": 1, - "z": 6.7 - } -] -``` - -This instance will fail validation and produce errors, but it's trivial to deduce examples for passing schemas that produce annotations. - -Specifically, the errors it will produce are: - -1. The second element in the `vertices` property is missing a `y` property. -2. The second element in the `vertices` property has a disallowed `z` property. -3. There are only two vertices, but three are required. - -Note that the wording in the error messages depicted in these examples is not a requirement of this specification. Implementations SHOULD craft error messages tailored for their audience. - -### Pass/Fail - -In the simplest case, merely the boolean result for the `valid` valid property needs to be fulfilled. - -```json -{ - "valid": false -} -``` - -Because no errors or annotations are returned with this format, it is RECOMMENDED that implementations use short-circuiting logic to return failure or success as soon as the outcome can be determined. For example, if an `anyOf` keyword contains five sub-schemas, and the second one passes, there is no need to check the other three. The logic can simply return with success. - -### Basic - -The **Basic** structure is a flat list of output units. - -```json -{ - "valid": false, - "errors": [ - { - "keywordLocation": "#", - "instanceLocation": "#", - "error": "A subschema had errors." - }, - { - "keywordLocation": "#/items/$ref", - "absoluteKeywordLocation": - "http://example.com/polygon#/definitions/point", - "instanceLocation": "#/1", - "error": "A subschema had errors." - }, - { - "keywordLocation": "#/items/$ref/required", - "absoluteKeywordLocation": - "http://example.com/polygon#/definitions/point/required", - "instanceLocation": "#/1", - "keyword": "required", - "error": "Required property 'y' not found." - }, - { - "keywordLocation": "#/items/$ref/additionalProperties", - "absoluteKeywordLocation": - "http://example.com/polygon#/definitions/point/additionalProperties", - "instanceLocation": "#/1/z", - "error": "Additional property 'z' found but was invalid." - }, - { - "keywordLocation": "#/minItems", - "instanceLocation": "#", - "keyword": "minItems", - "error": "Expected at least 3 items but found 2" - } - ] -} -``` - -### Detailed - -The **Detailed** structure is based on the schema and can be more readable for both humans and machines. Having the structure organized this way makes associations between the errors more apparent. For example, the fact that the missing `y` property and the extra `z` property both stem from the same location in the instance is not immediately obvious in the **Basic** structure. In a hierarchy, the correllation is more easily identified. - -The following rules govern the construction of the results object: - -1. All applicator keywords (`*Of`, `$ref`, `if`/`then`/`else`, etc.) require a node. -2. Nodes that have no children are removed. -3. Nodes that have a single child are replaced by the child. - -Branch nodes do not require an error message or an annotation. - -```json -{ - "valid": false, - "keywordLocation": "#", - "instanceLocation": "#", - "errors": [ - { - "valid": false, - "keywordLocation": "#/items/$ref", - "absoluteKeywordLocation": - "http://example.com/polygon#/definitions/point", - "instanceLocation": "#/1", - "errors": [ - { - "valid": false, - "keywordLocation": "#/items/$ref/required", - "absoluteKeywordLocation": - "http://example.com/polygon#/definitions/point/required", - "instanceLocation": "#/1", - "error": "Required property 'y' not found." - }, - { - "valid": false, - "keywordLocation": "#/items/$ref/additionalProperties", - "absoluteKeywordLocation": - "http://example.com/polygon#/definitions/point/additionalProperties", - "instanceLocation": "#/1/z", - "error": "Additional property 'z' found but was invalid." - } - ] - }, - { - "valid": false, - "keywordLocation": "#/minItems", - "instanceLocation": "#", - "error": "Expected at least 3 items but found 2" - } - ] -} -``` - -### Verbose - -The **Verbose** structure is a fully realized hierarchy that exactly matches that of the schema. This structure has applications in form generation and validation where the error's location is important. - -The primary difference between this and the **Detailed** structure is that *all* results are returned. This includes sub-schema validation results that would otherwise be removed (e.g. annotations for failed validations, successful validations inside a `not` keyword, etc.). Because of this, it is RECOMMENDED that each node also carry a `valid` property to indicate the validation result for that node. - -Because this output structure can be quite large, a smaller example is given here for brevity. The full output structure of the example above can be found [here](standardized-output-verbose.json) - -```json -// schema -{ - "$id": "http://example.com/polygon#", - "$schema": "http://json-schema.org/draft-08/schema#", - "type": "object", - "properties": { - "validProp": true, - }, - "additionalProperties": false -} - -// instance -{ - "validProp": 5, - "disallowedProp": "value" -} - -// result -{ - "valid": false, - "keywordLocation": "#", - "instanceLocation": "#", - "errors": [ - { - "valid": true, - "keywordLocation": "#/type", - "instanceLocation": "#" - }, - { - "valid": true, - "keywordLocation": "#/properties", - "instanceLocation": "#" - }, - { - "valid": false, - "keywordLocation": "#/additionalProperties", - "instanceLocation": "#", - "errors": [ - { - "valid": false, - "keywordLocation": "#/additionalProperties", - "instanceLocation": "#/disallowedProp", - "error": "Additional property 'disallowedProp' found but was invalid." - } - ] - } - ] -} -``` diff --git a/standardized-output-verbose.json b/standardized-output-verbose.json index a5254f65..ef50444f 100644 --- a/standardized-output-verbose.json +++ b/standardized-output-verbose.json @@ -6,20 +6,17 @@ { "valid": true, "keywordLocation": "#/definitions", - "instanceLocation": "#", - "keyword": "definitions" + "instanceLocation": "#" }, { "valid": true, "keywordLocation": "#/type", - "instanceLocation": "#", - "keyword": "type" + "instanceLocation": "#" }, { "valid": false, "keywordLocation": "#/items", "instanceLocation": "#", - "keyword": "items", "errors": [ { "valid": true, @@ -27,7 +24,6 @@ "absoluteKeywordLocation": "http://example.com/polygon#/items/$ref", "instanceLocation": "#/0", - "keyword": "$ref", "annotations": [ { "valid": true, @@ -41,32 +37,28 @@ "keywordLocation": "#/items/$ref/type", "absoluteKeywordLocation": "http://example.com/polygon#/definitions/point/type", - "instanceLocation": "#/0", - "keyword": "type" + "instanceLocation": "#/0" }, { "valid": true, "keywordLocation": "#/items/$ref/properties", "absoluteKeywordLocation": "http://example.com/polygon#/definitions/point/properties", - "instanceLocation": "#/0", - "keyword": "properties" + "instanceLocation": "#/0" }, { "valid": true, "keywordLocation": "#/items/$ref/required", "absoluteKeywordLocation": "http://example.com/polygon#/definitions/point/required", - "instanceLocation": "#/0", - "keyword": "required" + "instanceLocation": "#/0" }, { "valid": true, "keywordLocation": "#/items/$ref/additionalProperties", "absoluteKeywordLocation": "http://example.com/polygon#/definitions/point/additionalProperties", - "instanceLocation": "#/0", - "keyword": "additionalProperties" + "instanceLocation": "#/0" } ] } @@ -78,7 +70,6 @@ "absoluteKeywordLocation": "http://example.com/polygon#/items/$ref", "instanceLocation": "#/1", - "keyword": "$ref", "errors": [ { "valid": false, @@ -92,29 +83,21 @@ "keywordLocation": "#/items/$ref/type", "absoluteKeywordLocation": "http://example.com/polygon#/definitions/point/type", - "instanceLocation": "#/1", - "keyword": "type" + "instanceLocation": "#/1" }, { "valid": true, "keywordLocation": "#/items/$ref/properties", "absoluteKeywordLocation": "http://example.com/polygon#/definitions/point/properties", - "instanceLocation": "#/1", - "keyword": "properties" + "instanceLocation": "#/1" }, { "valid": false, "keywordLocation": "#/items/$ref/required", "absoluteKeywordLocation": "http://example.com/polygon#/definitions/point/required", - "instanceLocation": "#/1", - "keyword": "required", - "additionalInfo": { - "missing": [ - "y" - ] - } + "instanceLocation": "#/1" }, { "valid": false, @@ -122,7 +105,6 @@ "absoluteKeywordLocation": "http://example.com/polygon#/definitions/point/additionalProperties", "instanceLocation": "#/1", - "keyword": "additionalProperties", "errors": [ { "valid": false, @@ -142,12 +124,7 @@ { "valid": false, "keywordLocation": "#/minItems", - "instanceLocation": "#", - "keyword": "minItems", - "additionalInfo": { - "expected": 3, - "actual": 2 - } + "instanceLocation": "#" } ] } From df6351079b1b6f4108d5f21899541032e26be5db Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Fri, 7 Dec 2018 13:25:01 +1300 Subject: [PATCH 07/13] clarified requirements around implementation support --- jsonschema-core.xml | 11 +++++++---- 1 file changed, 7 insertions(+), 4 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index 9eff41c9..3d89b22f 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -2121,10 +2121,10 @@
-
+
- This specification defines four levels of output. See the *Output Structure* - section for the requirements of each level. + This specification defines four output formats. See the *Output Structure* + section for the requirements of each format. Flag - A boolean which simply indicates whether the schema was @@ -2142,7 +2142,10 @@ structure that matches the exact structure of the schema. - Implementations SHOULD specify which levels of validation they support. + Implementations SHOULD specify which formats of validation they support. Each + format is independent of the others. For example, if an implementation decides + to support the "detailed" format, it is not also required to support the "basic" + or "flag" formats, and vice versa.
From 06cd2689f18bb6d6059a73e10c57bc3800fbc968 Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Fri, 7 Dec 2018 13:29:44 +1300 Subject: [PATCH 08/13] addressed some wording concerns --- jsonschema-core.xml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index 3d89b22f..e3678d11 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -2127,8 +2127,8 @@ section for the requirements of each format. - Flag - A boolean which simply indicates whether the schema was - validated. + Flag - A boolean which simply indicates the overall validation result + with no further details. Basic - Provides validation information in a flat list structure. @@ -2168,7 +2168,7 @@ The relative location of the validating keyword that follows the validation path. The value MUST be expressed as a JSON Pointer, and it MUST include - "$ref" or "$recursiveRef" segments for any reference jumps. + any by-reference applicators such as "$ref" or "$recursiveRef".
From 5504ffca22c93b1f55d724e2781de60963c0473f Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Tue, 11 Dec 2018 09:24:22 +1300 Subject: [PATCH 09/13] added comment for relative pointers in output formatting --- jsonschema-core.xml | 20 ++++++++++++-------- 1 file changed, 12 insertions(+), 8 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index e3678d11..8502d02c 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -2123,7 +2123,7 @@
- This specification defines four output formats. See the *Output Structure* + This specification defines four output formats. See the "Output Structure" section for the requirements of each format. @@ -2177,6 +2177,10 @@ ]]>
+ + Note that this pointer may not be resolvable due to the inclusion of these + applicator keywords. + The JSON key for this information is "keywordLocation". @@ -2196,8 +2200,8 @@ http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum - This information MAY be omitted if either the relative location contains no - references or if the schema does not declare an absolute URI as its "$id". + This information MAY be omitted only if either the relative location contains + no references or if the schema does not declare an absolute URI as its "$id". The JSON key for this information is "absoluteKeywordLocation". @@ -2464,13 +2468,13 @@ http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum
- The **Verbose** structure is a fully realized hierarchy that exactly matches + The "Verbose" structure is a fully realized hierarchy that exactly matches that of the schema. This structure has applications in form generation and validation where the error's location is important. - The primary difference between this and the **Detailed** structure is that - *all* results are returned. This includes sub-schema validation results that + The primary difference between this and the "Detailed" structure is that + all results are returned. This includes sub-schema validation results that would otherwise be removed (e.g. annotations for failed validations, successful validations inside a `not` keyword, etc.). Because of this, it is RECOMMENDED that each node also carry a `valid` property to indicate the @@ -2479,7 +2483,7 @@ http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum Because this output structure can be quite large, a smaller example is given here for brevity. The full output structure of the example above can be found - [here](standardized-output-verbose.json) + here.
@@ -2540,7 +2544,7 @@ http://json-schema.org/draft-08/schema#/$defs/nonNegativeInteger/minimum
For convenience, a JSON Schema has been provided to validate output generated - by implementations. It can be found [here](schema-output.json). + by implementations. It can be found here.
From 32c624ba5860bd3eba48f6029df365484d88d146 Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Wed, 19 Dec 2018 09:12:07 +1300 Subject: [PATCH 10/13] Updated language around format support requirements --- jsonschema-core.xml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index 8502d02c..70a820b9 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -2142,17 +2142,19 @@ structure that matches the exact structure of the schema. - Implementations SHOULD specify which formats of validation they support. Each + Implementations MUST provide support for the "flag" format. It is RECOMMENDED + that implementations support at least one of the complex formats as well. Each format is independent of the others. For example, if an implementation decides to support the "detailed" format, it is not also required to support the "basic" - or "flag" formats, and vice versa. + or "verbose" formats, and vice versa. Implementations SHOULD specify in their + documentation which formats of validation they support.
- Beyond the simplistic "Flag" output, additional information is useful to aid in + Beyond the simplistic "flag" output, additional information is useful to aid in debugging a schema or instance. Each sub-result SHOULD contain the information contained within this section at a minimum. From fb312cc00fdea7fb699d0ed72cf5ea02df445c3f Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Sat, 22 Dec 2018 11:08:52 +1300 Subject: [PATCH 11/13] Set support requirements per conversation in slack --- jsonschema-core.xml | 9 +++------ 1 file changed, 3 insertions(+), 6 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index 70a820b9..3f2448c7 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -2142,12 +2142,9 @@ structure that matches the exact structure of the schema. - Implementations MUST provide support for the "flag" format. It is RECOMMENDED - that implementations support at least one of the complex formats as well. Each - format is independent of the others. For example, if an implementation decides - to support the "detailed" format, it is not also required to support the "basic" - or "verbose" formats, and vice versa. Implementations SHOULD specify in their - documentation which formats of validation they support. + Implementations MUST provide support for the "flag" format and SHOULD provide the + "basic" and "detailed" formats. The "verbose" format is OPTIONAL. Implementations + SHOULD specify in their documentation which formats of validation they support.
From 02295dc546337a938e9d38540dcdf7c28cab13b5 Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Tue, 25 Dec 2018 15:52:47 +1300 Subject: [PATCH 12/13] better wording for output --- jsonschema-core.xml | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index 3f2448c7..0df5cbb4 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -2143,8 +2143,9 @@ Implementations MUST provide support for the "flag" format and SHOULD provide the - "basic" and "detailed" formats. The "verbose" format is OPTIONAL. Implementations - SHOULD specify in their documentation which formats of validation they support. + "basic" and "detailed" formats, and MAY provide the "verbose" format. + Implementations SHOULD specify in their documentation which formats of validation + they support.
From 063fbe294a86aec572f8e6a676ca080bcafa5902 Mon Sep 17 00:00:00 2001 From: Greg Dennis Date: Thu, 27 Dec 2018 17:55:53 +1300 Subject: [PATCH 13/13] a bit more wordsmithing around output requirements --- jsonschema-core.xml | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/jsonschema-core.xml b/jsonschema-core.xml index 0df5cbb4..aea8bc51 100644 --- a/jsonschema-core.xml +++ b/jsonschema-core.xml @@ -2142,10 +2142,10 @@ structure that matches the exact structure of the schema. - Implementations MUST provide support for the "flag" format and SHOULD provide the - "basic" and "detailed" formats, and MAY provide the "verbose" format. - Implementations SHOULD specify in their documentation which formats of validation - they support. + An implementation SHOULD provide at least the "flag", "basic", or "detailed" + format and MAY provide the "verbose" format. If it provides one or more of the + complex formats, it MUST also provide the "flag" format. Implementations SHOULD + specify in their documentation which formats they support.