Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add field path details to the conformance runner output #333

Merged

Conversation

timostamm
Copy link
Member

For the following conformance failure, the runner shows exactly the same information for expected and actual result:

--- FAIL: standard_constraints/map (failed: 1, skipped: 0, passed: 0, total: 1)
    --- FAIL: recursive/invalid
       input: [type.googleapis.com/buf.validate.conformance.cases.MapRecursive]:{val:{key:1 value:{}}}
        want: 1 validation error(s)
          1. val[1].val: string.min_len (string.min_len)
             value length must be at least 3 characters
         got: 1 validation error(s)
          1. val[1].val: string.min_len (string.min_len)
             value length must be at least 3 characters

The cause for the failure is Violation.field: The path has an element with a int_key instead of the expected uint_key, which doesn't show in the string representation of the path.

This PR changes the output for a validation error result to include the Protobuf representation of field paths:

--- FAIL: standard_constraints/map (failed: 1, skipped: 0, passed: 0, total: 1)
    --- FAIL: recursive/invalid
       input: [type.googleapis.com/buf.validate.conformance.cases.MapRecursive]:{val:{key:1  value:{}}}
        want: validation error (1 violation)
          1. constraint_id: "string.min_len"
             message: "value length must be at least 3 characters"
             field: "val[1].val" elements:{field_number:1  field_name:"val"  field_type:TYPE_MESSAGE  key_type:TYPE_UINT32  value_type:TYPE_MESSAGE  uint_key:1}  elements:{field_number:1  field_name:"val"  field_type:TYPE_STRING}
             rule: "string.min_len" elements:{field_number:14  field_name:"string"  field_type:TYPE_MESSAGE}  elements:{field_number:2  field_name:"min_len"  field_type:TYPE_UINT64}
         got: validation error (1 violation)
          1. constraint_id: "string.min_len"
             message: "value length must be at least 3 characters"
             field: "val[1].val" elements:{field_number:1  field_name:"val"  field_type:TYPE_MESSAGE  key_type:TYPE_UINT32  value_type:TYPE_MESSAGE  int_key:1}  elements:{field_number:1  field_name:"val"  field_type:TYPE_STRING}
             rule: "string.min_len" elements:{field_number:14  field_name:"string"  field_type:TYPE_MESSAGE}  elements:{field_number:2  field_name:"min_len"  field_type:TYPE_UINT64}
  • The headline changes from "1 validation error(s)" to "validation error (1 violation)" to match API types
  • All values are labeled for clarity
  • Strings are shown as string literals for clarity
  • field and rule show the string representation and the prototext representation

@timostamm timostamm added the Conformance Related to the conformance testing cases or harness label Mar 11, 2025
Copy link

github-actions bot commented Mar 11, 2025

The latest Buf updates on your PR. Results from workflow Buf CI / buf (pull_request).

BuildFormatLintBreakingUpdated (UTC)
✅ passed✅ passed✅ passed✅ passedMar 11, 2025, 3:15 PM

@timostamm timostamm merged commit 51b65b6 into main Mar 11, 2025
10 checks passed
@timostamm timostamm deleted the tstamm/Add-field-path-details-to-the-conformance-runner-output branch March 11, 2025 17:28
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Conformance Related to the conformance testing cases or harness
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants