JSV

A JSON Schema Validation library for Elixir with full support for the 2020-12 JSON Schema specification.

Documentation
Getting started
- Installation
- Basic usage
Core concepts
Building schemas
Validation
Development
- Contributing
- Roadmap

Documentation

The API documentation is available on hexdocs.pm.

This document describes general considerations and recipes to use the library.

Getting started

Installation

def deps do
  [
    {:jsv, "~> 0.2"},
  ]
end

Additional dependencies can be added to support more features:

def deps do
  [
    # Optional libraries for better format validation support

    # Email validation
    {:mail_address, "~> 1.0"},

    # URI, IRI, JSON-pointers validation
    {:abnf_parsec, "~> 1.0"},


    # Optional libraries to decode schemas resolved from http

    {:jason, "~> 1.0"},
    # OR
    {:poison, "~> 6.0 or ~> 5.0"},
  ]
end

Basic usage

The following snippet describes the general usage of the library in any context.

The rest of the documentation describes how to use JSV in the context of an application.

# 1. Define a schema
schema = %{
  type: :object,
  properties: %{
    name: %{type: :string}
  },
  required: [:name]
}

# 2. Define a resolver
resolver = {JSV.Resolver.Httpc,
            allowed_prefixes: [
              "https://json-schema.org/",
              "https://my-company.example/"
            ]}

# 3. Build the schema
root = JSV.build!(schema, resolver: resolver)

# 4. Validate the data
case JSV.validate(%{"name" => "Alice"}, root) do
  {:ok, data} ->
    {:ok, data}

  {:error, validation_error} ->
    # Errors can be casted as JSON validator output to return them
    # to the producer of the invalid data
    {:error, JSON.encode!(JSV.normalize_error(validation_error))}
end

Core concepts

Input schema format

"Raw schemas" are schemas defined in Elixir data structures such as %{"type" => "integer"}.

JSV does not accept JSON strings. You will need to decode the JSON strings before giving them to the build function. There are three different possible formats for a schema:

A boolean. Booleans are valid schemas that accept anything (true) or reject everything (false).
A map with binary keys and values such as %{"type" => "integer"}.
A map with atom keys and/or values such as %{type :integer}. The :__struct__ property of structs is safely ignored.

The JSV.Schema struct can be used for autocompletion and provides a special behaviour over a raw map with atoms: any nil value found in the that will be ignored.

Raw maps and other structs have their nil values kept and treated as-is (it's generally invalid).

The :__struct__ property of other structs is safely ignored.

Atoms are converted to binaries internally so it is technically possible to mix atom with binaries in map keys or values but the behaviour for duplicate keys is not defined: %{"type" => "string", :type => "integer"}.

Meta-schemas: Introduction to vocabularies

JSV was built in compliance with the vocabulary mechanism of JSON schema, to support custom and optional keywords in the schemas.

It may be more clear with an example:

The well-known and official schema https://json-schema.org/draft/2020-12/schema defines the following vocabulary:

{
  "$vocabulary": {
    "https://json-schema.org/draft/2020-12/vocab/core": true,
    "https://json-schema.org/draft/2020-12/vocab/applicator": true,
    "https://json-schema.org/draft/2020-12/vocab/unevaluated": true,
    "https://json-schema.org/draft/2020-12/vocab/validation": true,
    "https://json-schema.org/draft/2020-12/vocab/meta-data": true,
    "https://json-schema.org/draft/2020-12/vocab/format-annotation": true,
    "https://json-schema.org/draft/2020-12/vocab/content": true
  }
}

The vocabulary is split in different parts, here one by object property. More information can be found on the official website.

Libraries such as JSV must map this vocabulary to implementations. For instance, in JSV, the https://json-schema.org/draft/2020-12/vocab/validation part that defines the type keyword is implemented with the JSV.Vocabulary.V202012.Validation Elixir module.
Finally, we can declare a schema that would like to use the type keyword. To let the library know what implementation to use for that keyword, the schema declares the https://json-schema.org/draft/2020-12/schema as its meta-schema (using the $schema keyword).
```
{
  "$schema": "https://json-schema.org/draft/2020-12/schema",
  "type": "integer"
}
```
This tells the library to pull the vocabulary from the meta-schema and apply it to the schema.
As JSV is compliant, it will use its implementation of https://json-schema.org/draft/2020-12/vocab/validation to validate types.

This also means that you can use a custom meta schema to skip some parts of the vocabulary, or add your own.

Resolvers overview

In order to build schemas properly, JSV needs to resolve the schema as a first step.

Resolving means fetching any remote resource whose data is needed and not available ; basically $schema, $ref or $dynamicRef properties pointing to an absolute URIs.

Those URIs are generally URLs with the http:// or https:// scheme, but other custom schemes can be used, and there are many ways to fetch HTTP resources in Elixir.

For security reasons, the default resolver, JSV.Resolver.Embedded, ships official meta-schemas as part of the source code and can only resolve those schemas.

For convenience reasons, a resolver that can fetch from the web is provided (JSV.Resolver.Httpc) but it needs to be manually declared by users of the JSV library. Refer to the documentation of this module for more information.

Users are encouraged to write their own resolver to benefit from HTTP libraries like Req or to resolve owned schemas from source or local repositories. See the custom resolvers section for more information.

Building schemas

In this chapter we will see how to build schemas from raw resources. The examples will mention the JSV.build/2 or JSV.build!/2 functions interchangeably. Everything described here applies to both.

Schemas are built according to their meta-schema vocabulary. JSV will assume that the $schema value is "https://json-schema.org/draft/2020-12/schema" by default if not provided.

Once built, a schema is converted into a JSV.Root, an internal representation of the schema that can be used to perform validation.

Custom resolvers

The Httpc resolver is here to help for common use cases, but you may need to resolve resources from custom locations instead of just fetching the web URL.

JSV.build/2 accepts any resolver implementation as long as it implements the JSV.Resolver behaviour. See the documentation of that behaviour for more information.

It is also possible to delegate to that resolver. If for instance we support a my-company:// custom scheme, we could define the behaviour like so:

defmodule MyApp.SchemaResolver do
  alias JSV.Resolver.BuiltIn

  def resolve("my-company://" <> _ = url, _opts) do
    uri = URI.parse(url)
    schema_dir = "priv/schemas"
    # In this example the hostname in URL is ignored
    schema_path = Path.join(schema_dir, uri.path)
    json_schema = File.read!(schema_path)
    JSON.decode(json_schema)
  end

  def resolve("https://" <> _url, _opts) do
    JSV.Resolver.BuiltIn.resolve(url,
      allowed_prefixes: [
        "https://json-schema.org/",
        "https://some-friend-company/",
        "https://some-other-provider/"
      ]
    )
  end
end

Of course a possible optimization would be to try the default JSV.Resolver.Embedded resolver for well known schemas instead of fetching them from the web.

The resolver can now be used:

JSV.build(raw_schema, resolver: MyApp.SchemaResolver)

Enable or disable format validation

By default, the https://json-schema.org/draft/2020-12/schema meta schema does not perform format validation. This is very counter intuitive, but it basically means that the following code will return {:ok, "not a date"}:

schema =
  JSON.decode!("""
  {
    "type": "string",
    "format": "date"
  }
  """)

root = JSV.build!(schema)

JSV.validate("not a date", root)

To always enable format validation when building a root schema, provide the formats: true option to JSV.build/2:

JSV.build(raw_schema, formats: true)

This is another reason to wrap JSV.build with a custom builder module!

Note that format validation is determined at build time. There is no way to change whether it is performed once the root schema is built.

You can also enable format validation by using the JSON Schema specification semantics, though we strongly advise to just use the :formats option and call it a day.

For format validation to be enabled, a schema should declare the https://json-schema.org/draft/2020-12/vocab/format-assertion vocabulary instead of the https://json-schema.org/draft/2020-12/vocab/format-annotation one that is included by default in the https://json-schema.org/draft/2020-12/schema meta schema.

So, first we would declare a new meta schema:

{
    "$id": "custom://with-formats-on/",
    "$schema": "https://json-schema.org/draft/2020-12/schema",
    "$vocabulary": {
        "https://json-schema.org/draft/2020-12/vocab/core": true,
        "https://json-schema.org/draft/2020-12/vocab/format-assertion": true
    },
    "$dynamicAnchor": "meta",
    "allOf": [
        { "$ref": "https://json-schema.org/draft/2020-12/meta/core" },
        { "$ref": "https://json-schema.org/draft/2020-12/meta/format-assertion" }
    ]
}

This example is taken from the JSON Schema Test Suite codebase and does not includes all the vocabularies, only the assertion for the formats and the core vocabulary.

Then we would declare our schema using that vocabulary to perform validation. Of course our resolver must be able to resolve the given URL for the new $schema property.

schema =
  JSON.decode!("""
  {
    "$schema": "custom://with-formats-on/",
    "type": "string",
    "format": "date"
  }
  """)

root = JSV.build!(schema, resolver: ...)

With this new meta-schema, JSV.validate/2 would return an error tuple without needing the formats: true.

{:error, _} = JSV.validate("hello", root)

In this case, it is also possible to disable the validation for schemas that use a meta-schema where the assertion vocabulary is declared:

JSV.build(raw_schema, formats: false)

Custom build modules

With that in mind, we suggest to define a custom module to wrap the JSV.build/2 function, so the resolver, formats and vocabularies can be defined only once.

That module could be implemented like this:

defmodule MyApp.SchemaBuilder do
  def build_schema!(raw_schema) do
    JSV.build!(raw_schema, resolver: MyApp.SchemaResolver, formats: true)
  end
end

Compile-time builds

It is strongly encouraged to build schemas at compile time, in order to avoid repeating the build step for no good reason.

For instance, if we have this function that should validate external data:

# Do not do this

def validate_order(order) do
  root =
    "priv/schemas/order.schema.json"
    |> File.read!()
    |> JSON.decode!()
    |> MyApp.SchemaBuilder.build_schema!()

  case JSV.validate(order, root) do
    {:ok, _} -> OrderHandler.handle_order(order)
    {:error, _} = err -> err
  end
end

The schema will be built each time the function is called. Building a schema is actually pretty fast but it is a waste of resources nevertheless.

One could do the following to get a net performance gain:

# Do this instead

@order_schema "priv/schemas/order.schema.json"
              |> File.read!()
              |> JSON.decode!()
              |> MyApp.SchemaBuilder.build_schema!()

defp order_schema, do: @order_schema

def validate_order(order) do
  case JSV.validate(order, order_schema()) do
    {:ok, _} -> OrderHandler.handle_order(order)
    {:error, _} = err -> err
  end
end

You can also define a module where all your schemas are built and exported as functions:

defmodule MyApp.Schemas do
  schemas = [
    order: "tmp/order.schema.json",
    shipping: "tmp/shipping.schema.json"
  ]

  Enum.each(schemas, fn {fun, path} ->
    root =
      path
      |> File.read!()
      |> JSON.decode!()
      |> MyApp.SchemaBuilder.build_schema!()

    def unquote(fun)() do
      unquote(Macro.escape(root))
    end
  end)
end

...and use it elsewhere:

def validate_order(order) do
  case JSV.validate(order, MyApp.Schemas.order()) do
    {:ok, _} -> OrderHandler.handle_order(order)
    {:error, _} = err -> err
  end
end

Validation

To validate a term, call the JSV.validate/3 function like so:

JSV.validate(data, root_schema, opts)

General considerations

JSV supports all keywords of the 2020-12 specification except:

The return value of JSV.validate/3 returns casted data. See the documentation of that function for more information.
The contentMediaType, contentEncoding and contentSchema keywords. They are ignored. Future support for custom vocabularies will allow you to validate data with such keywords.
The format keyword is largely supported but with many inconsistencies, mostly due to differences between Elixir and JavaScript (JSON Schema is largely based on JavaScript primitives). For most use cases, the differences are negligible.
The "integer" type will transform floats into integer when the fractional part is zero (such as 123.0). Support for floating-point numbers with large integer parts is using native Elixir semantics and may return incorrect results:
```
iex(1)> trunc(123456789123456789123456789.0)
# returns:    123456789123456791337762816
#                             |
#                             | Difference starts here
```
This is because 123456789123456789123456789.0 is decoded as 1.2345678912345679e26 from the shell as well as by the JSON module. This value should have 25 zeroes when truncated but that will not be the case.

When dealing with such data it may be better to discard the casted data, or to work with strings instead of floats.

Formats

JSV supports multiple formats out of the box with its default implementation, but some are only available under certain conditions that will be specified for each format.

The following listing describes the condition for support and return value type for these default implementations. You can override those implementations by providing your own, as well as providing new formats. This will be described later in this document.

Also, note that by default, JSV format validation will return the original value, that is, the string form of the data. Some format validators can also cast the string to a more interesting data structure, for instance converting a date string to a Date struct. You can enable returning casted values by passing the cast_formats: true option to JSV.validate/3.

The listing below describe values returned with that option enabled.

Important: Some formats require the abnf_parsec library to be available. But we have numerous problems with this library, yielding false negatives with many inputs. An alternative solution will be implemented in future versions.

date

support: Native.
input: "2020-04-22"
output: ~D[2020-04-22]
The format is implemented with the native Date module.
The native Date module supports the YYYY-MM-DD format only. 2024, 2024-W50, 2024-12 will not be valid.

date-time

support: Native.
input: "2025-01-02T00:11:23.416689Z"
output: ~U[2025-01-02 00:11:23.416689Z]
The format is implemented with the native DateTime module.
The native DateTime module supports the YYYY-MM-DD format only for dates. 2024T..., 2024-W50T..., 2024-12T... will not be valid.
Decimal precision is not capped to milliseconds. 2024-12-14T23:10:00.500000001Z will be valid.

duration

support: Requires Elixir 1.17
input: "P1DT4,5S"
output: %Duration{day: 1, second: 4, microsecond: {500000, 1}}
Elixir documentation states that Only seconds may be specified with a decimal fraction, using either a comma or a full stop: P1DT4,5S.
Elixir durations accept negative values.
Elixir durations accept out-of-range values, for instance more than 59 minutes.
Excessive precision (as in "PT10.0000000000001S") will be valid.

email

support: Requires {:mail_address, "~> 1.0"}.
input: "hello@json-schema.org"
output: "hello@json-schema.org" (same value)
Support is limited by the implementation of that library.
The idn-email format is not supported out-of-the-box.

hostname

support: Native
input: "some-host"
output: "some-host" (same value)
Accepts numerical TLDs and single letter TLDs.

ipv4

support: Native
input: "127.0.0.1"
output: {127, 0, 0, 1}

ipv6

support: Native
input: "::1"
output: {0, 0, 0, 0, 0, 0, 0, 1}

iri

support: Requires {:abnf_parsec, "~> 1.0"}.
input: "https://héhé.com/héhé"
output: %URI{scheme: "https", authority: "héhé.com", userinfo: nil, host: "héhé.com", port: 443, path: "/héhé", query: nil, fragment: nil}

iri-reference

support: Requires {:abnf_parsec, "~> 1.0"}.
input: "//héhé"
output: %URI{scheme: nil, authority: "héhé", userinfo: nil, host: "héhé", port: nil, path: nil, query: nil, fragment: nil}

json-pointer

support: Requires {:abnf_parsec, "~> 1.0"}.
input: "/foo/bar/baz"
output: "/foo/bar/baz" (same value)

regex

support: Native
input: "[a-zA-Z0-9]"
output: ~r/[a-zA-Z0-9]/
The format is implemented with the native Regex module.
The Regex module does not follow the ECMA-262 specification.

relative-json-pointer

support: Requires {:abnf_parsec, "~> 1.0"}.
input: "0/foo/bar"
output: "0/foo/bar" (same value)

time

support: Native
input: "20:20:08.378586"
output: ~T[20:20:08.378586]
The format is implemented with the native Time module.
The native Time implementation will completely discard the time offset information. Invalid offsets will be valid.
Decimal precision is not capped to milliseconds. 23:10:00.500000001 will be valid.

unknown

support: Native
input: "anything"
output: "anything" (same value)
No validation or transformation is done.

uri

support: Native, optionally uses {:abnf_parsec, "~> 1.0"}.
input: "http://example.com"
output: %URI{scheme: "http", authority: "example.com", userinfo: nil, host: "example.com", port: 80, path: nil, query: nil, fragment: nil}
Without the optional dependency, the URI module is used and a minimum checks on hostname and scheme presence are made.

uri-reference

support: Native, optionally uses {:abnf_parsec, "~> 1.0"}.
input: "/example-path"
output: %URI{scheme: nil, userinfo: nil, host: nil, port: nil, path: "/example-path", query: nil, fragment: nil}
Without the optional dependency, the URI module will cast most non url-like strings as a path.

uri-template

support: Requires {:abnf_parsec, "~> 1.0"}.
input: "http://example.com/search{?query,lang}"
output: "http://example.com/search{?query,lang}" (same value)

uuid

support: Native
input: "bf22824c-c8a4-11ef-9642-0fdaf117eeb9"
output: "bf22824c-c8a4-11ef-9642-0fdaf117eeb9" (same value)

Custom formats

In order to provide custom formats, or to override default implementations for formats, you may provide a list of modules as the value for the :formats options of JSV.build/2. Such modules must implement the JSV.FormatValidator behaviour.

For instance:

defmodule CustomFormats do
  @behaviour JSV.FormatValidator

  @impl true
  def supported_formats do
    ["greeting"]
  end

  @impl true
  def validate_cast("greeting", data) do
    case data do
      "hello " <> name -> {:ok, %Greeting{name: name}}
      _ -> {:error, :invalid_greeting}
    end
  end
end

With this module you can now call the builder with it:

JSV.build!(raw_schema, formats: [CustomFormats])

Note that this will disable all other formats. If you need to still support the default formats, a helper is available:

JSV.build!(raw_schema,
  formats: [CustomFormats | JSV.default_format_validator_modules()]
)

Format validation modules are checked during the build phase, in order. So you can override any format defined by a module that comes later in the list, including the default modules.

Development

Contributing

Pull requests are welcome given appropriate tests and documentation.

Roadmap

Support for custom vocabularies
Declare a JSON codec module directly as httpc resolver option. This will be implemented if needed, we do not think there will be a strong demand for that.

Files

README.md

Latest commit

History