Add json schemas to docs

[mvpatel2000]

Adds json schemas to docs. Adds a test to make sure the generated files are the most recent version

[mvpatel2000]

Still need to verify docs generate properly. Tagging for review for discussion of questions

[ravi-mosaicml]

Thanks for adding this! Wow, this will be powerful.

Discussed offline, but can we use jsonschema $defs or $refs to reduce the file size of the json schemas?

Afterwards, I'll take a look on how this shows on the docs site.

[moinnadeem]

Is this related to CO-669, CO-670, or CO-671? @ravi-mosaicml @mvpatel2000

[mvpatel2000]

Is this related to CO-669, CO-670, or CO-671? @ravi-mosaicml @mvpatel2000

CO-671. The first two are done, though there's some changes we had to make as a result of this PR here.

[ravi-mosaicml]

Overall this is awesome -- see comments.

In the JSONSchema generation (likely in the YAHP PR), one more area to make it more compact:

is it possible to specify properties and patternPropertiesin the same object? Trying to reduce some duplication -- e.g. this seems redundant

{
                    "additionalProperties": false,
                    "properties": {
                        "deeplabv3": {
                            "$ref": "#/$defs/DeepLabV3Hparams"
                        }
                    },
                    "type": "object"
                },
                {
                    "additionalProperties": false,
                    "patternProperties": {
                        "^deeplabv3\\+": {
                            "$ref": "#/$defs/DeepLabV3Hparams"
                        }
                    },
                    "type": "object"
                },

Before approving, would like to fix the docs build + bump the new yahp version and upgrade yahp in the dependencies.

[ravi-mosaicml]

Instead of generating a new jsonschema for each hparams class, and linking in the docs to that file, can we link the the definition of that hparams class within the experiment hparams json schema? The experiment hparams is the "root", and all possible hparams should be used within it. The JSONSchema plugin supports JSONPointer notation.

For example:

Suggested change

	.. jsonschema:: ../json_schemas/trainer_hparams.json
	.. jsonschema:: ../json_schemas/experiment_hparams.json#/defs/trainer_hparams

This should help reduce the duplication, and make the docs site more accessible (since there will be just one jsonschema entry)

[mvpatel2000]

You're robbing me of my 25k lines committed though :(

[mvpatel2000]

re offline discussion, we actually do need all the files to enable autocomplete

[ravi-mosaicml]

nit: can you add a shebang? #!/usr/bin/env python3 should suffice

[mvpatel2000]

is there a reason why? It looks like no other file has this?

[mvpatel2000]

Note: All glue related yaml files are failing because of this. The rest are failing because we still haven't bumped yahp version

[mvpatel2000]

this is not really human readable sad. Discussing alternatives offline

[mvpatel2000]

Closing this PR -- after a lot of offline discussion, we've decided not to insert JSON Schema into the docs. We'll instead do a skinny PR with just the test infra.