Schema Generation for nested yml configurations #1027

sladyn98 · 2019-08-27T20:37:35Z

This PR deals with the schema generation for nested yml files.
We would be focusing on
a) Implementation of a describeStructure function for all the configurators which would then be used in the schema generation.
b) Testing Framework for the Schema
Closes #1038
Closes #997

Your checklist for this pull request

🚨 Please review the guidelines for contributing to this repository.

Make sure you are requesting to pull a topic/feature/bugfix branch (right side) and not your master branch!
Ensure that the pull request title represents the desired changelog entry
Please describe what you did
Link to relevant issues in GitHub or in Jenkins JIRA
Link to relevant pull requests, esp. upstream and downstream changes
Did you provide a test-case? That demonstrates feature works or fixes the issue.

… fully validated

Bumped up schema version to v07. Added id instead of ref Added basic schema validation tests.

.../src/main/java/io/jenkins/plugins/casc/impl/configurators/HeteroDescribableConfigurator.java

plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java

oleg-nenashev · 2019-12-16T21:26:45Z

WDYT about #1027 (comment) @timja ?

sladyn98 · 2020-01-05T07:35:21Z

plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java

    /**
     * This function is for the JSONSchemaGeneration
     * @param instance Owner Instance
     * @param context Context to be passed
     * @return CNode object describing the structure of the node
     */
-    public CNode describeStructure(Owner instance, ConfigurationContext context) {
+    public CNode describeForSchema(Owner instance, ConfigurationContext context) {


DescribeStructure would then no longer be needed as the describeForSchema would essentially perform the check.

Do we still need the mode var?

We would not need it in the describeForSchema anymore because if isJsonSchema is set then it would anyways be calling describeStructure which would in turn call describeForSchema

ok please remove then

timja

Few questions on whether some of the code is still required and a little bit of cleanup

timja · 2020-01-07T18:44:17Z

plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java

@@ -111,6 +112,10 @@ boolean isIgnored() {
        return restrictions != null ? restrictions : EMPTY;
    }

+    public void setJsonSchema(boolean jsonSchema) {


is this still required?

plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java

timja · 2020-01-07T18:45:53Z

plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java

@@ -275,7 +323,12 @@ public CNode describe(Owner instance, ConfigurationContext context) throws Confi
     */
    private CNode _describe(Configurator c, ConfigurationContext context, Object value, boolean shouldBeMasked)
            throws Exception {
-        CNode node = c.describe(value, context);
+        CNode node;
+        if(isJsonSchema) {


is this still required?

plugin/src/main/java/io/jenkins/plugins/casc/ConfigurationContext.java

timja · 2020-01-08T08:56:46Z

@oleg-nenashev / @Casz anyone available to give this a (hopefully) final review?

oleg-nenashev

I do not see obvious implementation issues here. My suggestion would be to fix the tests, mark new API as '@restricted(Beta.class)' and release so that adoption can start

oleg-nenashev · 2020-01-09T18:09:10Z

plugin/src/main/java/io/jenkins/plugins/casc/Configurator.java

@@ -167,4 +168,20 @@ default CNode describe(T instance, ConfigurationContext context) throws Exceptio
        return mapping;
    }

+    @CheckForNull


Needs Javadoc since it will be consumed by plugins. @since TODO also

Yeah added javadoc. Confused about the @since TODO ?

timja

All comments look addressed I'll merge this later today if there's no other feedback

plugin/src/main/java/io/jenkins/plugins/casc/Configurator.java

jetersen

Minor code nitpicking

plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java

plugin/src/main/java/io/jenkins/plugins/casc/Configurator.java

Co-Authored-By: Tim Jacomb <t.jacomb@kainos.com>

Co-Authored-By: Joseph Petersen <josephp90@gmail.com>

jetersen · 2020-01-13T00:00:13Z

Not all nested elements are found:

example

                "vSphereCloud": {
                    "additionalProperties": false,
                    "type": "object",
                    "properties": {"idleMinutes": {
                        "description": "",
                        "type": "integer"
                    }}
                },

sladyn98 · 2020-01-13T03:35:00Z

@Casz Yeah we tried to maximize nesting as much as possible, but it still doesn't work in all cases

timja · 2020-01-13T06:37:04Z

There’s lots of limitations... this is a big improvement over what was previously there and it can be incrementally improved

* Add support to backup and restore automatically * Fix the potential output resource leak * Add configs about deploy into alauda update-center * Set alauda as the main branch * Add sonarqube properties file * Fix the potentital issues * Add a merge strategy extension point for the YAML config (#1) * Add YAML merge strategy * Add a new merge strategy * Remove override merge strategy * Clean the unused code lines * Clean the unused imports * Add test cases for merge strategy * Fix the wrong test cases * Take merge strategy name everytime * Give the default strategy a more readable name * Validate top README with the Integrations tests (jenkinsci#1229) * Fix the compilation error caused by merge strategy * Fix issues which found by spotbugs * Remove unused imports * Move profiles from pom.xml into settings file * Remove empty line * rename the shared-library * Schema Generation for nested yml configurations (jenkinsci#1027) * Trying to diagnose some flaky tests (jenkinsci#1243) * Update ci to include code coverage after jenkinsci#1215 (jenkinsci#1241) Co-authored-by: Oleg Nenashev <o.v.nenashev@gmail.com> * Checkstyle ignore release.properties (jenkinsci#1245) * [maven-release-plugin] prepare release configuration-as-code-1.35 * [maven-release-plugin] prepare for next development iteration * Bump jackson.version from 2.10.1 to 2.10.2 Bumps `jackson.version` from 2.10.1 to 2.10.2. Updates `jackson-databind` from 2.10.1 to 2.10.2 - [Release notes](https://github.com/FasterXML/jackson/releases) - [Commits](https://github.com/FasterXML/jackson/commits) Updates `jackson-dataformat-yaml` from 2.10.1 to 2.10.2 - [Release notes](https://github.com/FasterXML/jackson-dataformats-text/releases) - [Commits](FasterXML/jackson-dataformats-text@jackson-dataformats-text-2.10.1...jackson-dataformats-text-2.10.2) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * Correct spelling of hpi.pluginChangelogUrl tag (jenkinsci#1248) * Automatically request code reviews from all JCasC plugin developers (jenkinsci#1249) * [Security] Bump checkstyle from 8.26 to 8.29 Bumps [checkstyle](https://github.com/checkstyle/checkstyle) from 8.26 to 8.29. **This update includes a security fix.** - [Release notes](https://github.com/checkstyle/checkstyle/releases) - [Commits](checkstyle/checkstyle@checkstyle-8.26...checkstyle-8.29) Signed-off-by: dependabot-preview[bot] <support@dependabot.com> * It has been renamed to `master.componentName` (jenkinsci#1256) Co-authored-by: Tim Jacomb <t.jacomb@kainos.com> * Fix the version specification for Jenkins 2.199 (jenkinsci#1268) * Added example for credentials certificate (jenkinsci#1270) * Use the latest shared-library branch * Update test to match latest plugin version (jenkinsci#1273) * Update the integration test for active-directory (jenkinsci#1274) * Add integration test for pipeline-model-definition (jenkinsci#1276) * Fix the error when no casc user dir (#5) * Add missing of mode in the context * Use the same version of jackson-core * Use the same version of jackson-core * Use 2.8 as the major version * Fix the wrong parent id for the plugin releasing * Move the user config into a backup dir * Run build under java pod Co-authored-by: Victor Martinez <victormartinezrubio@gmail.com> Co-authored-by: Sladyn <sladynnunes98@gmail.com> Co-authored-by: Jesse Glick <jglick@cloudbees.com> Co-authored-by: Joseph Petersen <josephp90@gmail.com> Co-authored-by: Oleg Nenashev <o.v.nenashev@gmail.com> Co-authored-by: Tim Jacomb <t.jacomb@kainos.com> Co-authored-by: dependabot-preview[bot] <27856297+dependabot-preview[bot]@users.noreply.github.com> Co-authored-by: Mark Waite <mark.earl.waite@gmail.com> Co-authored-by: Jhon Mike <jhon.msdev@gmail.com> Co-authored-by: Dawid Gosławski <50369002+dg-nvm@users.noreply.github.com> Co-authored-by: choeffer <33625292+choeffer@users.noreply.github.com> Co-authored-by: Francisco Fernández <31063239+fcojfernandez@users.noreply.github.com>

Sladyn and others added 24 commits August 7, 2019 21:43

Added initial schema test

774506b

Added Schema Validation functions

7051ca9

Added test jsonSchemGenerator

f1b2d30

Added test to generate an intermediate schema

17079c7

Added schema generation for a single descriptor

886bc5a

Completed the string generation 2/4 of the schema

0271cd3

Added exclusion

522f4e5

generic get configurators

f264c70

Finished Descriptor Configuration generation

076fe14

Base configurator and HetroDescribable Configurator Schema Generation…

2bd7f9d

… fully validated

Added Base configurators with no attributes

0da381b

Added Base configurator attribute enumeration schema

3d45f6b

Added Non-enumerated attributes to schema

cd9a9dd

Added fully validated Configured schema

4bf306c

Added validation initial test

4657e0c

Using JSON objects to construct Schema

c734c0f

Added valid schema integration tests

fe1ce60

Converted class names to simple names

8108038

Upgraded schema version

38dad38

Bumped up schema version to v07. Added id instead of ref Added basic schema validation tests.

Changes to test and pom.xml

d89e24c

Added functions to split up the generateSchema() function

3c4510c

Added additional methods in generateSchema()

f079505

Code Cleanup

76c0932

Inital impl for describeStructure

623914f

sladyn98 commented Aug 27, 2019

View reviewed changes

.../src/main/java/io/jenkins/plugins/casc/impl/configurators/HeteroDescribableConfigurator.java Outdated Show resolved Hide resolved

.../src/main/java/io/jenkins/plugins/casc/impl/configurators/HeteroDescribableConfigurator.java Outdated Show resolved Hide resolved

Incremental Impl changes

13e893a

sladyn98 force-pushed the nested_yaml_schema branch from 521d972 to 13e893a Compare August 27, 2019 20:54

sladyn98 commented Aug 27, 2019

View reviewed changes

.../src/main/java/io/jenkins/plugins/casc/impl/configurators/HeteroDescribableConfigurator.java Outdated Show resolved Hide resolved

sladyn98 added the dev-tools JCasC Community Bridge Dev-Tools Project label Aug 27, 2019

sladyn98 self-assigned this Aug 27, 2019

timja reviewed Dec 14, 2019

View reviewed changes

plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java Outdated Show resolved Hide resolved

Improved warning string

423ee6c

oleg-nenashev self-requested a review December 16, 2019 21:26

Added describeForSchema Function

d5d10f4

sladyn98 requested a review from timja January 5, 2020 07:14

sladyn98 commented Jan 5, 2020

View reviewed changes

Sladyn added 3 commits January 5, 2020 13:25

Fixed Checkstyle

1cdd881

Removed boolean

1040a6a

Fixed ClassCast Exception

a242048

timja reviewed Jan 7, 2020

View reviewed changes

timja approved these changes Jan 7, 2020

View reviewed changes

Combined declaration and assignment

0cfde4a

oleg-nenashev reviewed Jan 9, 2020

View reviewed changes

Added javadoc and beta class

1aa84b1

sladyn98 requested review from oleg-nenashev and timja January 9, 2020 20:02

timja approved these changes Jan 10, 2020

View reviewed changes

plugin/src/main/java/io/jenkins/plugins/casc/Configurator.java Show resolved Hide resolved

jetersen approved these changes Jan 10, 2020

View reviewed changes

Sladyn and others added 4 commits January 10, 2020 20:20

Update plugin/src/main/java/io/jenkins/plugins/casc/Configurator.java

1ff2aaf

Co-Authored-By: Tim Jacomb <t.jacomb@kainos.com>

Update plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java

15a7500

Co-Authored-By: Joseph Petersen <josephp90@gmail.com>

Update plugin/src/main/java/io/jenkins/plugins/casc/Configurator.java

b468ee9

Co-Authored-By: Joseph Petersen <josephp90@gmail.com>

Update plugin/src/main/java/io/jenkins/plugins/casc/Attribute.java

772445a

Co-Authored-By: Joseph Petersen <josephp90@gmail.com>

timja merged commit b53ea06 into jenkinsci:master Jan 10, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Schema Generation for nested yml configurations #1027

Schema Generation for nested yml configurations #1027

sladyn98 commented Aug 27, 2019 •

edited by jetersen

Loading

oleg-nenashev commented Dec 16, 2019

sladyn98 Jan 5, 2020

timja Jan 5, 2020

sladyn98 Jan 6, 2020

timja Jan 6, 2020

sladyn98 Jan 7, 2020

timja left a comment

timja Jan 7, 2020

timja Jan 7, 2020

timja commented Jan 8, 2020

oleg-nenashev left a comment

oleg-nenashev Jan 9, 2020 •

edited by jetersen

Loading

sladyn98 Jan 9, 2020 •

edited by jetersen

Loading

timja left a comment

jetersen left a comment

jetersen commented Jan 13, 2020

sladyn98 commented Jan 13, 2020

timja commented Jan 13, 2020

Schema Generation for nested yml configurations #1027

Schema Generation for nested yml configurations #1027

Conversation

sladyn98 commented Aug 27, 2019 • edited by jetersen Loading

Your checklist for this pull request

oleg-nenashev commented Dec 16, 2019

sladyn98 Jan 5, 2020

Choose a reason for hiding this comment

timja Jan 5, 2020

Choose a reason for hiding this comment

sladyn98 Jan 6, 2020

Choose a reason for hiding this comment

timja Jan 6, 2020

Choose a reason for hiding this comment

sladyn98 Jan 7, 2020

Choose a reason for hiding this comment

timja left a comment

Choose a reason for hiding this comment

timja Jan 7, 2020

Choose a reason for hiding this comment

timja Jan 7, 2020

Choose a reason for hiding this comment

timja commented Jan 8, 2020

oleg-nenashev left a comment

Choose a reason for hiding this comment

oleg-nenashev Jan 9, 2020 • edited by jetersen Loading

Choose a reason for hiding this comment

sladyn98 Jan 9, 2020 • edited by jetersen Loading

Choose a reason for hiding this comment

timja left a comment

Choose a reason for hiding this comment

jetersen left a comment

Choose a reason for hiding this comment

jetersen commented Jan 13, 2020

sladyn98 commented Jan 13, 2020

timja commented Jan 13, 2020

sladyn98 commented Aug 27, 2019 •

edited by jetersen

Loading

oleg-nenashev Jan 9, 2020 •

edited by jetersen

Loading

sladyn98 Jan 9, 2020 •

edited by jetersen

Loading