Make JSON write and parse incompatible values as Variant consistently

[Xrayez]

Fixes #33161, Closes #11866.
Remake of #33163.
Helps #33137.

TL;DR: try to write compatible JSON with native types, but when it can't, just stuff vars into JSON strings. This should work as before but "the right way".

---

The JSON writer is rewritten to treat vars by JSON types. This is needed to better control the writing state. Namely, object keys determine whether a Variant can be written as another JSON compatible type, or whether it should be converted with VariantWriter (aka var2str) to be embedded into a JSON string which can be parsed later with VariantParser (aka str2var).

This effectively removes the need to manually convert incompatible types with var2str and str2var via scripting. JSON writer and parser use Variant serialization methods over its visual representation which makes the conversion behavior possible, see issues like #27529, godotengine/godot-proposals#1351, #48169.

Example

Here's an example JSON serialized from dictionary with to_json(dict):

JSON

{
    "10": "Color( 1, 1, 1, 1 )",
    "foo": "bar",
    "nested": {
        "{Basis( 1, 0, 0, 0, 1, 0, 0, 0, 1 ):Plane( 0, 0, 0, 0 )}": [
            "3D"
        ]
    },
    "{[3,2]:1}": {
        "[9,8]": 7
    },
    "{PoolStringArray( \"foo\", \"bar\", \"baz\" ):\"wow\"}": "PoolColorArray( 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1 )",
    "[1,2]": "Vector2( 10, -10.5 )",
    "[null,true,10,20.5,\"foobar\",Vector2( 10, 20 ),Rect2( 0, 0, 10, 20 ),Vector3( 1, 2, 3 ),Transform2D( 2, 0, 0, 2, 0, 0 ),Plane( 0, 0, 10, 10 ),Quat( 0, 0, 10, 10 ),AABB( 0, 0, 0, 1, 1, 1 ),Basis( 1, 0, 0, 0, 1, 0, 0, 0, 1 ),Transform( 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0 ),Color( 1, 0, 1, 1 ),NodePath(\"node\"),[9.0,8.0,7.0],PoolByteArray(  ),PoolIntArray( 1, 2, 3 ),PoolRealArray( 1.5, 2.5, 3.5 ),PoolStringArray( \"foo\", \"bar\", \"baz\" ),PoolVector2Array( 1, 0, 0, -1, -1, 0, 0, 1 ),PoolVector3Array( 0, 0, 0, 0, 0, 0, 0, 0, 0 ),PoolColorArray( 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1 )]": [
        null,
        true,
        10,
        20.5,
        "foobar",
        "Vector2( 10, 20 )",
        "Rect2( 0, 0, 10, 20 )",
        "Vector3( 1, 2, 3 )",
        "Transform2D( 2, 0, 0, 2, 0, 0 )",
        "Plane( 0, 0, 10, 10 )",
        "Quat( 0, 0, 10, 10 )",
        "AABB( 0, 0, 0, 1, 1, 1 )",
        "Basis( 1, 0, 0, 0, 1, 0, 0, 0, 1 )",
        "Transform( 1, 0, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0 )",
        "Color( 1, 0, 1, 1 )",
        "NodePath(\"node\")",
        [
            9,
            8,
            7
        ],
        "PoolByteArray(  )",
        [
            1,
            2,
            3
        ],
        [
            1.5,
            2.5,
            3.5
        ],
        [
            "foo",
            "bar",
            "baz"
        ],
        "PoolVector2Array( 1, 0, 0, -1, -1, 0, 0, 1 )",
        "PoolVector3Array( 0, 0, 0, 0, 0, 0, 0, 0, 0 )",
        "PoolColorArray( 1, 0, 0, 1, 0, 1, 0, 1, 0, 0, 1, 1 )"
    ],
    "PoolIntArray( 1, 2, 3, 4 )": [
        "foo",
        "bar",
        "baz"
    ]
}

Original dictionary parse_json(json):

Dictionary

var variants = [
	null,
	true,
	10, # 2
	20.5,
	"foobar",
	Vector2(10, 20),
	Rect2(0, 0, 10, 20),
	Vector3(1, 2, 3),
	Transform2D(Vector2(2, 0), Vector2(0, 2), Vector2(0, 0)),
	Plane(0, 0, 10, 10),
	Quat(0, 0, 10, 10),
	AABB(Vector3.ZERO, Vector3.ONE),
	Basis(),
	Transform(Basis(), Vector3.ZERO),
	Color(1, 0, 1),
	NodePath("node"),
	Array([9.0, 8.0, 7.0]),
	PoolByteArray(),
	PoolIntArray([1, 2, 3]), # 18
	PoolRealArray([1.5, 2.5, 3.5]), # 19
	PoolStringArray(["foo", "bar", "baz"]), # 20
	PoolVector2Array([Vector2.RIGHT, Vector2.UP, Vector2.LEFT, Vector2.DOWN]),
	PoolVector3Array([Vector3.AXIS_X, Vector3.AXIS_Y, Vector3.AXIS_Z]),
	PoolColorArray([Color.red, Color.green, Color.blue])
]

var dict_sample = {
	10: Color(1,1,1,1),
	"foo": "bar",
	{[3, 2] : 1}: {[9, 8] : 7},
	{PoolStringArray(["foo", "bar", "baz"]): "wow"} : PoolColorArray([Color.red, Color.green, Color.blue]),
	[1, 2]: Vector2(10, -10.5),
	PoolIntArray([1, 2, 3, 4]): PoolStringArray(["foo", "bar", "baz"]),
	variants: variants,
	"nested": {{Basis(): Plane()}: ["3D"]}
}

Test project

I had to write some tests to keep regressions from occurring during development. See the script comments on what, how, and why.

json-convert-incompatible.zip

[Xrayez]

There could be certain performance issues having to write and parse inner JSON strings for Variants, but it might be negligible, I haven't tested this. I'd prefer perfectly working feature over speed for this personally. I was thinking that some convention could be used to encode that JSON string contains a Variant, for instance @ reserved character could be inserted at the beginning:

{
    "my_color": "@Color( 1, 1, 1, 1 )",
}

This could also avoid v_err != OK || _is_compatible_value(value)) check which I added because some strings could be falsely converted to other types. For instance, str2var("3D") would be parsed as a number, returning 3. This case is covered in the test project actually.

[Xrayez]

I'm really not sure whether something like this has a chance to be merged, considering that it's a core feature used by many, and I wouldn't want to be responsible for regressions. Nonetheless, it was nice to make this work. Feel free to close this if you do not want to compromise the stability.

I'm unlikely to create a proposal at GIP to discuss this, and tbh I personally don't need this myself at the moment.

[aaronfranke]

@Xrayez Thanks for being upfront about your lack of interest, I'm closing this and adding "salvageable" in case others want to continue your work.

[Xrayez]

There was some discussion on Discord that this feature could be made opt-in, but considering the amount of complexity this adds it's probably not worth it.

But I have to say that we should either forbid converting JSON-incompatible types altogether, or actually allow the types to be converted properly by the Variant parser and writer (as this PR proposes), currently it's just a mix of both (people don't know what to expect), or perhaps come up with Godot's own way to encode that in JSON compatible manner (which is probably a lot more work in comparison) without relying on Variant parser and writer.

The test project is valuable in and of itself.

Thanks for being upfront about your lack of interest

Well I think I have to be more honest, the most important reason for the lack of interest is not because I'm not personally interested in this (this PR originates from the issues I've stumbled upon while developing #33137, and makes the feature set more complete), but because I haven't received positive feedback for core devs throughout this time, and sorry but I don't have enough lifetime to write formal proposals for each bug/feature I want to integrate, and given that the problem must exist, I don't see the point either. 😛

[Xrayez]

I could resurrect this given the fact that we have testing framework integrated #40148, the test project would have to be converted to doctest cases to ensure no regressions occur, but again this takes some work to write a test suite for JSON parser and writer, and I'd still like to receive some feedback on this in any case before doing anything.

[ghost]

Tested out this code and works well.

I have tested it on all the built in types Vector2, Vector3, Transform2D, Color and so forth. Converts a Godot type into it's string equivalent then restores it back into a Godot object.

The way Godot currently handles JSON is odd as it allows exporting of unsupported types but breaks them. The only way to fix this is using complex scripts.

This implementation allows data to be exchanged between Godot applications as JSON objects and data to be saved to JSON then restored. Only Godot understands these custom string datatypes and to any other application is just string. I think is a good compromise without breaking JSON specs.