Fix fill_value serialization issues

[moradology]

The current serialization of fill_value in ArrayV2Metadata does not fully conform to the spec, particularly for:

• NaN and Infinity values, which must be serialized as strings ("NaN", "Infinity", "-Infinity").
• Complex numbers (np.complex64, np.complex128), which must be stored as two-element arrays [real, imag] according to the above Nan/Inf rules.

Changes

• Updated _sanitize_fill_value() to enforce correct JSON serialization.
• Fixed test_v2meta_fill_value_serialization() to compare expected and actual JSON using a normalized representation.
• Introduced property-based testing with Hypothesis to generate valid input cases and verify compliance.
• Enforce compliance (to some degree) by setting allow_nan to False on json.dumps

Resolves: #2741

TODO:

• Add unit tests and/or doctests in docstrings
• Add docstrings and API docs for any new/modified user-facing classes and functions
• New/modified features documented in docs/user-guide/*.rst
• Changes documented as a new file in changes/
• GitHub Actions have all passed
• Test coverage is 100% (Codecov passes)

[moradology]

The bad news is, this issue is going to be slightly more than I've said here. The good news is that the property-based tests caught some edge cases.

[dcherian]

Nice, I've been meaning to add this to Zarr:

@st.composite
def v3_array_metadata(draw: st.DrawFn) -> bytes:
    from zarr.codecs.bytes import BytesCodec
    from zarr.core.chunk_grids import RegularChunkGrid
    from zarr.core.chunk_key_encodings import DefaultChunkKeyEncoding
    from zarr.core.metadata.v3 import ArrayV3Metadata

    # separator = draw(st.sampled_from(['/', '\\']))
    shape = draw(array_shapes)
    ndim = len(shape)
    chunk_shape = draw(npst.array_shapes(min_dims=ndim, max_dims=ndim))
    dtype = draw(zrst.v3_dtypes())
    fill_value = draw(npst.from_dtype(dtype))
    dimension_names = draw(
        st.none() | st.lists(st.none() | simple_text, min_size=ndim, max_size=ndim)
    )

    metadata = ArrayV3Metadata(
        shape=shape,
        data_type=dtype,
        chunk_grid=RegularChunkGrid(chunk_shape=chunk_shape),
        fill_value=fill_value,
        attributes=draw(simple_attrs),
        dimension_names=dimension_names,
        chunk_key_encoding=DefaultChunkKeyEncoding(separator="/"),  # FIXME
        codecs=[BytesCodec()],
        storage_transformers=(),
    )

    return metadata.to_buffer_dict(prototype=default_buffer_prototype())["zarr.json"]

What do you think of a array_metadata_json(zarr_formats...) strategy that just returns the JSON and we can test whether that satisfies the spec for V2 and V3?

[moradology]

I love the idea. I was thinking the other day that the obvious path out of the bugs that are currently popping up would be property-based testing, so I was pretty pleased to see that there's already some work in that direction

Out of curiosity, do we have something like json schema that we could apply against the outputs to at least verify structure? We'd still need to define all the rules that exist in terms of value/type dependencies etc. but that's an easy win if it exists somewhere

[dcherian]

Out of curiosity, do we have something like json schema that we could apply against the outputs to at least verify structure?

Don't know. ping @jhamman @d-v-b

[d-v-b]

I'm not aware of a JSON schema definition for the array metadata. If one existed, it would necessarily only support partial validation, because JSON schema can't express certain invariants in the metadata document, like the requirement that dimensional attributes (shape, chunk_shape, etc) be consistent.

[dcherian]

A fairly easy alternative way to handle this would be to simply write a test that takes the arrays strategy, extracts the metadata, converts to JSON, and then asserts that the JSON meets spec (as best as we can).

I still think a generic metadata strategy is probably useful.

[moradology]

Yeah, you'd definitely need json-schema and custom validation rules that encode relationships among different fields

[moradology]

Did some refactoring and organization of property-based testing code and added round trip testing for ArrayV2Metadata (which hopefully captures some important expectations about these things. Still need to fully satisfy the linter and make sure similar guarantees are tested for ArrayV3Metadata

[dcherian]

Hye @moradology I added an array_metadata strategy in

zarr-python/src/zarr/testing/strategies.py

Line 110 in e8bfb64

def array_metadata(

would you mind merging your changes in with that strategy please?

[moradology]

Definitely - do you mind my pulling the generators up into their own submodule as done here, given their similar and unique functionality?

[dcherian]

I think testing/strategies.py is right place. Everything in there already handles the v2 vs v3 complexity

[moradology]

@dcherian Maybe I got out ahead of myself. Do you think the small amount of extra organization I added is undesirable? Basically, I broke strategies.py out into strategies/array_metadata_generators.py, strategies/array_generators.py, strategies/dtype_generators.py while using strategies/__init__.py to keep imports the same.

My thought was just that continuing to add property-based tests is probably a good idea and the single strategies.py file will likely get a bit long

[dcherian]

It feels a bit premature to me. In any case, I find it nice to keep such refactoring PRs separate so that we can review (much) smaller diffs. In my experience, keeping the diffs small is the best way to keep open-source contributions easy to merge.

How about just keeping them in strategies for now and we can refactor later as needed.

[moradology]

Sounds like a plan. As a relatively new contributor to this library, I appreciate the firm opinions!

[dcherian]

Thank you for considering the opinions in a constructive manner!

[moradology]

OK, so to get the property-based tests working I had to make some decisions about what serialization strategies looked like. I tried to follow the instructions available here, but other issues that have come to the surface in the history of zarr-python suggest there are likely blindspots in this document. This part, in particular, deserves special attention

[moradology]

A bit confused about the code coverage going down. The tested constraints are definitely a bit tighter than they were, if anything

[dcherian]

This could look more like https://github.com/zarr-developers/zarr-python/pull/2847/files#diff-d318cba7c9e4a6983338cf21df1db66aab796137a2fb4a76ce48c0afa17de2f9

We could abstract out a assert_valid_v2_json_dict and similarly for v3.

[moradology]

Makes sense to me - would you like me to wait for that branch to go in and then add these validations to the broader test you have in mind?

[dcherian]

Let's just copy the test over and you can expand them here. I can handle the merge conflicts. It may very well be that this one goes in first :)

[moradology]

🫡

[moradology]

OK, so I'm curious about something. See this line: https://github.com/zarr-developers/zarr-python/blob/main/src/zarr/core/metadata/v3.py#L413C1-L414C1
Is that type correct? If so, it seems that numpy types should be serialized after to_dict. As of now, they're not (this PR changes that for v2 but not yet for v3). So what's the desired behavior?

[dcherian]

don't know. ping @d-v-b

[d-v-b]

no, that type is wrong and it bugs me! for v3 metadata to_dict returns an instance of DataType for the data_type key, and I think we registered that type with a custom JSON encoder.

[d-v-b]

I don't remember why this decision was made but we should definitely fix it. fwiw, dict[str, JSON] is also sub-optimal, given that the keys of the metadata document are (almost) entirely static, and so we could do typeddicts here

[moradology]

OK, so the immediately obvious thing is that the serialization logic I've added to to_dict needs to be pushed down into to_buffer_dict and to_dict should retain its (potentially) not-directly-serializable python types. I assumed serialization should happen in to_dict because of JSON in the type sig but also because there's some serialization happening in that function already: https://github.com/zarr-developers/zarr-python/blob/main/src/zarr/core/metadata/v2.py#L199-L204. I will address that, too

From there, a typed-dict can be implemented for to_dict output.

[dcherian]

as promised, I fixed the conflict :)