Manifest arrays use arrayv3metadata #429
Conversation
| """ | ||
| Individual chunk size by number of elements. | ||
| """ | ||
| if isinstance(self._metadata.chunk_grid, RegularChunkGrid): |
There was a problem hiding this comment.
I think this should be checked at construction time, because the concatenation methods assume a regular chunk grid too.
| """ | ||
| Array shape by number of elements along each dimension. | ||
| """ | ||
| return tuple(int(length) for length in list(self.metadata.shape)) |
There was a problem hiding this comment.
Can you not just return self.metadata.shape?
| new_zarray = first_arr.zarray.replace( | ||
| shape=tuple(new_shape), | ||
| ) | ||
| metadata_copy = first_arr.metadata.to_dict().copy() |
There was a problem hiding this comment.
This seems overly verbose, and something we will want in multiple places. Let's make a replace constructor function, before adding a method upstream.
…not happy about this)
| @@ -87,8 +87,7 @@ def variable_to_kerchunk_arr_refs(var: Variable, var_name: str) -> KerchunkArrRe | |||
| for chunk_key, entry in marr.manifest.dict().items() | |||
| } | |||
|
|
|||
| zarray = marr.zarray.replace(zarr_format=2) | |||
|
|
|||
| array_v2_metadata = convert_v3_to_v2_metadata(marr.metadata) | |||
There was a problem hiding this comment.
I don't think we want to go with this approach, perhaps we should allow for manifest arrays to have ArrayV2Metadata for writing to kerchunk.
There was a problem hiding this comment.
don't think we want to go with this approach
Why not?
perhaps we should allow for manifest arrays to have
ArrayV2Metadatafor writing to kerchunk.
I really don't see how that would make anything easier. Inside variable_to_kerchunk_arr_refs is the only place you should need to deal with v2 metadata, so why try to generalize it in the rest of the package if you know you won't need it anywhere else?
There was a problem hiding this comment.
that makes sense to me, I think I was just concerned about bugs trying to go between ArrayV3Metadata and ArrayV2Metadata. But having a single interface for metadata in memory is probably safer over all.
There was a problem hiding this comment.
Oh, I recalled, I am not sure yet about the best way to go from ArrayV3Metadata codecs to V2 compressors and filters. I don't see anything yet in zarr-python for how to do that. I welcome any suggestions.
There was a problem hiding this comment.
Looking closer at https://github.com/zarr-developers/zarr-python/blob/main/src/zarr/core/array.py#L4290-L4334 it looks like we can parse the codecs in the ArrayV3Metadata:
filters, serializer, codecs = array_v3_metadata.codecsAlthough serializer has no significance in Zarr v2, as far as I can tell.
virtualizarr/zarr.py
Outdated
| from zarr.abc.codec import ArrayArrayCodec, ArrayBytesCodec, BytesBytesCodec | ||
|
|
||
|
|
||
| def identify_codec(codec): |
There was a problem hiding this comment.
@d-v-b we have to extract compressors and filters from ArrayV3Metadata for use in writing V2 metadata to kerchunk and also for use in the icechunk writer where we call zarr-python's group.require_array - is this method the best way to do so?
There was a problem hiding this comment.
yes, that function reflects the mapping between words and class names we use in zarr-python. But this conceptualization is specific to zarr v3: in zarr v2, any function could be a compressor, and any function could be a filter. It wasn't strictly typed at all. So there is no guarantee that something which was a filter in zarr v2 is a valid filter in zarr v3.
This is still very much a WIP - many tests and implementations still need to be fixed.
A few notes:
writers/kerchunk.py - variable_to_kerchunk_arr_refs requires a ManifestArray, but we will need to convert metadata to valid kerchunk v2 metadataQuestions
{"name": "zlib", "configuration": {"level": 1}}) or numcodecs ({"id": "zlib", "level": 1})? I think we have to convert to zarr codec objects to use zarr's get_codec_class anyways.Checklist
docs/releases.rstapi.rst