Skip to content

Commit

Permalink
update numcodecs tests and docs for zarr-python 3rc1
Browse files Browse the repository at this point in the history
  • Loading branch information
@normanrz
normanrz committed Jan 4, 2025
1 parent 095de5c commit 411d6df
Show file tree
Hide file tree
Showing 3 changed files with 76 additions and 66 deletions.
43 changes: 23 additions & 20 deletions docs/zarr3.rst
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ Zarr 3 codecs
.. automodule:: numcodecs.zarr3


Bytes-to-bytes codecs
---------------------
Compressors (bytes-to-bytes codecs)
-----------------------------------
.. autoclass:: Blosc()

.. autoattribute:: codec_name
Expand Down Expand Up @@ -40,58 +40,61 @@ Bytes-to-bytes codecs
.. autoattribute:: codec_name


Array-to-array codecs
---------------------
.. autoclass:: Delta()
Checksum codecs (bytes-to-bytes codecs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Need to be used as ``compressors`` in zarr-python.

.. autoclass:: CRC32()

.. autoattribute:: codec_name

.. autoclass:: BitRound()
.. autoclass:: CRC32C()

.. autoattribute:: codec_name

.. autoclass:: FixedScaleOffset()
.. autoclass:: Adler32()

.. autoattribute:: codec_name

.. autoclass:: Quantize()
.. autoclass:: Fletcher32()

.. autoattribute:: codec_name

.. autoclass:: PackBits()
.. autoclass:: JenkinsLookup3()

.. autoattribute:: codec_name

.. autoclass:: AsType()

.. autoattribute:: codec_name
Filters (array-to-array codecs)
-------------------------------
.. autoclass:: Delta()

.. autoattribute:: codec_name

Bytes-to-bytes checksum codecs
------------------------------
.. autoclass:: CRC32()
.. autoclass:: BitRound()

.. autoattribute:: codec_name

.. autoclass:: CRC32C()
.. autoclass:: FixedScaleOffset()

.. autoattribute:: codec_name

.. autoclass:: Adler32()
.. autoclass:: Quantize()

.. autoattribute:: codec_name

.. autoclass:: Fletcher32()
.. autoclass:: PackBits()

.. autoattribute:: codec_name

.. autoclass:: JenkinsLookup3()
.. autoclass:: AsType()

.. autoattribute:: codec_name


Array-to-bytes codecs
---------------------

Serializers (array-to-bytes codecs)
-----------------------------------
.. autoclass:: PCodec()

.. autoattribute:: codec_name
Expand Down
84 changes: 43 additions & 41 deletions numcodecs/tests/test_zarr3.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -66,17 +66,19 @@ def test_docstring(codec_class: type[numcodecs.zarr3._NumcodecsCodec]):
numcodecs.zarr3.Shuffle,
],
)
def test_generic_codec_class(store: StorePath, codec_class: type[numcodecs.zarr3._NumcodecsCodec]):
def test_generic_compressor(

Check warning on line 69 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L69 

Added line #L69 was not covered by tests
store: StorePath, codec_class: type[numcodecs.zarr3._NumcodecsBytesBytesCodec]
):
data = np.arange(0, 256, dtype="uint16").reshape((16, 16))

with pytest.warns(UserWarning, match=EXPECTED_WARNING_STR):
a = Array.create(
a = zarr.create_array(

Check warning on line 75 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L75 

Added line #L75 was not covered by tests
store / "generic",
shape=data.shape,
chunk_shape=(16, 16),
chunks=(16, 16),
dtype=data.dtype,
fill_value=0,
codecs=[BytesCodec(), codec_class()],
compressors=[codec_class()],
)

a[:, :] = data.copy()
Expand All @@ -100,62 +102,61 @@ def test_generic_codec_class(store: StorePath, codec_class: type[numcodecs.zarr3
)
def test_generic_filter(
store: StorePath,
codec_class: type[numcodecs.zarr3._NumcodecsCodec],
codec_class: type[numcodecs.zarr3._NumcodecsArrayArrayCodec],
codec_config: dict[str, JSON],
):
data = np.linspace(0, 10, 256, dtype="float32").reshape((16, 16))

with pytest.warns(UserWarning, match=EXPECTED_WARNING_STR):
a = Array.create(
a = zarr.create_array(

Check warning on line 111 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L111 

Added line #L111 was not covered by tests
store / "generic",
shape=data.shape,
chunk_shape=(16, 16),
chunks=(16, 16),
dtype=data.dtype,
fill_value=0,
codecs=[
filters=[
codec_class(**codec_config),
BytesCodec(),
],
)

a[:, :] = data.copy()
a = Array.open(store / "generic")
a = zarr.open_array(store / "generic", mode="r")

Check warning on line 123 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L123 

Added line #L123 was not covered by tests
np.testing.assert_array_equal(data, a[:, :])


def test_generic_filter_bitround(store: StorePath):
data = np.linspace(0, 1, 256, dtype="float32").reshape((16, 16))

with pytest.warns(UserWarning, match=EXPECTED_WARNING_STR):
a = Array.create(
a = zarr.create_array(

Check warning on line 131 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L131 

Added line #L131 was not covered by tests
store / "generic_bitround",
shape=data.shape,
chunk_shape=(16, 16),
chunks=(16, 16),
dtype=data.dtype,
fill_value=0,
codecs=[numcodecs.zarr3.BitRound(keepbits=3), BytesCodec()],
filters=[numcodecs.zarr3.BitRound(keepbits=3)],
)

a[:, :] = data.copy()
a = Array.open(store / "generic_bitround")
a = zarr.open_array(store / "generic_bitround", mode="r")

Check warning on line 141 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L141 

Added line #L141 was not covered by tests
assert np.allclose(data, a[:, :], atol=0.1)


def test_generic_filter_quantize(store: StorePath):
data = np.linspace(0, 10, 256, dtype="float32").reshape((16, 16))

with pytest.warns(UserWarning, match=EXPECTED_WARNING_STR):
a = Array.create(
a = zarr.create_array(

Check warning on line 149 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L149 

Added line #L149 was not covered by tests
store / "generic_quantize",
shape=data.shape,
chunk_shape=(16, 16),
chunks=(16, 16),
dtype=data.dtype,
fill_value=0,
codecs=[numcodecs.zarr3.Quantize(digits=3), BytesCodec()],
filters=[numcodecs.zarr3.Quantize(digits=3)],
)

a[:, :] = data.copy()
a = Array.open(store / "generic_quantize")
a = zarr.open_array(store / "generic_quantize", mode="r")

Check warning on line 159 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L159 

Added line #L159 was not covered by tests
assert np.allclose(data, a[:, :], atol=0.001)


Expand All @@ -164,27 +165,27 @@ def test_generic_filter_packbits(store: StorePath):
data[0:4, :] = True

with pytest.warns(UserWarning, match=EXPECTED_WARNING_STR):
a = Array.create(
a = zarr.create_array(

Check warning on line 168 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L168 

Added line #L168 was not covered by tests
store / "generic_packbits",
shape=data.shape,
chunk_shape=(16, 16),
chunks=(16, 16),
dtype=data.dtype,
fill_value=0,
codecs=[numcodecs.zarr3.PackBits(), BytesCodec()],
filters=[numcodecs.zarr3.PackBits()],
)

a[:, :] = data.copy()
a = Array.open(store / "generic_packbits")
a = zarr.open_array(store / "generic_packbits", mode="r")

Check warning on line 178 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L178 

Added line #L178 was not covered by tests
np.testing.assert_array_equal(data, a[:, :])

with pytest.raises(ValueError, match=".*requires bool dtype.*"):
Array.create(
zarr.create_array(

Check warning on line 182 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L182 

Added line #L182 was not covered by tests
store / "generic_packbits_err",
shape=data.shape,
chunk_shape=(16, 16),
chunks=(16, 16),
dtype="uint32",
fill_value=0,
codecs=[numcodecs.zarr3.PackBits(), BytesCodec()],
filters=[numcodecs.zarr3.PackBits()],
)


Expand All @@ -198,26 +199,30 @@ def test_generic_filter_packbits(store: StorePath):
numcodecs.zarr3.JenkinsLookup3,
],
)
def test_generic_checksum(store: StorePath, codec_class: type[numcodecs.zarr3._NumcodecsCodec]):
def test_generic_checksum(

Check warning on line 202 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L202 

Added line #L202 was not covered by tests
store: StorePath, codec_class: type[numcodecs.zarr3._NumcodecsBytesBytesCodec]
):
data = np.linspace(0, 10, 256, dtype="float32").reshape((16, 16))

with pytest.warns(UserWarning, match=EXPECTED_WARNING_STR):
a = Array.create(
a = zarr.create_array(

Check warning on line 208 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L208 

Added line #L208 was not covered by tests
store / "generic_checksum",
shape=data.shape,
chunk_shape=(16, 16),
chunks=(16, 16),
dtype=data.dtype,
fill_value=0,
codecs=[BytesCodec(), codec_class()],
compressors=[codec_class()],
)

a[:, :] = data.copy()
a = Array.open(store / "generic_checksum")
a = zarr.open_array(store / "generic_checksum", mode="r")

Check warning on line 218 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L218 

Added line #L218 was not covered by tests
np.testing.assert_array_equal(data, a[:, :])


@pytest.mark.parametrize("codec_class", [numcodecs.zarr3.PCodec, numcodecs.zarr3.ZFPY])
def test_generic_bytes_codec(store: StorePath, codec_class: type[numcodecs.zarr3._NumcodecsCodec]):
def test_generic_bytes_codec(

Check warning on line 223 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L223 

Added line #L223 was not covered by tests
store: StorePath, codec_class: type[numcodecs.zarr3._NumcodecsArrayBytesCodec]
):
try:
codec_class()._codec # noqa: B018
except ValueError as e:
Expand All @@ -231,15 +236,13 @@ def test_generic_bytes_codec(store: StorePath, codec_class: type[numcodecs.zarr3
data = np.arange(0, 256, dtype="float32").reshape((16, 16))

with pytest.warns(UserWarning, match=EXPECTED_WARNING_STR):
a = Array.create(
a = zarr.create_array(

Check warning on line 239 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L239 

Added line #L239 was not covered by tests
store / "generic",
shape=data.shape,
chunk_shape=(16, 16),
chunks=(16, 16),
dtype=data.dtype,
fill_value=0,
codecs=[
codec_class(),
],
serializer=codec_class(),
)

a[:, :] = data.copy()
Expand All @@ -250,18 +253,17 @@ def test_delta_astype(store: StorePath):
data = np.linspace(0, 10, 256, dtype="i8").reshape((16, 16))

with pytest.warns(UserWarning, match=EXPECTED_WARNING_STR):
a = Array.create(
a = zarr.create_array(

Check warning on line 256 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L256 

Added line #L256 was not covered by tests
store / "generic",
shape=data.shape,
chunk_shape=(16, 16),
chunks=(16, 16),
dtype=data.dtype,
fill_value=0,
codecs=[
filters=[
numcodecs.zarr3.Delta(dtype="i8", astype="i2"), # type: ignore[arg-type]
BytesCodec(),
],
)

a[:, :] = data.copy()
a = Array.open(store / "generic")
a = zarr.open_array(store / "generic", mode="r")

Check warning on line 268 in numcodecs/tests/test_zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/tests/test_zarr3.py#L268 

Added line #L268 was not covered by tests
np.testing.assert_array_equal(data, a[:, :])
15 changes: 10 additions & 5 deletions numcodecs/zarr3.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,12 +8,13 @@
>>> import zarr
>>> import numcodecs.zarr3
>>>
>>> codecs = [zarr.codecs.BytesCodec(), numcodecs.zarr3.BZ2(level=5)]
>>> array = zarr.open(
... "data.zarr", mode="w",
... shape=(1024, 1024), chunks=(64, 64),
>>> array = zarr.create_array(
... store="data.zarr",
... shape=(1024, 1024),
... chunks=(64, 64),
... dtype="uint32",
... codecs=codecs)
... filters=[numcodecs.zarr3.Delta()],
... compressors=[numcodecs.zarr3.BZ2(level=5)])
>>> array[:] = np.arange(*array.shape).astype(array.dtype)
.. note::
Expand Down Expand Up @@ -119,6 +120,10 @@ def to_dict(self) -> dict[str, JSON]:
def compute_encoded_size(self, input_byte_length: int, chunk_spec: ArraySpec) -> int:
raise NotImplementedError # pragma: no cover

# Override __repr__ because dynamically constructed classes don't seem to work otherwise
def __repr__(self) -> str:
return f"{self.__class__.__name__}(codec_name={self.codec_name!r}, codec_config={self.codec_config!r})"

Check warning on line 125 in numcodecs/zarr3.py

View check run for this annotation

Codecov / codecov/patch

numcodecs/zarr3.py#L124-L125 

Added lines #L124 - L125 were not covered by tests


class _NumcodecsBytesBytesCodec(_NumcodecsCodec, BytesBytesCodec):
def __init__(self, **codec_config: JSON) -> None:
Expand Down

0 comments on commit 411d6df

Please sign in to comment.