What is the best way to convert a zarr file to parquet?

[vlulla]

Given this code snippet, from zarr's tutorial,

import zarr
z1 = zarr.open('tst/example.zarr', mode='w', shape=(10_000, 10_000), chunks=(1_000, 1_000), dtype='i4')

how can I save z1 to parquet file format? Specifically, how can I export this as a table with 10_000 rows and 10_000 columns, i.e., without doing any transpose? Are there any memory requirements that I ought to be aware of?

[vlulla]

I think a little more context might help.

In our attempts to accomplish this task we have tried a few ideas and have run into issues which have stumped us. For instance, we have run into issues with jupyter notebook crashing when we try to use the to_dask_dataframe function. The details, and discussion surrounding this issue can be found on #6811 .

My colleague, @lewfish, has discovered another issue (https://dask.discourse.group/t/workers-don't-have-promised-key-error-and-delayed-computation/936 ) which I think is pertinent to this discussion as it is not clear whether the issue is with xarray or dask.

[max-sixty]

If it's really only 800MB, there's no need for dask — z1.to_pandas().to_parquet("x.parquet") should work

[vlulla]

Thanks @max-sixty ! We are trying to write a 80GB zarr subset for our current experiments. We are trying to learn this to eventually convert a 7 TB zarr into parquet.

azavea/noaa-hydro-data#89 (comment) lists the workaround I currently have for converting the 80GB zarr to parquet but that is infeasible for the complete dataset.

[max-sixty]

OK. I think the main question is then how to reshape the data into something that dask can manage — the parquet part is then trivial.

I just saw the discussion in #6811. I'm not an expert here but to the extent the data is already chunked in a way that doesn't require shuffle between workers, I'd be surprised if we're seeing crashes.