Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Full support for dask arrays #42

Closed
imagejan opened this issue Sep 28, 2023 · 2 comments
Closed

Full support for dask arrays #42

imagejan opened this issue Sep 28, 2023 · 2 comments

Comments

@imagejan
Copy link
Member

We should consider supporting DaskArray throughout the pipeline, in faim_hcs.io.* and in faim_hcs.Zarr.

For example, if the assemble_fn parameter for the get_well_image_C[Z]YX functions can be a Callable accepting dask arrays and returning a single (stacked/blocked) dask array, the function should not create a np.zeros() array first and fill it with the tile pixel data, but instead keep a list of arrays and use da.stack() with appropriate chunking options to create the CYX or CZYX output.

This would allow the pixel reading operation to be deferred until the writing step in Zarr.write_image_and_metadata.

@imagejan
Copy link
Member Author

Related:

  • metadata handling could (should?) be separated from the pixel data, as in some formats (e.g. CellVoyager) the metadata are already completely separate (.mlf etc.) from the pixel data (e.g. .tif without spatial calibration).
  • a remaining question is how to deal with the histograms (UIntHistogram): should we still read all pixel values to create them immediately when processing/stacking the image, or rather delay creation until we actually write the well/field into the zarr store?

@tibuch
Copy link
Contributor

tibuch commented Dec 13, 2023

Solved with #58 and #59.

@tibuch tibuch closed this as completed Dec 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants