Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] ZARR upload not working #109

Closed
jo-mueller opened this issue Jul 2, 2024 · 15 comments
Closed

[BUG] ZARR upload not working #109

jo-mueller opened this issue Jul 2, 2024 · 15 comments

Comments

@jo-mueller
Copy link
Contributor

jo-mueller commented Jul 2, 2024 •
edited
Loading

Describe the bug

I was trying to upload a .zarr image to omero with the most recent version and encountered an error message. Full traceback:

Using OMERO.java-5.6.11-ice36
Joined session for Johannes@172.26.122.141:4064. Idle timeout: 10 min. Current group: BiaPoL
ERROR:root:Import of [c:\...\notebooks\medium_image.zarr](file:///C:/.../notebooks/medium_image.zarr) has failed!
ERROR:root:File [c:\...\notebooks\medium_image.zarr](file:///C:/.../notebooks/medium_image.zarr) has not been imported
ERROR:root:No image ids to organize
WARNING:root:Missing annotation or namespace, skipping annotations
id_image None

To Reproduce

I essentially used this snippet to check whether the upload works. Full code:

import numpy as np
import zarr
import ezomero

from ome_zarr.io import parse_url
from ome_zarr.writer import write_image

# from https://forum.image.sc/t/images-batch-import-into-a-dataset-in-omero-using-ezomero-2-1-0/89529/12

def test_import(conn):

    # generate numpy data...
    # shape = (1, 4, 15, 9247, 8332)
    shape = (1, 2, 3, 1500, 3000)
    mean_val=10
    rng = np.random.default_rng(0)
    data = rng.poisson(mean_val, size=shape).astype(np.uint8)
    print(data.shape)

    # # write to ome-zarr...
    filename = "medium_image.zarr"
    chunks = (1, 1, 1, 1024, 1024)
    store = parse_url(filename, mode="w").store
    root = zarr.group(store=store)
    write_image(image=data, group=root, axes="tczyx", storage_options=dict(chunks=chunks))

    # import...
    id_image = ezomero.ezimport(conn=conn, target=filename)
    print('id_image', id_image)


if __name__ == '__main__':
    conn = ezomero.connect(host='my_omero_ip', port=4064, user='Johannes', group='my_group', secure=True)
    test_import(conn)

Expected behavior
The uploaded image should upload to the omero server with a new id.

Desktop (please complete the following information):

  • OS: Windows 10
  • Python version 3.9.18
  • ezomero version: 3.0.1

Further context

It seems related to this issue on image.sc but then again, the problem should have been fixed with the recent release 🤔

cc @zoccoler @cwetzker
@erickmartins
Copy link
Collaborator

I'm trying to reproduce this locally, but having trouble with the code snippet there (it fails at writing the zarr itself). I'd still expect that something like ezomero.ezimport(conn=conn, target=filename, depth=10) is necessary to get zarr imports working properly - have you tried that?

@jo-mueller
Copy link
Contributor Author

Hi @erickmartins ,

sorry for the late reply - I only got around to testing just now. This is an updated snippet that hopefully reproduces the error. Requires pip install bioio bioio-ome-zarr.

from bioio.writers.ome_zarr_writer import OmeZarrWriter

writer = OmeZarrWriter('test.zarr')
writer.write_image(
    np.random.rand(1, 1, 64, 128, 128).astype(np.float32),
    chunk_dims=(1, 1, 64, 128, 128),
    scale_num_levels=1,
    dimension_order='TCZYX',
    image_name='test',
    channel_names=['test'],
    channel_colors=None,
    physical_pixel_sizes = None,
)

ezomero.ezimport(conn=conn, target='test.zarr', dataset=255, depth=10)

That's the traceback:

Using OMERO.java-5.6.10-ice36
Joined session for johamuel@omero-int.biotec.tu-dresden.de:4064. Idle timeout: 10 min. Current group: Dye Lab
2024-08-01 18:03:06,231 146        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.2
2024-08-01 18:03:06,240 155        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 7.1.0 revision: 05c7b2413cfad19a73b619c61ddf77ca2d038ce7 date: 11 December 2023
2024-08-01 18:03:06,270 185        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2024-08-01 18:03:06,452 367        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2024-08-01 18:03:06,562 477        [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /home/pol_haase/johamuel/Documents/Git/single-cell-analysis-of-organoids/notebooks/00_data_management/test.zarr/0/0/0/0/0/0
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Double (java.lang.Integer and java.lang.Double are in module java.base of loader 'bootstrap')
	at loci.formats.in.ZarrReader.parseOmeroMetadata(ZarrReader.java:704)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:170)
	at loci.formats.FormatReader.setId(FormatReader.java:1480)
	at loci.formats.ImageReader.setId(ImageReader.java:865)
	at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
	at loci.formats.ChannelFiller.setId(ChannelFiller.java:258)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
	at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:317)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
	at loci.formats.Memoizer.setId(Memoizer.java:698)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
	at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
	at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:387)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
...
2024-08-01 18:03:06,957 872        [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 5.15.0-25-generic
No imports due to errors!
ERROR:root:No image ids to organize
WARNING:root:Missing annotation or namespace, skipping annotations

@joshmoore
Copy link

Can you share the test.zarr, @jo-mueller? (Or at least, the metadata files in the fileset)

cc: @dgault

@jo-mueller
Copy link
Contributor Author

@joshmoore coming right up 👍

https://datashare.tu-dresden.de/s/2tA2rqLb7XjYBL5

@joshmoore
Copy link

Thanks! It certainly validates:

image

The line of code doesn't match the current reader though: https://github.com/ome/ZarrReader/blob/main/src/loci/formats/in/ZarrReader.java

Do you know what version you are using?

@jo-mueller
Copy link
Contributor Author

Ezomero version is 3.0.1 if that's what you mean?

@joshmoore
Copy link

Ezomero version is 3.0.1 if that's what you mean?

Sorry, no. Of the ZarrReader.

@erickmartins
Copy link
Collaborator

erickmartins commented Aug 1, 2024 •
edited
Loading

yeah, this looks like an error at the omero import level, and not specific to ezomero - in fact, it might be worth seeing if a "naked" omero import works, and if it doesn't, this is probably an issue worth opening against the relevant OME repo (ZarrReader, I assume?).

@jo-mueller
Copy link
Contributor Author

@joshmoore these are the versions I used, which I think are the latest ones

bioio 1.0.3
bioio-ome-zarr = 1.0.1
ome-zarr 0.9.0
omero-py 5.13.1

@erickmartins
Copy link
Collaborator

erickmartins commented Aug 2, 2024 •
edited
Loading

can you confirm if this works with omero import?

@jo-mueller
Copy link
Contributor Author

jo-mueller commented Aug 2, 2024 •
edited
Loading

CLI import command

omero import --name "Test" --depth 10 ./test.zarr/

and full traceback

Using OMERO.java-5.6.10-ice36
Using session for johamuel@omero-int.biotec.tu-dresden.de:4064. Idle timeout: 10 min. Current group: default
2024-08-02 15:33:43,029 145        [      main] INFO          ome.formats.importer.ImportConfig - OMERO.blitz Version: 5.7.2
2024-08-02 15:33:43,038 154        [      main] INFO          ome.formats.importer.ImportConfig - Bioformats version: 7.1.0 revision: 05c7b2413cfad19a73b619c61ddf77ca2d038ce7 date: 11 December 2023
2024-08-02 15:33:43,069 185        [      main] INFO   formats.importer.cli.CommandLineImporter - Log levels -- Bio-Formats: ERROR OMERO.importer: INFO
2024-08-02 15:33:43,247 363        [      main] INFO      ome.formats.importer.ImportCandidates - Depth: 10 Metadata Level: MINIMUM
2024-08-02 15:33:43,356 472        [      main] ERROR     ome.formats.importer.cli.ErrorHandler - FILE_EXCEPTION: /home/pol_haase/johamuel/Documents/Git/single-cell-analysis-of-organoids/notebooks/00_data_management/./test.zarr/0/0/0/0/0/0
java.lang.ClassCastException: class java.lang.Integer cannot be cast to class java.lang.Double (java.lang.Integer and java.lang.Double are in module java.base of loader 'bootstrap')
	at loci.formats.in.ZarrReader.parseOmeroMetadata(ZarrReader.java:704)
	at loci.formats.in.ZarrReader.initFile(ZarrReader.java:170)
	at loci.formats.FormatReader.setId(FormatReader.java:1480)
	at loci.formats.ImageReader.setId(ImageReader.java:865)
	at ome.formats.importer.OMEROWrapper$4.setId(OMEROWrapper.java:167)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
	at loci.formats.ChannelFiller.setId(ChannelFiller.java:258)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
	at loci.formats.ChannelSeparator.setId(ChannelSeparator.java:317)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
	at loci.formats.Memoizer.setId(Memoizer.java:698)
	at loci.formats.ReaderWrapper.setId(ReaderWrapper.java:692)
	at ome.formats.importer.ImportCandidates.singleFile(ImportCandidates.java:427)
	at ome.formats.importer.ImportCandidates.handleFile(ImportCandidates.java:576)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:387)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:384)
	at org.apache.commons.io.DirectoryWalker.walk(DirectoryWalker.java:355)
	at ome.formats.importer.ImportCandidates.execute(ImportCandidates.java:381)
	at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:222)
	at ome.formats.importer.ImportCandidates.<init>(ImportCandidates.java:174)
	at ome.formats.importer.cli.CommandLineImporter.<init>(CommandLineImporter.java:148)
	at ome.formats.importer.cli.CommandLineImporter.main(CommandLineImporter.java:1021)
2024-08-02 15:33:43,358 474        [      main] INFO      ome.formats.importer.ImportCandidates - 4 file(s) parsed into 0 group(s) with 1 call(s) to setId in 109ms. (112ms total) [0 unknowns]
2024-08-02 15:33:43,391 507        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Attempting initial SSL connection to omero-int.biotec.tu-dresden.de:4064
2024-08-02 15:33:43,593 709        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Insecure connection requested, falling back
2024-08-02 15:33:43,758 874        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Pinging session every 300s.
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Server: 5.6.3
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Client: 5.7.2
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - Java Version: 11.0.24
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Name: Linux
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Arch: amd64
2024-08-02 15:33:43,765 881        [      main] INFO       ome.formats.OMEROMetadataStoreClient - OS Version: 5.15.0-25-generic

so yeah...issue may have to be moved upstream.

@erickmartins
Copy link
Collaborator

Yeah, it looks like that - @joshmoore I assume to ZarrReader?

@joshmoore
Copy link

Yes, though I'd suggest trying to upgrade first:

omero import --fetch-jars 5.6.12
Downloading https://downloads.openmicroscopy.org/omero/5.6.12/OMERO.java.zip

If that changes things, then you'll also need a server upgrade to have the ZarrReader on both sides. Otherwise, --> github.com/ome/ZarrReader

@jo-mueller
Copy link
Contributor Author

Same error, so I guess that's where I'll turn next. Thanks for the help on this end!

@jo-mueller jo-mueller mentioned this issue Aug 2, 2024
Open
@jo-mueller
Copy link
Contributor Author

---> Follow-up here

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@joshmoore @erickmartins @jo-mueller