Generating multiple readers with Bio-Formats can be very slow #865
Comments
petebankhead
added a commit
to petebankhead/qupath
that referenced
this issue
Dec 17, 2021
Aims to fix qupath#865 This takes a different approach to parallelization, managing a pool of ImageReaders with each tile-requesting thread taking the next available reader. If there are no readers available, and the total number is less than some maximum value (based upon the number if available processors), a new reader is generated on another thread and added to the queue when ready. This should * avoid generating more readers than needed, with a limit separate from the number of tile requesting threads * avoid attempting to initialize multiple readers simultaneously, which can be a bottleneck In addition, more tests have been added.
|
Upon further investigation, memoization can greatly reduce the severity of the problem - which is probably why it hasn't generated more complaints. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug report
Describe the bug
BioFormatsImageServerlazily creates a new reader for each thread as required. This is ok if the readers are fast to initialize, but can be a major bottleneck if they are not.This was noticed when working with large CZI images (>30 GB). Initializing a reader took ~3-5s. However, because the method was synchronized and 32 threads were wanting to get pixels for the viewer, many other threads were blocked. This meant that even viewing the image properly was not possible for well over a minute.
Once the readers were created, performance was fine.
To Reproduce
Unfortunately, I'm not aware of any public images images that can be used to test this. It might be evident with any large Axioscan images (I'm not entirely sure).
Once the image is open, zoom in and wait for tiles to appear. If experiencing the problem, this will take an unreasonable amount of time. VisualVM indicates that the bottleneck is initializing readers.
Expected behavior
No major delay in requesting tiles once the image has been opened.
Desktop (please complete the following information):
Screenshots
Compare the 'total time' (>100s) with the actual time spent using the CPU (3.6s) for a thread requesting image tiles.
Additional context
A few things could help:
isThisType()when creating a reader (when the class of the reader can be known)The text was updated successfully, but these errors were encountered: