Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

GPU memory not freed when using OpenCV DNN #841

Closed
petebankhead opened this issue Nov 9, 2021 · 3 comments
Closed

GPU memory not freed when using OpenCV DNN #841

petebankhead opened this issue Nov 9, 2021 · 3 comments
Assignees
@petebankhead
Labels
bug
Milestone
v0.4.0

Comments

@petebankhead
Copy link
Member

Bug report

Describe the bug
GPU memory isn't released when using a DnnModel repeatedly. This can be seen when using StarDist.

To Reproduce
See qupath/qupath-extension-stardist#11
Copied here because the fix may need to be implemented within the core, rather than the StarDist extension.

Expected behavior
GPU memory released after processing is complete.

Desktop (please complete the following information):

  • OS: Likely any with CUDA (Windows, Linux)
  • QuPath v0.3.0

Additional context
I guess a blob may need to be reset... or something. Although that may be tricky to achieve if allowing multithreading and trying not to unduly harm performance (or perhaps it won't be).
@petebankhead petebankhead self-assigned this Nov 9, 2021
@petebankhead petebankhead added this to the v0.3.1 milestone Nov 10, 2021
@petebankhead
Copy link
Member Author

Calling Net.close() is reported to fix this... although it's still not entirely obvious when or how this should be done.

One option could be to use a Cleaner.

If this works, it shouldn't require any API change and so could be included in v0.3.1.

@petebankhead
Copy link
Member Author

Looking into this some more, it is more awkward than I had realised.

I had thought that wrapping everything in a PointerScope would help, by ensuring that the Net was closed properly without further effort - however the use if Net.retainReference() thwarts this.

Calling System.gc() before/after using a DNN can help patch over the problem, however a better solution is likely to require a more thorough revision of OpenCVDnn - possibly by updating DnnModel to implement AutoCloseable.

This makes it more of a v0.4.0 issue than a v0.3.1 issue.

@petebankhead petebankhead modified the milestones: v0.3.1, v0.4.0 Nov 15, 2021
petebankhead added a commit to petebankhead/qupath-extension-stardist that referenced this issue Nov 15, 2021
@petebankhead
Adding PointScope
3bc1ae0 
Changes were intended to fix qupath#11

However, this was only partially successful for the reasons described at qupath/qupath#841

The changes may cause complications if used with an existing `DnnModel` that lazily creates any subclass of `Pointer`, so a better solution is needed.
petebankhead added a commit to petebankhead/qupath that referenced this issue Dec 22, 2021
@petebankhead
Implement AutoCloseable with DnnModel
f7a16fa 
Aims to provide a mechanism to address
* qupath#841
* qupath/qupath-extension-stardist#11
by ensuring that an OpenCV Net wrapped in a DnnModel can be closed without needing a reference to the Net itself.
The StarDist and TensorFlow extensions should be updated to make use of this.
@petebankhead petebankhead mentioned this issue Dec 22, 2021
Merged
@petebankhead
Copy link
Member Author

It's now possible to call DnnModel.close(). It will also be possible to call StarDist2D.close() in the next release of the extension.

@MarkZaidi MarkZaidi mentioned this issue Dec 7, 2022
Closed
4 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@petebankhead petebankhead

Labels
bug
Projects
None yet
Milestone
v0.4.0
Development

No branches or pull requests
1 participant
@petebankhead