Enable loading custom models in the QuPath extension

[constantinpape]

Hi @ksugar,
we would like to implement loading of custom SAM models (e.g. from micro_sam) in the QuPath plugin.
This should be easy on the server side, and I think my PR in ksugar/samapi#11 already adds everything that's necessary there.

But I am not so sure what needs to be changed here. From a quick check I think I would need to add the option for a custom URL here:

qupath-extension-sam/src/main/java/org/elephant/sam/SAMCommand.java

Lines 101 to 102 in 609289d

	private final ObjectProperty<SAMModel> model = PathPrefs.createPersistentPreference(
	"ext.SAM.model", SAMModel.VIT_L, SAMModel.class);

Is that correct, or is there some other relevant place in the code? Also, how should the default (no custom URL) be represented? In python it's just None.

Thanks!

[petebankhead]

That sounds great! It looks like the use of an enum for SAMModel would need to change. The way we did it in the WSInfer extension is to use a String or alternative non-enum class instead, and populate the combobox by querying the server for the list of available models on the server.

• Current SAM approach here
• Alternative WSInfer approach here

Then hopefully adding more models becomes easy, and doesn't require any change to the extension.

[ksugar]

Hi @constantinpape, thank you for the PR!
I checked your PR and found that it is a good idea to handle all models (incl. currently available models) with a single interface with just the checkpoint_url argument.

On the client side (i.e., QuPath), in addition to the preset model options (vit_h, vit_l, vit_b and vit_m), the user will have the option to add a custom model by specifying the URL with a nickname. Once it is added, it will remain until the user explicitly deletes it. QuPath's PathPrefs will manage a list of custom urls.

I will work on it later today.

[petebankhead]

QuPath's PathPrefs will manage a list of custom urls.

I'm not sure PathPrefs will be very good at that by default. The trouble is that Java's Preferences class doesn't support arrays/lists. It should still work if you don't mind squeezing everything into a single string, although this won't support extremely long lists (the maximum string length is 8192 characters).

My guess is that the client supporting a list of models returned by the server would be easier to develop and manage, so that any extra URLs are handled when starting up the server.

[ksugar]

Hi @petebankhead, thank you for the feedback.

I was expecting that PathPrefs.createPersistentJsonPreference() can handle arrays/lists of String but it seems that it is not the case.

Following your suggestion, I will implement an endpoint for getting a list of the available models on the server side.

[petebankhead]

I forgot about PathPrefs.createPersistentJsonPreference() :)
Probably it should work (although I guess maybe doesn't).

[constantinpape]

Thanks for the quick replies @ksugar and @petebankhead! From what I understand you will take care of this for now @ksugar :). Let me know if you need any more info, or once it's ready to be tested!

[ksugar]

Hi @constantinpape and @petebankhead, I have implemented the function for loading custom model weights from URL.

Please try qupath-extension-sam-0.4.0-SNAPSHOT.jar with the updated version of samapi from the dev branch.

python -m pip install -U git+https://github.com/ksugar/samapi.git@dev

Since this version, it is important to launch the server with --workers 2 (or more) to enable cancellation of a download of a weights file.

export PYTORCH_ENABLE_MPS_FALLBACK=1 # Required for running on Apple silicon
uvicorn samapi.main:app --workers 2

The relevant documentation is found here.
https://github.com/ksugar/qupath-extension-sam/tree/dev#register-sam-weights-from-url

[constantinpape]

Thanks @ksugar ! I will try it out as soon as possible and let you know what I find.

[constantinpape]

I tested it now @ksugar and it all works as expected!

[ksugar]

Thank you @constantinpape for testing it! I will release it soon.
I'm thinking that it would be helpful to have a catalog for weights in README so that the users can find them easily.
How do you think?

Type	Name (customizable)	URL	Citation
vit_h	vit_h_lm	https://zenodo.org/record/8250299/files/vit_h_lm.pth?download=1	Archit, A. et al. Segment Anything for Microscopy. bioRxiv 2023. doi:10.1101/2023.08.21.554208 https://github.com/computational-cell-analytics/micro-sam
vit_b	vit_b_lm	https://zenodo.org/record/8250281/files/vit_b_lm.pth?download=1
vit_h	vit_h_em	https://zenodo.org/record/8250291/files/vit_h_em.pth?download=1
vit_b	vit_b_em	https://zenodo.org/record/8250260/files/vit_b_em.pth?download=1

[constantinpape]

Thank you @constantinpape for testing it! I will release it soon. I'm thinking that it would be helpful to have a catalog for weights in README so that the users can find them easily. How do you think?
Type Name (customizable) URL Citation
vit_h vit_h_lm https://zenodo.org/record/8250299/files/vit_h_lm.pth?download=1 Archit, A. et al. Segment Anything for Microscopy. bioRxiv 2023. doi:10.1101/2023.08.21.554208

https://github.com/computational-cell-analytics/micro-sam
vit_b vit_b_lm https://zenodo.org/record/8250281/files/vit_b_lm.pth?download=1
vit_h vit_h_em https://zenodo.org/record/8250291/files/vit_h_em.pth?download=1
vit_b vit_b_em https://zenodo.org/record/8250260/files/vit_b_em.pth?download=1

Yes, that would be great!