Where to download pre-trained model weights?

[SaharIravani]

Hello,

Thanks for the great work.
I am trying to use Orthrus and follow the setup instructions. However, I cannot find any information about where to download the pre-trained model weights.

• Are pre-trained models available for download or will they be?
• If so, could you provide a link or instructions on how to obtain them?
• If not, is training from scratch the only option?

Any guidance would be greatly appreciated.

Thank you!

[phil-fradkin]

Hello thanks for your interest in our work! The weights are available on huggingface

The pre-trained Orthrus models are available at https://huggingface.co/antichronology/orthrus (you will find it in files and versions), with individual models available for inference at: https://huggingface.co/quietflamingo/orthrus-base-4-track and https://huggingface.co/quietflamingo/orthrus-large-6-track

These should be easier to perform inference on and would look something like:

from transformer import AutoModel

model = AutoModel.from_pretrained("quietflamingo/orthrus-large-6-track")

Please let us know if you have any other issues!

Also all the embeddings are pre-computed and are available for download from the zenood repository
https://zenodo.org/records/14708163

[wconnell]

Hi there, is it possible to share a minimal working example of successfully loading the HF model and performing inference? Preferably in a notebook format with required installations?

I am attempting to work through this on a google colab GPU, but the specific dependencies are causing issues (ie mamba-ssm).

[SaharIravani]

@phil-fradkin, Thank you very much for providing this information. I'll give a try and get back to you as soon as I successfully run it.

[IanShi1996]

Hi @wconnell, here's a link to a Colab that should work out-of-the-box: https://colab.research.google.com/drive/1Rb6VC92YoKRPyF2LG4m8zIXjDszm1NZW?usp=sharing

We've also updated the HF model cards to show a (hopefully working) example of Orthrus embedding:
https://huggingface.co/quietflamingo/orthrus-large-6-track

We'll double check the dependencies specified in the requirements file. Since mamba-ssm is under active dev, the newer versions don't always support the PyTorch and CUDA versions we used to pre-train the model. I believe the following set of core dependencies should work:

PyTorch 2.2.2
CUDA 12.1
causal_conv1d==1.2.0.post2
mamba-ssm==1.2.0.post1

Let me know if you run into any further issues!