1.29.14

1.29.14 (2025-01-22)

Fix

• fix: Fix zeta alpha mistral (#1736)

• fix zeta alpha mistral

• update use_instructions

• update training datasets

• Update mteb/models/e5_instruct.py

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>

• update float

• Update mteb/models/e5_instruct.py

---

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com> (4985da9)

• fix: Hotfixed public_training_data type annotation (#1857)

Fixed public_training_data flag type to include boolean, as this is how all models are annotated (4bd7328)

Unknown

• Add more annotations (#1833)

• apply additions from #1794

• add annotations for rumodels

• add nomic training data

• fix metadata

• update rest of model meta

• fix bge reranker (12ed9c5)

1.29.13

1.29.13 (2025-01-22)

Fix

• fix: Fixed leaderboard search bar (#1852)

Fixed leaderboard search bar (fe33061)

1.29.12

1.29.12 (2025-01-21)

Fix

• fix: Leaderboard Refinements (#1849)

• Added better descriptions to benchmarks and removed beta tags

• Fixed zero-shot filtering on app loading

• Added zero-shot definition in an accordion

• NaN values are now filled with blank

• Added type hints to filter_models (a8cc887)

1.29.11

1.29.11 (2025-01-21)

Fix

• fix: Add reported annotation and re-added public_training_data (#1846)

• fix: Add additional dataset annotations

• fix: readded public training data

• update voyage annotations (a7a8144)

1.29.10

1.29.10 (2025-01-20)

Fix

• fix: Remove default params, public_training_data and memory usage in ModelMeta (#1794)

• fix: Leaderboard: K instead of M
  Fixes #1752

• format

• fixed existing annotations to refer to task name instead of hf dataset

• added annotation to nvidia

• added voyage

• added uae annotations

• Added stella annotations

• sentence trf models

• added salesforce and e5

• jina

• bge + model2vec

• added llm2vec annotations

• add jasper

• format

• format

• Updated annotations and moved jina models

• make models parameters needed to be filled

• fix tests

• remove comments

• remove model meta from test

• fix model meta from split

• fix: add even more training dataset annotations (#1793)

• fix: update max tokens for OpenAI (#1772)
  update max tokens

• ci: skip AfriSentiLID for now (#1785)

• skip AfriSentiLID for now

• skip relevant test case instead

---

Co-authored-by: Isaac Chung <isaac.chung@team.wrike.com>

• 1.28.7
  Automatically generated by python-semantic-release
• ci: fix model loading test (#1775)
• pass base branch into the make command as an arg
• test a file that has custom wrapper
• what about overview
• just dont check overview
• revert instance check
• explicitly omit overview and init
• remove test change
• try on a lot of models
• revert test model file

---

Co-authored-by: Isaac Chung <isaac.chung@team.wrike.com>

• feat: Update task filtering, fixing bug which included cross-lingual tasks in overly many benchmarks (#1787)
• feat: Update task filtering, fixing bug on MTEB

• Updated task filtering adding exclusive_language_filter and hf_subset
• fix bug in MTEB where cross-lingual splits were included
• added missing language filtering to MTEB(europe, beta) and MTEB(indic, beta)
  The following code outlines the problems:

import mteb
from mteb.benchmarks import MTEB_ENG_CLASSIC
task = [t for t in MTEB_ENG_CLASSIC.tasks if t.metadata.name == "STS22"][0]
# was eq. to:
task = mteb.get_task("STS22", languages=["eng"])
task.hf_subsets
# correct filtering to English datasets:
# ['en', 'de-en', 'es-en', 'pl-en', 'zh-en']
# However it should be:
# ['en']
# with the changes it is:
task = [t for t in MTEB_ENG_CLASSIC.tasks if t.metadata.name == "STS22"][0]
task.hf_subsets
# ['en']
# eq. to
task = mteb.get_task("STS22", hf_subsets=["en"])
# which you can also obtain using the exclusive_language_filter (though not if there was multiple english splits):
task = mteb.get_task("STS22", languages=["eng"], exclusive_language_filter=True)

• format
• remove "en-ext" from AmazonCounterfactualClassification
• fixed mteb(deu)
• fix: simplify in a few areas
• fix: Add gritlm
• 1.29.0
  Automatically generated by python-semantic-release
• fix: Added more annotations!
• fix: Added C-MTEB (#1786)
  Added C-MTEB
• 1.29.1
  Automatically generated by python-semantic-release
• docs: Add contact to MMTEB benchmarks (#1796)
• Add myself to MMTEB benchmarks
• lint
• fix: loading pre 11 (#1798)
• fix loading pre 11
• add similarity
• lint
• run all task types
• 1.29.2
  Automatically generated by python-semantic-release
• fix: allow to load no revision available (#1801)
• fix allow to load no revision available
• lint
• add require_model_meta to leaderboard
• lint
• 1.29.3
  Automatically generated by python-semantic-release

---

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>
Co-authored-by: Isaac Chung <isaac.chung@team.wrike.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Márton Kardos <power.up1163@gmail.com>

• fig merges
• update models info
• change public_training_code to str
• change public_training_code=False to None
• remove annotations
• remove annotations
• remove changed annotations
• remove changed annotations
• remove public_training_data and memory usage
• make framework not optional
• make framework non-optional
• empty frameworks
• add framework
• fix tests
• Update mteb/models/overview.py
  Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>

---

Co-authored-by: Kenneth Enevoldsen <kennethcenevoldsen@gmail.com>
Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>
Co-authored-by: Isaac Chung <isaac.chung@team.wrike.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Márton Kardos <power.up1163@gmail.com> (0a83e38)

• fix: subsets to run (#1830)

• fix split evals

• add test

• lint

• fix moka

• add assert (8be6b2e)

1.29.9

1.29.9 (2025-01-17)

Fix

• fix: Fixed eval split for MultilingualSentiment in C-MTEB (#1804)

• Fixed eval split for MultilingualSentiment in C-MTEB

• FIxed splits for atec, bq and stsb in C-MTEB (96f639b)

1.29.8

1.29.8 (2025-01-17)

Fix

• fix: Added Misc Chinese models (#1819)

• Added moka and piccolo models to overview file

• Added Text2Vec models

• Added various Chinese embedding models

---

Co-authored-by: Isaac Chung <chungisaac1217@gmail.com> (9823529)

• fix: Added way more training dataset annotations (#1765)

• fix: Leaderboard: K instead of M
  Fixes #1752

• format

• fixed existing annotations to refer to task name instead of hf dataset

• added annotation to nvidia

• added voyage

• added uae annotations

• Added stella annotations

• sentence trf models

• added salesforce and e5

• jina

• bge + model2vec

• added llm2vec annotations

• add jasper

• format

• format

• Updated annotations and moved jina models

• fix: add even more training dataset annotations (#1793)

• fix: update max tokens for OpenAI (#1772)

update max tokens

• ci: skip AfriSentiLID for now (#1785)

• skip AfriSentiLID for now

• skip relevant test case instead

---

Co-authored-by: Isaac Chung <isaac.chung@team.wrike.com>

• 1.28.7

Automatically generated by python-semantic-release

• ci: fix model loading test (#1775)

• pass base branch into the make command as an arg

• test a file that has custom wrapper

• what about overview

• just dont check overview

• revert instance check

• explicitly omit overview and init

• remove test change

• try on a lot of models

• revert test model file

---

Co-authored-by: Isaac Chung <isaac.chung@team.wrike.com>

• feat: Update task filtering, fixing bug which included cross-lingual tasks in overly many benchmarks (#1787)

• feat: Update task filtering, fixing bug on MTEB

• Updated task filtering adding exclusive_language_filter and hf_subset
• fix bug in MTEB where cross-lingual splits were included
• added missing language filtering to MTEB(europe, beta) and MTEB(indic, beta)

The following code outlines the problems:

import mteb
from mteb.benchmarks import MTEB_ENG_CLASSIC

task = [t for t in MTEB_ENG_CLASSIC.tasks if t.metadata.name == "STS22"][0]
# was eq. to:
task = mteb.get_task("STS22", languages=["eng"])
task.hf_subsets
# correct filtering to English datasets:
# ['en', 'de-en', 'es-en', 'pl-en', 'zh-en']
# However it should be:
# ['en']

# with the changes it is:
task = [t for t in MTEB_ENG_CLASSIC.tasks if t.metadata.name == "STS22"][0]
task.hf_subsets
# ['en']
# eq. to
task = mteb.get_task("STS22", hf_subsets=["en"])
# which you can also obtain using the exclusive_language_filter (though not if there was multiple english splits):
task = mteb.get_task("STS22", languages=["eng"], exclusive_language_filter=True)

• format

• remove "en-ext" from AmazonCounterfactualClassification

• fixed mteb(deu)

• fix: simplify in a few areas

• fix: Add gritlm

• 1.29.0

Automatically generated by python-semantic-release

• fix: Added more annotations!

• fix: Added C-MTEB (#1786)

Added C-MTEB

• 1.29.1

Automatically generated by python-semantic-release

• docs: Add contact to MMTEB benchmarks (#1796)

• Add myself to MMTEB benchmarks

• lint

• fix: loading pre 11 (#1798)

• fix loading pre 11

• add similarity

• lint

• run all task types

• 1.29.2

Automatically generated by python-semantic-release

• fix: allow to load no revision available (#1801)

• fix allow to load no revision available

• lint

• add require_model_meta to leaderboard

• lint

• 1.29.3

Automatically generated by python-semantic-release

---

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>
Co-authored-by: Isaac Chung <isaac.chung@team.wrike.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Márton Kardos <power.up1163@gmail.com>

---

Co-authored-by: Roman Solomatin <samoed.roman@gmail.com>
Co-authored-by: Isaac Chung <chungisaac1217@gmail.com>
Co-authored-by: Isaac Chung <isaac.chung@team.wrike.com>
Co-authored-by: github-actions <github-actions@github.com>
Co-authored-by: Márton Kardos <power.up1163@gmail.com> (3b2d074)

• fix: bm25s (#1827)

Co-authored-by: sam021313 <40773225+sam021313@users.noreply.github.com> (96420a2)

• fix: Added Chinese Stella models (#1824)

Added Chinese Stella models (74b495c)

1.29.7

1.29.7 (2025-01-16)

Ci

• ci: only return 1 model_name per file (#1818)

• only return 1 model_name per file

• fix args parse

• revert test change (d7a7791)

Fix

• fix: add bge-m3 ModelMeta (#1821)

add bge (4ac59bc)

Unknown

• Add model inf-retriever-v1 (#1744)

• feat(models): add infly/inf-retriever-v1 model metadata- Add inf_models.py file with metadata for infly/inf-retriever-v1 model

• Update overview.py to include inf_models in model imports

• Reformat code

• Update inf-retriever-v1 ModelMeta

• Fill more information for inf-retriever-v1

• Add license information for inf-retriever-v1

---

Co-authored-by: Samuel Yang <samuelyang150@gmail.com> (60c4980)

1.29.6

1.29.6 (2025-01-15)

Fix

• fix: Added more Chinese models' ModelMeta (#1814)

• Added Multilingual USE models

• Added Moka models

• Added dmeta models

• Added jina-zh

• Added piccolo models (748955c)

1.29.5

1.29.5 (2025-01-15)

Fix

• fix: Add additional contacts (#1817)

add contacts from #1790 (c4ee9fe)

Unknown

• Update points table (e3a3df8)