fix: remove `*` imports #1569

Samoed · 2024-12-08T17:57:21Z

Checklist

Run tests locally to make sure nothing is broken using make test.
Run the formatter to format the code using make lint.

Ref #1463
Also merged changes from main and found 3 datasets that previously never imported:

Ddisco
GeorgianSentimentClassification
WongnaiReviewsClassification

* add more stat * add more stat * update statistics

Automatically generated by python-semantic-release

Bugfixes with data parsing in main figure

* Fixed task result loading from disk * Fixed task result loading from disk

Automatically generated by python-semantic-release

* fix * lint

Automatically generated by python-semantic-release

* fix: Removed column wrapping on the table, so that it remains readable * Added disclaimer to figure * fix: Added links to task info table, switched out license with metric

* small fix * fix: fix

Automatically generated by python-semantic-release

swap touche2020 for parity

Automatically generated by python-semantic-release

* add sum per lang * add sort by sum option * make lint

Automatically generated by python-semantic-release

* feat: add CUREv1 dataset --------- Co-authored-by: nadshe <nadia.sheikh@clinia.com> Co-authored-by: olivierr42 <olivier.rousseau@clinia.com> Co-authored-by: Daniel Buades Marcos <daniel@buad.es> * feat: add missing domains to medical tasks * feat: modify benchmark tasks * chore: benchmark naming --------- Co-authored-by: nadshe <nadia.sheikh@clinia.com> Co-authored-by: olivierr42 <olivier.rousseau@clinia.com>

Automatically generated by python-semantic-release

* check if model attr of model exists * lint * Fix retrieval evaluator

Automatically generated by python-semantic-release

* Made get_scores error tolerant * Added join_revisions, made get_scores failsafe * Fetching metadata fixed fr HF models * Added failsafe metadata fetching to leaderboard code * Added revision joining to leaderboard app * fix * Only show models that have metadata, when filter_models is called * Ran linting

Automatically generated by python-semantic-release

Filtering for models that have metadata

Automatically generated by python-semantic-release

* align readme with current mteb * align with mieb branch * fix test

Automatically generated by python-semantic-release

* add lang family mapping and map to task table * make lint * add back some unclassified lang codes

Automatically generated by python-semantic-release

* Correction of SICK-R metadata * Correction of SICK-R metadata --------- Co-authored-by: rposwiata <rposwiata@opi.org.pl>

…05` and `text-multilingual-embedding-002` (#1562) * fix: google_models batching and prompt * feat: add text-embedding-005 and text-multilingual-embedding-002 * chore: `make lint` errors * fix: address PR comments

Automatically generated by python-semantic-release

fix: bm25s implementation

# Conflicts: # docs/create_tasks_table.py # docs/tasks.md # mteb/abstasks/AbsTaskClassification.py # mteb/abstasks/AbsTaskClusteringFast.py # mteb/abstasks/AbsTaskInstructionRetrieval.py # mteb/abstasks/AbsTaskMultilabelClassification.py # mteb/abstasks/AbsTaskPairClassification.py # mteb/abstasks/AbsTaskReranking.py # mteb/abstasks/AbsTaskRetrieval.py # mteb/abstasks/AbsTaskSTS.py # mteb/descriptive_stats/InstructionRetrieval/Core17InstructionRetrieval.json # mteb/descriptive_stats/MultilabelClassification/MultiEURLEXMultilabelClassification.json # mteb/descriptive_stats/Reranking/AskUbuntuDupQuestions.json # mteb/descriptive_stats/Reranking/ESCIReranking.json # mteb/descriptive_stats/Reranking/WikipediaRerankingMultilingual.json # mteb/descriptive_stats/Retrieval/AppsRetrieval.json # mteb/descriptive_stats/Retrieval/BelebeleRetrieval.json # mteb/descriptive_stats/Retrieval/COIRCodeSearchNetRetrieval.json # mteb/descriptive_stats/Retrieval/CodeEditSearchRetrieval.json # mteb/descriptive_stats/Retrieval/CodeFeedbackMT.json # mteb/descriptive_stats/Retrieval/CodeFeedbackST.json # mteb/descriptive_stats/Retrieval/CodeSearchNetCCRetrieval.json # mteb/descriptive_stats/Retrieval/CodeSearchNetRetrieval.json # mteb/descriptive_stats/Retrieval/CodeTransOceanContest.json # mteb/descriptive_stats/Retrieval/CodeTransOceanDL.json # mteb/descriptive_stats/Retrieval/CosQA.json # mteb/descriptive_stats/Retrieval/JaqketRetrieval.json # mteb/descriptive_stats/Retrieval/NFCorpus.json # mteb/descriptive_stats/Retrieval/StackOverflowQA.json # mteb/descriptive_stats/Retrieval/SyntheticText2SQL.json # mteb/descriptive_stats/Retrieval/Touche2020.json # mteb/descriptive_stats/Retrieval/Touche2020Retrieval.v3.json # mteb/descriptive_stats/Retrieval/mFollowIRCrossLingualInstructionRetrieval.json # mteb/descriptive_stats/Retrieval/mFollowIRInstructionRetrieval.json # mteb/evaluation/MTEB.py # mteb/evaluation/evaluators/RetrievalEvaluator.py # mteb/leaderboard/table.py # mteb/model_meta.py # mteb/models/arctic_models.py # mteb/models/e5_models.py # mteb/models/nomic_models.py # mteb/models/sentence_transformers_models.py # mteb/tasks/PairClassification/multilingual/XStance.py # mteb/tasks/Reranking/zho/CMTEBReranking.py # mteb/tasks/STS/por/SickBrSTS.py # tests/test_benchmark/mock_tasks.py

Automatically generated by python-semantic-release

* fix: bm25s implementation * correct library name --------- Co-authored-by: Daniel Buades Marcos <daniel.buades@clinia.com>

* fix: Add training dataset to model meta Adresses #1556 * Added docs * format

… for visualization (#1564) * feat: batch requests to cohere models * fix: use correct task_type * feat: use tqdm with openai * fix: explicitely set `show_progress_bar` to False

… `answer` (#1565)

Automatically generated by python-semantic-release

# Conflicts: # mteb/model_meta.py

isaac-chung · 2024-12-08T19:03:05Z

Nice! Was wondering actually, if this could be merged to main instead? I don't think there were any major compatible changes? (as in, everything should run the same as before)

KennethEnevoldsen

Looks great. I'm very happy about this!

It was a big frustration for me in #1567 so I am very happy to see it. Did you do it all manually?

Nice! Was wondering actually, if this could be merged to main instead? I don't think there were any major compatible changes? (as in, everything should run the same as before)

Before you could e.g. do

from mteb import load_datasets

I believe this will no longer be possible

mteb/__init__.py

isaac-chung · 2024-12-09T08:15:20Z

I see a script being used to generate these. To see changes from this PR, I was switching the commits one at a time.

Might be nice if the merge from main was separated.

Samoed · 2024-12-09T08:20:08Z

Yes, as @isaac-chung mentioned, I wrote a script to generate imports for tasks, but for other directories, I did it manually. For future PRs I won't merge main and v2 with feature at the same time

isaac-chung

Thanks for tackling this :)
Might be worth running the same benchmarks in #1463 again as a comparison.

Samoed · 2024-12-09T18:11:59Z

Here is profile results of profiling, but it is hard to tell difference

Samoed and others added 30 commits November 14, 2024 11:52

fix: Count unique texts, data leaks in calculate metrics (#1438)

dd5d226

* add more stat * add more stat * update statistics

fix: update task metadata to allow for null (#1448)

04ac3f2

Update tasks table

f6a49fe

1.19.5

78c0e4e

Automatically generated by python-semantic-release

Fix: Made data parsing in the leaderboard figure more robust (#1450)

4e86cea

Bugfixes with data parsing in main figure

Fixed task loading (#1451)

039d010

* Fixed task result loading from disk * Fixed task result loading from disk

fix: publish (#1452)

feb1ab7

1.19.6

3397633

Automatically generated by python-semantic-release

fix: Fix load external results with None mteb_version (#1453)

14d7523

* fix * lint

1.19.7

68eb498

Automatically generated by python-semantic-release

WIP: Polishing up leaderboard UI (#1461)

58c459b

* fix: Removed column wrapping on the table, so that it remains readable * Added disclaimer to figure * fix: Added links to task info table, switched out license with metric

fix: loading pre 1.11.0 (#1460)

1b920ac

* small fix * fix: fix

1.19.8

a988fef

Automatically generated by python-semantic-release

fix: swap touche2020 to maintain compatibility (#1469)

9b2aece

swap touche2020 for parity

1.19.9

8bb4a29

Automatically generated by python-semantic-release

docs: Add sum per language for task counts (#1468)

2fb6fe7

* add sum per lang * add sort by sum option * make lint

fix: pinned datasets to <3.0.0 (#1470)

fde124a

1.19.10

7186e04

Automatically generated by python-semantic-release

Update tasks table

4408717

1.20.0

3ff38ec

Automatically generated by python-semantic-release

fix: check if model attr of model exists (#1499)

917ad7f

* check if model attr of model exists * lint * Fix retrieval evaluator

1.20.1

cde720e

Automatically generated by python-semantic-release

1.20.2

594f643

Automatically generated by python-semantic-release

fix: leaderboard only shows models that have ModelMeta (#1508)

35245d3

Filtering for models that have metadata

1.20.3

9282796

Automatically generated by python-semantic-release

fix: align readme with current mteb (#1493)

942f212

* align readme with current mteb * align with mieb branch * fix test

1.20.4

09f004c

Automatically generated by python-semantic-release

docs: Add lang family mapping and map to task table (#1486)

cfd43ac

* add lang family mapping and map to task table * make lint * add back some unclassified lang codes

github-actions and others added 17 commits December 6, 2024 11:18

1.21.8

a6ce6f9

Automatically generated by python-semantic-release

docs: Correction of SICK-R metadata (#1558)

fc64791

* Correction of SICK-R metadata * Correction of SICK-R metadata --------- Co-authored-by: rposwiata <rposwiata@opi.org.pl>

feat(google_models): fix issues and add support for `text-embedding-0…

611b6a1

…05` and `text-multilingual-embedding-002` (#1562) * fix: google_models batching and prompt * feat: add text-embedding-005 and text-multilingual-embedding-002 * chore: `make lint` errors * fix: address PR comments

1.22.0

5e7e033

Automatically generated by python-semantic-release

fix(bm25s): search implementation (#1566)

ac44e58

fix: bm25s implementation

1.22.1

b8ff89c

Automatically generated by python-semantic-release

docs: Fix dependency library name for bm25s (#1568)

03347eb

* fix: bm25s implementation * correct library name --------- Co-authored-by: Daniel Buades Marcos <daniel.buades@clinia.com>

fix: Add training dataset to model meta (#1561)

6489fca

* fix: Add training dataset to model meta Adresses #1556 * Added docs * format

feat: (cohere_models) cohere_task_type issue, batch requests and tqdm…

1d21818

… for visualization (#1564) * feat: batch requests to cohere models * fix: use correct task_type * feat: use tqdm with openai * fix: explicitely set `show_progress_bar` to False

fix(publichealth-qa): ignore rows with None values in question or…

68bd8ac

… `answer` (#1565)

1.23.0

2550a27

Automatically generated by python-semantic-release

fix wongnai

d474451

update inits

2015ee5

fix tests

23fb642

lint

54a7f5c

Merge branch 'refs/heads/main' into update_imports

07f1391

# Conflicts: # mteb/model_meta.py

Samoed requested review from KennethEnevoldsen and isaac-chung December 8, 2024 17:57

KennethEnevoldsen reviewed Dec 9, 2024

View reviewed changes

mteb/__init__.py Outdated Show resolved Hide resolved

Samoed added 3 commits December 9, 2024 11:29

update imports

d67225b

fix tests

8653c27

lint

4ba6ff5

isaac-chung approved these changes Dec 9, 2024

View reviewed changes

isaac-chung merged commit d0aa3a7 into v2.0.0 Dec 9, 2024
10 checks passed

isaac-chung deleted the update_imports branch December 9, 2024 20:28

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

fix: remove `*` imports #1569

fix: remove `*` imports #1569

Samoed commented Dec 8, 2024

isaac-chung commented Dec 8, 2024

KennethEnevoldsen left a comment

isaac-chung commented Dec 9, 2024

Samoed commented Dec 9, 2024 •

edited

Loading

isaac-chung left a comment

Samoed commented Dec 9, 2024 •

edited

Loading

fix: remove * imports #1569

fix: remove * imports #1569

Conversation

Samoed commented Dec 8, 2024

Checklist

isaac-chung commented Dec 8, 2024

KennethEnevoldsen left a comment

Choose a reason for hiding this comment

isaac-chung commented Dec 9, 2024

Samoed commented Dec 9, 2024 • edited Loading

isaac-chung left a comment

Choose a reason for hiding this comment

Samoed commented Dec 9, 2024 • edited Loading

fix: remove `*` imports #1569

fix: remove `*` imports #1569

Samoed commented Dec 9, 2024 •

edited

Loading

Samoed commented Dec 9, 2024 •

edited

Loading