Script for finding candidate models for deprecation #29686

amyeroberts · 2024-03-15T20:42:22Z

What does this PR do?

Adds a script to find a list of candidate models to deprecate. By default, the list of models are those which are over a year old and have had fewer than 5,000 downloads over the last month.

Things to note:

As the script iterates over all of the checkpoints on the hub, it can be fairly slow to run (a few mins). The option to save the retrieved information --save_model_info is added, which will save all of the models, alongside their date of first commit and number of downloads. This is useful when changing the threshold date of number of downloads to select models as you can run --use_cache to iterate quickly
There are assumptions made about the model file structures and names. Therefore some "models" will be found for deprecation, even if they many downloads e.g. "data2vec" as there's many separate data2vec modeling files e.g. modeling_data2vec_vision.py

HuggingFaceDocBuilderDev · 2024-03-15T21:02:54Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

amyeroberts · 2024-04-23T14:35:50Z

Gentle ping @ydshieh

ydshieh · 2024-04-23T17:36:45Z

Sorry, didn't notice this PR. Will review asap.

ydshieh

Thank you for working on this @amyeroberts 👍

A first round of (quick) review comments 🤗

utils/models_to_deprecate.py

amyeroberts · 2024-04-24T10:55:26Z

Thanks for the review @ydshieh! I've addressed all your comments

ydshieh · 2024-04-24T13:14:49Z

utils/models_to_deprecate.py

+            model = model_path.split("/")[-2]
+            if model in models_info:
+                continue
+            commits = repo.git.log("--diff-filter=A", "--", model_path).split("\n")


each call takes 0.5 seconds, and we have ~270 models in the repository. So it takes 2 mins to get this info.

Since this is static (other than the new added models), we could probably build this info, save to somewhere and load it.

But probably 2 mins is nothing compared to the Hub connections. So it's merely a possible improvements which don't need to be in this PR.

ydshieh · 2024-04-24T13:16:40Z

utils/models_to_deprecate.py

+        for i, hub_model in enumerate(api.list_models()):
+            if i % 100 == 0:
+                print(f"Processing model {i}")
+            if max_num_models != -1 and i > max_num_models:
+                break
+            if hub_model.private:
+                continue
+            for tag in hub_model.tags:
+                tag = tag.lower().replace("-", "_")
+                if tag in models_info:
+                    models_info[tag]["downloads"] += hub_model.downloads


Do you know how long this will take. The PR description says the whole script runs in a few minutes, but the above commit info extraction already takes ~2 minutes I believe.

ydshieh · 2024-04-24T13:22:34Z

There are assumptions made about the model file structures and names. Therefore some "models" will be found for deprecation, even if they many downloads e.g. "data2vec" as there's many separate data2vec modeling files e.g. modeling_data2vec_vision.py

If IIRC, the dictionary model_info use the model directory (so it is data2vec) but the tags of models on the Hub could be data2vec-vision etc, so it won't be added to the dictionary.

ydshieh

Great! Curious about the time required to run this script, but PR is good!

ydshieh · 2024-04-24T13:28:15Z

utils/models_to_deprecate.py

+            for tag in hub_model.tags:
+                tag = tag.lower().replace("-", "_")
+                if tag in models_info:
+                    models_info[tag]["downloads"] += hub_model.downloads


@Wauplin Is hub_model.downloads the number of downloads over the last month?

Good point 👀

Yes it is 👍

Over the last 30 days to be more precise.

amyeroberts · 2024-04-24T15:24:26Z

@ydshieh I updated a little bit, so that we filter the hub requests based on model tags in da62f7e. Running this way, it ran in 161 seconds. Previously it was 240s. Not ideal - not sure of a faster way to get the downloads info - but as we're running sporadically I don't think too bad either

ydshieh · 2024-04-24T15:37:20Z

OK, I was wondering why you pushed the last commit but now I understand it is to make things a bit fast. Nice!

ydshieh · 2024-04-24T15:39:16Z

My only left question is just about if the download number is over the last month. I don't seem to find this is mentioned in the HfApi. But since it is mentioned in your PR, let's wait @Wauplin to confirm. (or if it's urgent, you can merge and we check with him later)

ydshieh · 2024-04-24T15:39:43Z

Ah, sorry.

ydshieh · 2024-04-24T15:42:28Z

Would be nice to have something like

# Copyright 2024 The HuggingFace Team. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

amyeroberts · 2024-04-24T15:59:47Z

let's wait @Wauplin to confirm. (or if it's urgent, you can merge and we check with him later)

Yep, let's wait 👍 It's not urgent - just a few other PRs lined up after this one to handle automated deprecation and one which will deprecate a bunch of models that are flagged from this script

Wauplin · 2024-04-25T08:50:26Z

Didn't want to make you wait longer to I replied 😄 And it's all good 👍

ydshieh · 2024-04-25T08:54:16Z

OK, thanks for confirmation. BTW, for class ModelInfo:, my local file shows

        downloads (`int`):
            Number of downloads of the dataset.

Might be better to make it more precise (if not done yet on the latest version)

Wauplin · 2024-04-25T09:08:25Z

Opened huggingface/huggingface_hub#2250 to fix that 🤗

amyeroberts changed the title ~~Add utility for finding candidate models for deprecation~~ Script for finding candidate models for deprecation Apr 11, 2024

amyeroberts requested a review from ydshieh April 11, 2024 13:26

ydshieh self-assigned this Apr 23, 2024

ydshieh reviewed Apr 24, 2024

View reviewed changes

utils/models_to_deprecate.py Outdated Show resolved Hide resolved

amyeroberts added 6 commits April 24, 2024 11:47

Add utility for finding candidate models for deprecation

96c965c

Better model filtering

15e85b9

Update

5106edc

Add warning tip

17f92ae

Fix up

550f49f

Review comments

98183b9

amyeroberts force-pushed the find-models-to-deprecate branch from 7964b2f to 98183b9 Compare April 24, 2024 10:51

ydshieh reviewed Apr 24, 2024

View reviewed changes

ydshieh approved these changes Apr 24, 2024

View reviewed changes

ydshieh reviewed Apr 24, 2024

View reviewed changes

Filter requests based on tags

da62f7e

Add copyright header

b21b9f6

Wauplin mentioned this pull request Apr 25, 2024

Better document download attribute huggingface/huggingface_hub#2250

Merged

amyeroberts merged commit 30ee508 into huggingface:main Apr 25, 2024
8 checks passed

amyeroberts deleted the find-models-to-deprecate branch April 25, 2024 09:10

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Script for finding candidate models for deprecation #29686

Script for finding candidate models for deprecation #29686

amyeroberts commented Mar 15, 2024 •

edited

Loading

HuggingFaceDocBuilderDev commented Mar 15, 2024

amyeroberts commented Apr 23, 2024

ydshieh commented Apr 23, 2024

ydshieh left a comment

amyeroberts commented Apr 24, 2024

ydshieh Apr 24, 2024

ydshieh Apr 24, 2024

ydshieh commented Apr 24, 2024

ydshieh left a comment

ydshieh Apr 24, 2024

amyeroberts Apr 24, 2024

Wauplin Apr 25, 2024

Wauplin Apr 25, 2024

amyeroberts commented Apr 24, 2024

ydshieh commented Apr 24, 2024

ydshieh commented Apr 24, 2024

ydshieh commented Apr 24, 2024

ydshieh commented Apr 24, 2024

amyeroberts commented Apr 24, 2024

Wauplin commented Apr 25, 2024

ydshieh commented Apr 25, 2024

Wauplin commented Apr 25, 2024

Script for finding candidate models for deprecation #29686

Script for finding candidate models for deprecation #29686

Conversation

amyeroberts commented Mar 15, 2024 • edited Loading

What does this PR do?

HuggingFaceDocBuilderDev commented Mar 15, 2024

amyeroberts commented Apr 23, 2024

ydshieh commented Apr 23, 2024

ydshieh left a comment

Choose a reason for hiding this comment

amyeroberts commented Apr 24, 2024

ydshieh Apr 24, 2024

Choose a reason for hiding this comment

ydshieh Apr 24, 2024

Choose a reason for hiding this comment

ydshieh commented Apr 24, 2024

ydshieh left a comment

Choose a reason for hiding this comment

ydshieh Apr 24, 2024

Choose a reason for hiding this comment

amyeroberts Apr 24, 2024

Choose a reason for hiding this comment

Wauplin Apr 25, 2024

Choose a reason for hiding this comment

Wauplin Apr 25, 2024

Choose a reason for hiding this comment

amyeroberts commented Apr 24, 2024

ydshieh commented Apr 24, 2024

ydshieh commented Apr 24, 2024

ydshieh commented Apr 24, 2024

ydshieh commented Apr 24, 2024

amyeroberts commented Apr 24, 2024

Wauplin commented Apr 25, 2024

ydshieh commented Apr 25, 2024

Wauplin commented Apr 25, 2024

amyeroberts commented Mar 15, 2024 •

edited

Loading