feat: use final redirect URL for chunk download #39
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
pretty sure it's always already been the case in |
|
@McPatate for reference, can you link to internal discussion of this subject? (e.g. a Slack link) |
|
This conversation for example: https://huggingface.slack.com/archives/C02EMARJ65P/p1717775944145709. We discussed this mostly offline w/ @XciD |
XciD
approved these changes
Jul 22, 2024
9092db2 to
b7e5ea7
Compare
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Only call the final redirect URL to avoid overloading the Hub with requests and also altering the download count
cc @osanseviero & @Pierrci regarding download count.
@Wauplin we should check that we do the same in huggingface_hub for download but also when using the multipart upload, we want to make sure
parts_urls: Vec<String>only contains s3 urls and nothugginface.co(though this won't affect download count, more a thing about reducing load on hf.co).Also included in this PR: