Enhancing SFT Training Efficiency Using Packing and FlashAttention2 with Position IDs #31629

RhuiDih · 2024-06-26T06:29:32Z

What does this PR do?

Improve throughput as well as train time and memory utilization for instruction tuning by enabling padding-free and attention mask-free attention.

Specifically, this PR adds the capability to utilize position_ids in FlashAttention2 _flash_attention_forward() in the case of packing (attention_mask=None) to models which use position_ids in their respective DecoderLayer implementations.

This PR also adds a new off-the-shelf data collator DataCollatorWithFlattening which packs the examples in a mini batch into one long sequence and return position_ids as well and turns the first token of labels to -100 to prevent the last token of previous example predicting the first token of next example.

This enables the following:

Use of packing for instruction tuning without incorrect cross-example attention
Significant increase in training throughput and reduction in memory utilization

Example Result 1
dataset: OrcaMath subset
setup: FSDP with 8 GPUs

Model	DataProcess	Time	Throughput (token/s)	Memory (MB)
Llama2-7B	Padding	790	1269	22305
Llama2-7B	ThisPR	574	1746	20950
Mistral-7B	Padding	812	1216	23603
Mistral-7B	ThisPR	596	1658	22409

Example Result 2
dataset: FLAN subset
setup: FSDP with 8 GPUs

Model	DataProcess	Time	Throughput (token/s)	Memory (MB)
Llama2-7B	Padding	1526	771	29234
Llama2-7B	ThisPR	809	1455	23854
Mistral-7B	Padding	742	742	30625
Mistral-7B	ThisPR	1408	1408	24549

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
Did you read the contributor guideline,
Pull Request section?
Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.
Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.
Did you write any new necessary tests?

Who can review?

Models:

text models: @ArthurZucker

Edit - add more description on data collator

ArthurZucker

Thanks! I think we can wait a tad bit, #31446 is almost ready for merge!

ArthurZucker · 2024-07-10T08:21:41Z

src/transformers/models/dbrx/modeling_dbrx.py

-            attn_output = flash_attn_func(
-                query_states, key_states, value_states, dropout, softmax_scale=softmax_scale, causal=causal
-            )
+            if (position_ids[:,-1]==position_ids.size(1)-1).all():


this is an input dependent control flow which won't be supported by compile, but compile is already not supported.
This needs a comment: we are checking that the input is not padded, and that we are doing prefill right?

ArthurZucker · 2024-07-10T08:22:40Z

src/transformers/models/gemma/modeling_gemma.py

@@ -49,6 +49,7 @@
 )
 from .configuration_gemma import GemmaConfig

+from ..llama.modeling_llama import prepare_fa2_from_position_ids


we never import from another modeling code in transformers, we use "copied from" or we define the general function in falsh_attention_utils for example

ArthurZucker · 2024-07-10T08:23:04Z

src/transformers/models/llama/modeling_llama.py

@@ -59,6 +59,20 @@
 _CONFIG_FOR_DOC = "LlamaConfig"


+def prepare_fa2_from_position_ids(query, key, value, position_ids, query_length):


this needs to be documented to explain what is happening

RhuiDih · 2024-07-12T03:39:07Z

@ArthurZucker #31446 has just merged, let me rebase and update this PR

RhuiDih · 2024-07-12T08:35:13Z

@ArthurZucker Rebase is done.

I have move the codes to modeling_flash_attention_utils.py and added required comments/description.

Regarding the control flow, ideally the cu_seq_len should be prepared in data collator and model should accept it as arg to trigger flash_attn_varlen_func without relying on position_ids, but this is the quick & fast way to do padding free.

ArthurZucker

This LGTM! Only thing missing is a test!
Testing this codepath in particuler, + datacollator in the trainer tests maybe?

ArthurZucker · 2024-07-15T14:54:57Z

src/transformers/data/data_collator.py

+        warnings.warn(
+            "Using `DataCollatorForBatchFlattening` will flatten the entire mini batch into single long sequence."
+            "Make sure your attention computation is able to handles it!"
+        )


I think we keep track of a list of models that support FA2, if needed for guidance here!

ArthurZucker · 2024-07-15T14:57:35Z

src/transformers/modeling_flash_attention_utils.py

+
+    # if position_ids is provided and check not all examples (row) contain only 1 sequence,
+    # then use `flash_attn_varlen_func` to prevent cross-example attention and also allow padding free approach
+    elif position_ids is not None and not (position_ids[:,-1]==position_ids.size(1)-1).all():


this is an input dependent control flow, but we already don't support compile with FA2 AFAIK.
Alright for now!

mayank31398 · 2024-07-16T14:50:57Z

I think loss computation is wrong for everything
it needs to be:
https://github.com/ibm-granite/dolomite-engine/blob/7e2527528ac54cb3388040a3052ef3b6d845a40d/dolomite_engine/hf_models/models/gpt_dolomite/main.py#L185-L191

RhuiDih · 2024-07-17T01:43:40Z

@mayank31398
it can be done within the model or when labels preparation in data collator
https://github.com/huggingface/transformers/blob/808fd638182289cf1903a1d4c536f00c48809fb8/src/transformers/data/data_collator.py#L1634-L1637

wynterl · 2024-07-17T03:20:48Z

Hello @mayank31398 We are providing the capability to define the labels as the user wishes in the data collator in this PR via transformers/src/transformers/data/data_collator.py

RhuiDih · 2024-07-17T03:56:30Z

@ArthurZucker
I have added tests for the new data collator and the models which already have FA2 tests in place. For the trainer, i can't find existing FA2 + Trainer testing combo, could you point me there ?

ArthurZucker

LGTM, I just want a second look from our @fxmarty .
This should improve performances for everyone. but let's make sure we don't break BC! WDYT @fxmarty ?

ArthurZucker · 2024-07-19T09:04:20Z

No worries I think the tests you added are enough

fxmarty

Looks good!

I strongly think this should be documented somewhere, maybe https://huggingface.co/docs/transformers/main/en/perf_train_gpu_one#flash-attention-2.

fxmarty · 2024-07-19T09:54:49Z

tests/models/falcon/test_modeling_falcon.py

+    @require_flash_attn
+    @require_torch_gpu
+    @slow
+    def test_flash_attention_2_padding_matches_padding_free_with_position_ids(self):
+        super().test_flash_attention_2_padding_matches_padding_free_with_position_ids()


As this is already in test_modeling_common.py, adding this here is not needed.

fxmarty · 2024-07-19T09:54:59Z

tests/models/gemma/test_modeling_gemma.py

+    @require_flash_attn
+    @require_torch_gpu
+    @slow
+    def test_flash_attention_2_padding_matches_padding_free_with_position_ids(self):
+        super().test_flash_attention_2_padding_matches_padding_free_with_position_ids()
+


same for all

i have removed them in latest commit

fxmarty · 2024-07-19T09:58:53Z

tests/test_modeling_common.py

+                # flatten
+                padfree_inputs_dict = {
+                    k:v[dummy_attention_mask.bool()].unsqueeze(0) for k,v in inputs_dict.items() if not k=="attention_mask"
+                }


I am not a big fan as having unpadded inputs as [1, total_seqlens]. To me, it would make more sense to have support for one dimensional tensors as input_ids ([total_tokens]), which becomes after embedding inputs_embeds ([total_tokens, hidden_size]), as anyway flash attention varlen frontend expects 3D tensors, not 4D.

Thus

query = query.view(-1, query.size(-2), query.size(-1)) key = key.view(-1, key.size(-2), key.size(-1)) value = value.view(-1, value.size(-2), value.size(-1))

would not be needed, and neither calculating the cumulative seqlens at each layer. This maybe can be left for an other PR.

very much agree it is less than ideal that cu_seq_len is calculated again in every layers. It can be done in data_collator or in the first forward() and passed along all DecoderLayer, not sure how it links to switching from [bs, seq_len, ... ] to just [seq_len, ...]. This removal of bs dimension requires substantial change to the model implementation, should HF decide to do this switch, those 3 lines can be just removed.

fxmarty · 2024-07-19T10:00:03Z

src/transformers/data/data_collator.py

+        warnings.warn(
+            "Using `DataCollatorWithFlattening` will flatten the entire mini batch into single long sequence."
+            "Make sure your attention computation is able to handle it!"
+        )


How should a user make sure?

no easy way, the user will have to make sure the model supports FA2 and have position_ids
until the model accepting cu_seq_len which will allow every model remove padding once for all, i think this serve as a good warning

HuggingFaceDocBuilderDev · 2024-07-23T09:40:49Z

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

ruff format tests src utils

ArthurZucker

🚀 nice getting this done!

olisicky · 2024-08-25T16:07:51Z

Hi, thank you for this! I tried to use it without DataCollatorWithFlattening but with data prepared with input_ids and position_ids, but I encounter RuntimeError: CUDA error: an illegal memory access was encountered just after elif position_ids is not None and not (torch.diff(position_ids, dim=-1) >= 0).all() and query_length != 1: was called.

I am using:

llama3
flash_attn version: 2.5.8
deepspeed - ZeRO1
transformers 4.44.1

Thank you!

RhuiDih · 2024-08-26T06:55:15Z

@olisicky Hi, could you provide a minimal code to reproduce the error ? that would help greatly

olisicky · 2024-08-27T08:13:38Z

@olisicky Hi, could you provide a minimal code to reproduce the error ? that would help greatly

Hi @RhuiDih . I found a mistake in my data as I was preparing a minimal code for you:D. So thank you for encouraging me to do so! As I am preparing data without DataCollatorWithFlattening I kept padding there to context_length even for bs=1 and it resulted in plenty of 0 in position_ids tensor on indexes corresponding to PAD tokens.

However, I have an additional question. You are setting -100 in labels for the first token in each sequence in the batch. I wonder why. I thought that the use of position_ids itself should reduce cross-example attention but here you mentioned that the -100 should prevent the last token of previous example predicting the first token of next example.

If I would have a sequences which start with BOS token and then it will be ignored during training by setting -100 in labels, then I would lost the importance of this first token. Is it necessary? Or maybe shouldn't we use some dummy token there rather then a part of the sequence itself?

Thank you very much!

RhuiDih · 2024-08-27T09:19:46Z

@olisicky
The setting of -100 is due to causal loss implementation, if we were flattening all sequences into batch of 1 without setting -100 accordingly, it would incur unwanted loss.

First token of input_ids is still used to produce logits, hence no importance is lost. What we want to ignore, its the first token of labels due to the mentioned causal loss implementation.
Hope it is clear.

olisicky · 2024-09-02T06:44:23Z

@olisicky The setting of -100 is due to causal loss implementation, if we were flattening all sequences into batch of 1 without setting -100 accordingly, it would incur unwanted loss.

First token of input_ids is still used to produce logits, hence no importance is lost. What we want to ignore, its the first token of labels due to the mentioned causal loss implementation. Hope it is clear.

Sorry for late reply. Yes, thank you. I will try it.

MahmoudAshraf97 · 2024-09-06T09:00:05Z

Is packing usable in transformers generate method? AFAIK only padding is supported

ArthurZucker · 2024-09-06T12:19:10Z

Packing is supported with the collator introduced. We don't pack when generating tho, PRs are welcome for this! 🤗

qibao77 · 2024-10-15T09:07:21Z

What does this PR do?

Improve throughput as well as train time and memory utilization for instruction tuning by enabling padding-free and attention mask-free attention.

Specifically, this PR adds the capability to utilize position_ids in FlashAttention2 _flash_attention_forward() in the case of packing (attention_mask=None) to models which use position_ids in their respective DecoderLayer implementations.

This PR also adds a new off-the-shelf data collator DataCollatorWithFlattening which packs the examples in a mini batch into one long sequence and return position_ids as well and turns the first token of labels to -100 to prevent the last token of previous example predicting the first token of next example.

This enables the following:

Use of packing for instruction tuning without incorrect cross-example attention

Significant increase in training throughput and reduction in memory utilization

Example Result 1 dataset: OrcaMath subset setup: FSDP with 8 GPUs

Model DataProcess Time Throughput (token/s) Memory (MB)
Llama2-7B Padding 790 1269 22305
Llama2-7B ThisPR 574 1746 20950
Mistral-7B Padding 812 1216 23603
Mistral-7B ThisPR 596 1658 22409
Example Result 2 dataset: FLAN subset setup: FSDP with 8 GPUs

Model DataProcess Time Throughput (token/s) Memory (MB)
Llama2-7B Padding 1526 771 29234
Llama2-7B ThisPR 809 1455 23854
Mistral-7B Padding 742 742 30625
Mistral-7B ThisPR 1408 1408 24549

Before submitting

This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).

Did you read the contributor guideline,
Pull Request section?

Was this discussed/approved via a Github issue or the forum? Please add a link
to it if that's the case.

Did you make sure to update the documentation with your changes? Here are the
documentation guidelines, and
here are tips on formatting docstrings.

Did you write any new necessary tests?

Who can review?

Models:

text models: @ArthurZucker

Edit - add more description on data collator

How to use this feature for pretraining?

RhuiDih · 2024-10-16T07:53:51Z

@qibao7 usually one would truncate and pack to max length during pretraining, this PR will not benefit pretraining, unless most of your pretraining data are short

* Added mamba.py backend (#30139) * Update README.md * tests: forward ok * backward test done * done testing * removed check. scripts * Update README.md * added use_mambapy arg * fixed typo in warning * protected imports w/ mambapy package * delete pscan.py + raise rather than assert * Update import_utils.py * fix whitespaces and unused import * trailing whitespace + import block unformatted * Update modeling_mamba.py * transpose before pscan * shape comment * ran make style * use_mambapy=False by default Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * ran make fix-copies --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Rename Phi-3 rope scaling type (#31436) * renamed phi3 rope_scaling type * fixed trailing whitespaces * fixed test * added warning * fixed format * Revert "Incorrect Whisper long-form decoding timestamps " (#32148) Revert "Incorrect Whisper long-form decoding timestamps (#32003)" This reverts commit cd48553fc8375e1a28d4d82cfe231dedf6a23af8. * Fix typing to be compatible with later py versions (#32155) * feat(cache): StaticCache uses index_copy_ to avoid useless copy (#31857) * feat(cache): StaticCache uses index_copy_ to avoid useless copy Using index_copy_ allows for explicit in-place change of the tensor. Some backends (XLA) will otherwise copy the tensor, making the code slower and using more memory. Proposed implementation will end up using less memory and on XLA will result in less compilation, but the change is also quite generic, making no change whatsoever on CUDA or CPU backend. * feat(cache): SlidingWindowCache uses index_copy_ to avoid useless copy Applying the same change done in StaticCache. * fix(cache): fallback of index_copy_ when not implemented * fix(cache): in index_copy_ ensure tensors are on same device * [run slow] llama * fix(cache): add move of cache_position to same device in SlidingWindowCache * Revert "[run slow] llama" This reverts commit 02608dd14253ccd464e31c108e0cd94364f0e8b9. * Added additional kwarg for successful running of optuna hyperparameter search (#31924) Update integration_utils.py Added additional kwarg * Enhancing SFT Training Efficiency Using Packing and FlashAttention2 with Position IDs (#31629) * add DataCollatorBatchFlattening * Update data_collator.py * change name * new FA2 flow if position_ids is provided * add comments * minor fix * minor fix data collator * add test cases for models * add test case for data collator * remove extra code * formating for ruff check and check_repo.py * ruff format ruff format tests src utils * custom_init_isort.py * Updated `ruff` to the latest version (#31926) * Updated ruff version and fixed the required code accorindg to the latest version. * Updated ruff version and fixed the required code accorindg to the latest version. * Added noqa directive to ignore 1 error shown by ruff * Dev version: v4.44.0.dev0 * Llama 3.1 conversion Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com> * fix (#32162) * fix: Fixed an if condition that is always evaluating to true (#32160) Fixed an if condition always evaluating to true. * [docs] change temperature to a positive value (#32077) fix * adds: extra_repr() to MambaRMSNorm to include hidden size / size of weights in the layer (#32171) * adds: extra_repr() to MambaRMSNorm to include the hidden size of the layer * style fix with ruff: * fix: default value reflects the runtime environment variables rather than the ones present at import time. (#32153) * fix: default value reflects the runtime environment variables rather than the ones present at import time. * Fix: Change `deterministic` to None by default; use env var if None * Update qwen2.md (#32108) * Update qwen2.md outdated description * Update qwen2.md amended * Update qwen2.md Update * Update qwen2.md fix wrong version code, now good to go * Remove conversational pipeline tests (#32099) Remove conversation pipeline tests * RoPE: relaxed rope validation (#32182) * relaxed rope check * lets also accept rope_type=None, defaulting to the original implementation * type and rope_type can coexist * let's not warn when someone is running a forward (#32176) * let's not warn when someone is running a foward without cache + self.training * more models * fixup * Fix resize embedding with Deepspeed (#32192) fix resize when deepspeed * Fix float8_e4m3fn in modeling_utils (#32193) * Fix float8_e4m3fn in modeling_utils * style * fix * comment * Support dequantizing GGUF FP16 format (#31783) * support gguf fp16 * support gguf bf16 with pytorch * add gguf f16 test * remove bf16 * :rotating_light: No more default chat templates (#31733) * No more default chat templates * Add the template to the GPT-SW3 tests since it's not available by default now * Fix GPT2 test * Fix Bloom test * Fix Bloom test * Remove default templates again * fix: Replaced deprecated `unittest method` with the correct one (#32198) Replaced deprecated unittest method with the correct one. * [whisper] fix short-form output type (#32178) * [whisper] fix short-form output type * add test * make style * update long-form tests * fixes * last fix * finalise test * remove unnecessary guard code related with pytorch versions 1.4.2 ~ 1.7.0 (#32210) remove unnecessary guard code related with pytorch versions 1.4.2 ~ 1.7.0 * Update question_answering.py (#32208) * [BigBird Pegasus] set _supports_param_buffer_assignment to False (#32222) set _supports_param_buffer_assignment to False * [warnings] fix E721 warnings (#32223) fix E721 warnings * Follow up for #31973 (#32025) * fix * [test_all] trigger full CI --------- Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> * translate philosophy.md to chinese (#32177) * translate philosophy.md to chinese * add the missing link * Allow a specific microphone to be used by the ffmpeg audio pipeline utility functions. Default to using the currently active microphone on Mac (#31846) * use currently active microphone on mac for ffmpeg_microphone * Allow ffmpeg_microphone device to be specified Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Fix code snippet for Grounding DINO (#32229) Fix code snippet for grounding-dino * Generation: stop at `eos` for assisted decoding (#31301) * fix * move changes to prompt lookup * add test * set eos in assistant model * style * fix flakiness * changes for new `main` * Update tests/generation/test_utils.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update tests/generation/test_utils.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * add comment to explain --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Llava: generate without images (#32183) * llava w/o images * tests * Resize embeds with DeepSpeed (#32214) * fix resize when deepspeed * deepsped uses new embeds * we needed this * don't log base model architecture in wandb if log model is false (#32143) * don't log base model architecture in wandb is log model is false * Update src/transformers/integrations/integration_utils.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * convert log model setting into an enum * fix formatting --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Refactor: Removed un-necessary `object` base class (#32230) * Refactored to remove un-necessary object base class. * small fix. * Adds: extra_repr for RMSNorm layers in most models (#32204) * adds: extra_repr() to RMSNorm layers in multiple models * adds: extra_repr for deprecated models as well * formatting as per style guide * Add check for `target_sizes is None` in `post_process_image_guided_detection` for owlv2 (#31934) * Add check for target_sizes is None in post_process_image_guided_detection * Make sure Owlvit and Owlv2 in sync * Fix incorrect indentation; add check for correct size of target_sizes * [tests] fix `static` cache implementation is not compatible with `attn_implementation==flash_attention_2` (#32039) * add flash attention check * fix * fix * Flash-Attn: fix generation when no attention mask or no pading (#32241) * fix * fix prev test (half of failures) * [run-slow] llama, gemma2 * [run-slow] llama, gemma2 * More flexible trigger condition (#32251) update Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> * Llama 3.1: replace for loop by tensor ops at inv_freq initialization (#32244) * replace for loop by tensor ops * rm assert; readability * 🚨 Bloom support for cache class (#31445) * bloom dynamic cache * bloom follows standard cache format * no skips for bloom anymore * use cache position when possible * clean up * codestyle * Update src/transformers/models/bloom/modeling_bloom.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/bloom/modeling_bloom.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/bloom/modeling_bloom.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * pr comments * isinstance fix * address comments * make musicgen test happy * [run-slow] bloom --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Upload new model failure report to Hub (#32264) upload Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> * Optimize t5 tokenize logic to avoid redundant calls (#32270) * Optimize t5 tokenize logic to avoid redundant calls * fix and overwrite copies * fix: Fixed wrong argument passed to `convert_blip_checkpoint` function call (#32262) Removed one wrong argument passed to convert_blip_checkpoint function call. * Repo: remove exceptions in `check_docstrings` (#32259) remove exceptions * make `p_mask` a numpy array before passing to `select_starts_ends` (#32076) * fix * bug fix * refine * fix * fix(docs): Fixed a link in docs (#32274) Fixed a link in docs. * Generate: end-to-end compilation (#30788) * mvp * added test (a few models need fixes) * fix a few test cases * test nits * harder test 😈 * revert changes in stablelm * test with improved condition * add todo * tmp commit * merged with main * nits * add todo * final corrections * add docs for generation compilation * docs nits * add tip * PR suggestions * add more details to the compilation docs * fix cache positions * cache is now init in generate; update docs * tag test as flaky * docs * post rebase make fixup and other nits * remove unintended changes * whisper (encoder-decoder) not supported * move token default updates to ; add tests for token defaults * push changes * manual rebase * chameleon doesn't support this * fix test_static_cache_mha_mqa_gqa (broken in another PR) * docs: dynamic is better with end-to-end compilation * Whisper tokenizer word level timestamps (#32197) * fix _fix_key in PreTrainedModel * fix _find_longest_common_sequence * add test * remove result.json * nit * update test * [pipeline] fix padding for 1-d tensors (#31776) * [pipeline] fix padding for 1-d tensors * add test * make style * Update tests/pipelines/test_pipelines_automatic_speech_recognition.py Co-authored-by: Kamil Akesbi <45195979+kamilakesbi@users.noreply.github.com> * Update tests/pipelines/test_pipelines_automatic_speech_recognition.py --------- Co-authored-by: Kamil Akesbi <45195979+kamilakesbi@users.noreply.github.com> * Make static cache compatible with torch.export (#32168) * Add stream messages from agent run for gradio chatbot (#32142) * Add stream_to_gradio method for running agent in gradio demo * use torch 2.4 in 2 CI jobs (#32302) Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> * Docs: fix GaLore optimizer code example (#32249) Docs: fix GaLore optimizer example Fix incorrect usage of GaLore optimizer in Transformers trainer code example. The GaLore optimizer uses low-rank gradient updates to reduce memory usage. GaLore is quite popular and is implemented by the authors in [https://github.com/jiaweizzhao/GaLore](https://github.com/jiaweizzhao/GaLore). A few months ago GaLore was added to the HuggingFace Transformers library in https://github.com/huggingface/transformers/pull/29588. Documentation of the Trainer module includes a few code examples of how to use GaLore. However, the `optim_targe_modules` argument to the `TrainingArguments` function is incorrect, as discussed in https://github.com/huggingface/transformers/pull/29588#issuecomment-2006289512. This pull request fixes this issue. * Fix GGUF dequantize for `gguf==0.9.1` (#32298) * fix gguf dequantize for gguf==0.9.1 * fix old version * make style * Cast epochs_trained to int when resuming training (#32286) * fix epochs_trained as int when resuming training * refactor --------- Co-authored-by: teddyferdinan <teddy.ferdinan@pwr.edu.pl> * feat(ci): set `fetch-depth: 0` in trufflehog checkout step (#31663) * Fix M4T for ASR pipeline (#32296) * tentative fix * do the same for M4T * Docs: formatting nits (#32247) * doc formatting nits * ignore non-autodocs * Apply suggestions from code review Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/esm/modeling_esm.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/esm/modeling_esm.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * make fixup --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Alternative agent plan (#32295) * new agent plan * plan type assertion * style corrections * better prompt naming * make fixup * fix: Added missing raise keyword for few exceptions (#32333) Fixed raising of few exceptions. * fixes to properly shard FSDP across cpu and meta for cpu_efficient_loading for prequantized 4bit (#32276) * fixes #32329 : The Torch code is correct - to get an average of 10% o… (#32335) fixes #32329 : The Torch code is correct - to get an average of 10% of the total, we want to take 50% of the remainder after we've already masked 80% with [MASK] in the previous step. * Repo checks: skip docstring checks if not in the diff (#32328) * tmp * skip files not in the diff * use git.Repo instead of an external subprocess * add tiny change to confirm that the diff is working on pushed changes * add make quality task * more profesh main commit reference * Fix slow GemmaTokenizer and improve SPM slow -> fast conversion process (#32191) * Remove user-defined tokens which can be obtained through merges * Remove debug line * formatting * Refactor spm slow -> fast converter * revert unnecessary refactor * set comprehension * remove test files * Use `vocab_scores` * Always replace spiece underline with space in decode * we no longer need token filtering * Add save fast load slow unit test * Remove tokenizers version check * Remove duplicate code * Make `<start_of_turn>` and `<end_of_turn>` special tokens * Bias merge priority with length if score is the same * Add unit test for merge priority * CI * LLaVA-NeXT: fix anyres shapes (#32314) fix * Gemma2 and flash-attention (#32188) * enable flash-attn & static cache * this works, not the prev * fix for sliding window layers * not needed anymore * Llama 3.1: Fix incorrect `inv_freq` assignment (#32330) fix 💩 * [Idefics2] - Fix FA2 call for Perceiver layer (#32275) * Fix FA2 call for Perciever layer * [run_slow] idefics2 * [run_slow] idefics2 * [run_slow] idefics2 * Fix up * [run_slow] idefics2 * [run_slow] idefics2 * [run_slow] idefics2 * Gemma 2: support assisted generation (#32357) * Fix error when streaming to gradio with non-string tool arguments (#32360) Fix error when streaming agent run to gradio with non-string tool arguments * >3-5x faster torch.compile forward compilation for autoregressive decoder models (#32227) * draft * apply changes to all relevant archs * rerun ci - check_docstrings.py failing? * fix docstring * move 2D->4D mask creation to modeling file * repo consistency * fix the batch size = 1 case - calling contiguous is not enough * nit * style * propagate to gemma/gemma-2 * prepare inputs for gemma generation * implement test and tiny fix in gemma2 * Update src/transformers/models/bloom/modeling_bloom.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * fix copies * ci pass * fix gemma's test_compile_static_cache tests * flacky * retrigger ci --------- Co-authored-by: sanchit-gandhi <sanchit@huggingface.co> Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * fix: Removed unnecessary `@staticmethod` decorator (#32361) * Fixed staticmethods with self as first argument. * Fixed staticmethods with self as first argument. * Fixed staticmethods with self as first argument. * Fixed staticmethods with self as first argument. * fix: warmup_steps check for training_args (#32236) * LLaVa: add cache class attribute (#32278) cache class flag * [enc-dec cache] fix bug in indexing (#32370) * [whisper] compile compatibility with long-form decoding (#31772) * [whisper] compile compatibility with long-form decoding * clarify comment * fix after rebase * finalise * fix bsz * fix cache split * remove contiguous * style * finish * update doc * prevent cuda graph trace * Remove size check between attn_weights and kv_seq_len for phi3 (#32339) * Remove size check between attn_weights and kv_seq_len * add unit tests * add missing attribute _supports_param_buffer_assignment for gpt-j. (#32359) Co-authored-by: Guoming Zhang <37257613+nv-guomingz@users.noreply.github.com> * Check device map for saving tokenizer config on TPU (fix for issue #31971) (#32043) * Remove TPU device map for saving tokenizer config * Update tokenization_utils_base.py * Fix error msg when passing non-string device into tokenizer * Fix error message for non-string tokenizer device * Print out tokenizer device type in error msg * Update tokenization_utils_base.py * update clean_up_tokenization_spaces warning (#32371) * Empty list in defaults for LLaMA special tokens during weights conversion (#32342) empty list in defaults * Fix conflicting key in init kwargs in PreTrainedTokenizerBase (#31233) * Fix conflicting key in init kwargs in PreTrainedTokenizerBase * Update code to check for callable key in save_pretrained * Apply PR suggestions * Invoke CI * Updates based on PR suggestion * Offloaded KV Cache (#31325) * Initial implementation of OffloadedCache * enable usage via cache_implementation * Address feedback, add tests, remove legacy methods. * Remove flash-attn, discover synchronization bugs, fix bugs * Prevent usage in CPU only mode * Add a section about offloaded KV cache to the docs * Fix typos in docs * Clarifications and better explanation of streams * Docker: add `speech` dep to the consistency docker image (#32374) * Fixed Hybrid Cache Shape Initialization. (#32163) * fixed hybrid cache init, added test * Fix Test Typo --------- Co-authored-by: Aaron Haag <aaron.haag@siemens.com> * Yell at the user if zero-3 init wasn't performed, but expected to have been done (#32299) * Test this zach * Test for improper init w/o zero3 * Move back * Apply suggestions from code review Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Get rid of stars in warning * Make private * Make clear --------- Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update docs (#32368) nits * RoPE: Add numerical tests ✨ (#32380) tests! :D * [generate] only require an attention mask for mps with torch<2.4 (#32367) * up * style * stopping * fix: (issue #32124) Exception raised when running `transformers/examples/flax/language-modeling/t5_tokenizer_model.py`. (#32157) fix: Exception raised when running . * MixtralFlashAttention2: put "plus 1" inside parentheses when calculating rotary_seq_len, allowing None position_ids input. (#31500) * Mixtral: remove unnecessary plus 1 when calculating rotary_seq_len, allowing position_ids=None (no auto position_ids generation could be unsafe) * fix typo [:-1] to [:, -1] * to meet formatting requirement * to meet formatting requirement * remove white space * MixtralFlashAttention2: put "+ 1" inside parentheses when calculating rotary_seq_len, allowing None position_ids input. Fix format/style issue. * propagate to startcoder2, phi3, mixtral and qwen2 * update qwen2_moe * Bump keras from 2.8.0 to 2.13.1 in /examples/research_projects/decision_transformer (#32393) Bump keras in /examples/research_projects/decision_transformer Bumps [keras](https://github.com/keras-team/keras) from 2.8.0 to 2.13.1. - [Release notes](https://github.com/keras-team/keras/releases) - [Commits](https://github.com/keras-team/keras/compare/v2.8.0...v2.13.1) --- updated-dependencies: - dependency-name: keras dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * fix: SeamlessM4TFeatureExtractor stride remainder (#32088) * fix: SeamlessM4TFeatureExtractor stride remainder * Added attention mask size test * Reran ruff for style correction * Phi3 tests: fix typing for Python 3.8 (#32388) fix phi * #32184 save total_vocab_size (#32240) * save total_vocab_size = vocab_size + user added tokens to speed up operation * updating length when added_tokens_decoder is set * add test len(tokenizer) * add values for neftune (#32399) I always forget what typical values are, and I have to look at the paper everytime. This will be a helpful reminder. * Fix documentation references to google/bit-50 model (#32407) * Persist embedding type of BART and mBART models after resize (#32242) * fix: persist embedding type of MBartConditonalGeneration after resize * fix: persist embedding type of BartConditonalGeneration after resize * fix: Updated `test_embeded_special_tokens` for luke and mluke models (#32413) Fixed tokenizertests for luke, mluke models. * Respect the config's attn_implementation if set (#32383) * Respect the config's attn if set * Update test - can override in from_config * Fix * Fix documentation links and code reference to model llava-next (#32434) * Cache: create docs (#32150) * draft * updates * works? * try adding python example in hidden section * another try * hwo do i render python * format as html code? * Update docs/source/en/kv_cache.md Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update docs/source/en/kv_cache.md Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update docs/source/en/kv_cache.md Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update docs/source/en/kv_cache.md Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update docs/source/en/kv_cache.md Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * one more small update * should render hidden secrtion now * add outputs * fix links * check links * update all links * update with offloaded cache * all cache is importable, so they appear in docs * fix copies * docstring... --------- Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Llava: fix checkpoint_doc (#32458) fix: add new llava like model bug * add the missing flash attention test marker (#32419) * add flash attention check * fix * fix * add the missing marker * bug fix * add one more * remove order * add one more * Update kwargs validation for `preprocess` with decorator (#32024) * BLIP preprocess * BIT preprocess * BRIDGETOWER preprocess * CHAMELEON preprocess * CHINESE_CLIP preprocess * CONVNEXT preprocess * DEIT preprocess * DONUT preprocess * DPT preprocess * FLAVA preprocess * EFFICIENTNET preprocess * FUYU preprocess * GLPN preprocess * IMAGEGPT preprocess * INTRUCTBLIPVIDEO preprocess * VIVIT preprocess * ZOEDEPTH preprocess * VITMATTE preprocess * VIT preprocess * VILT preprocess * VIDEOMAE preprocess * VIDEOLLAVA * TVP processing * TVP fixup * SWIN2SR preprocess * SIGLIP preprocess * SAM preprocess * RT-DETR preprocess * PVT preprocess * POOLFORMER preprocess * PERCEIVER preprocess * OWLVIT preprocess * OWLV2 preprocess * NOUGAT preprocess * MOBILEVIT preprocess * MOBILENETV2 preprocess * MOBILENETV1 preprocess * LEVIT preprocess * LAYOUTLMV2 preprocess * LAYOUTLMV3 preprocess * Add test * Update tests * Fix get large model config for Switch Transformer encoder only tester (#32438) * Dependencies: fix typo (#32389) deps_2 * Add Nemotron HF Support (#31699) * Add nemotron support * fix inference * add unit test * add layernorm1p as a class to avoid meta device mismatch * test fixed * Add copied_from statements * remove pretraining_tp args * remove nemotronlayernorm * force LN computation done in FP32 * remove nemotrontokenizer and use llamatokenizer * license update * add option for kv_channels for minitron8b * remove assert * o_proj fixed * o_proj reshape * add gated_proj option * typo * remove todos * fix broken test after merging latest main * remove nezha/nat after meging main * chnage default config to 15b model * add nemo conversion script * rename conversion script * remove gate_proj option * pr comment resolved * fix unit test * rename kv_channels to head_dim * resolve PR issue * add nemotron md * fix broken tests * refactor rope for nemotron * test fix * remove linearscaling * whitespace and import * fix some copied-from * code style fix * reformatted * add position_embedding to nemotronattention * rope refactor to only use config, copied-from fix * format * Run make fix-copies * nemotron md with autodoc * doc fix * fix order * pass check_config_docstrings.py * fix config_attributes * remove all llama BC related code * Use PreTrainedTokenizerFast * ruff check examples * conversion script update * add nemotron to toctree * Generate: fix end to end compilation (#32465) * Add codestral mamba2 (#32080) * add new model like * draft cuda forward - mismatched keys (sharding on conv1) * match keys successfully * fix split * get generation/forward running (wrong gens, norm?) * :update * some refactoring * fixes * works up until copy to cache * fix * update * NON WORKING VERSION * version that work? * nit * fix config * fix conversion script * working cuda forward * nit * update * simplifcation * make mamba slow simple work * no einops * todo * fix style * no einops * update fix no einsum * nit * remove einops * bug: scan_output differs strongly * add rms norm option * fix fast + slow generation with and w/o cache :heavy_check_mark: * draft integration tests * remove a big chunk of the einsum * fix slow, fast generations, without any einsum * fix copies * fix structure * fix up modeling and tests * fix tests * clamping is indeed worse * recover mamba2 cache test * fix copies * no cache position (yet) * fix tf tests * fix matmul for generate * fixup * skip cache tests for now * [run-slow]mamba2 * tune out hidden states for padding * test batched generation * propagate attention mask changes * fix past length * fix integration test * style * address comments * update readme * add mamba2 version check * fix tests * [run-slow]mamba2 * skip edge tests * [run-slow]mamba2 * last fixup * [run-slow]mamba2 * update README --------- Co-authored-by: Arthur Zucker <arthur.zucker@gmail.com> * Migrate import checks not need accelerate, and be more clear on min versions (#32292) * Migrate import checks to secondary accelerate calls * better errs too * Revert, just keep the import checks + remove accelerate-specific things * Rm extra' * Empty commit for ci * Small nits * Final * Documentation: BOS token_id deprecation change for NLLB (#32443) Update nllb.md * dev version 4.45.0 * `is_torchdynamo_compiling` -- cast a wide exception net (#32476) * cast a wide net * make fix-copies with a few manual changes * add copied from * Revert "fixes to properly shard FSDP across cpu and meta for cpu_effcient_loading for prequantized 4bit (#32276)" (#32477) * Revert "fixes to properly shard FSDP across cpu and meta for cpu_efficient_loading for prequantized 4bit (#32276)" This reverts commit 62c60a30181a65e1a3a7f19c3055a240a6a21335. We uncovered an issue with this change that caused our training runs to hang. * `is_torchdynamo_compiling` -- cast a wide exception net (#32476) * cast a wide net * make fix-copies with a few manual changes * add copied from --------- Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * 🌐 [i18n-KO] Translated `mask_generation.md` to Korean (#32257) * docs: ko: tasks/mask_generation.md * feat: nmt draft * fix : toc local * fix : manual edits * fix : ko-toctree * fix: resolve suggestions Co-authored-by: boyunJang <gobook1234@naver.com> Co-authored-by: Chaewon Song <chaewon1019@ewhain.net> * fix: resolve suggestions Co-authored-by: boyunJang <gobook1234@naver.com> Co-authored-by: Chaewon Song <chaewon1019@ewhain.net> * fix: resolve suggestions * fix: resolve suggestions * fix: resolve suggestions --------- Co-authored-by: boyunJang <gobook1234@naver.com> Co-authored-by: Chaewon Song <chaewon1019@ewhain.net> * 🌐 [i18n-KO] Translated `idefics.md` to Korean (#32258) * docs: ko: tasks/idefics.md * feat: nmt draft * fix: manual edits * fix: resolve suggestions Co-authored-by: Chaewon Song <chaewon1019@ewhain.net> Co-authored-by: Harheem Kim <49297157+harheem@users.noreply.github.com> Co-authored-by: timdalxx <48753785+jeongiin@users.noreply.github.com> --------- Co-authored-by: Chaewon Song <chaewon1019@ewhain.net> Co-authored-by: Harheem Kim <49297157+harheem@users.noreply.github.com> Co-authored-by: timdalxx <48753785+jeongiin@users.noreply.github.com> * 🌐 [i18n-KO] Translated `image_to_image.md` to Korean (#32327) * docs: ko: tasks/image_to_image.md * feat: nmt draft * fix: manual edits * fix: resolve suggestions Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com> Co-authored-by: Jiwook Han <33192762+mreraser@users.noreply.github.com> * fix: handle remaining suggestions Co-authored-by: Jiwook Han <33192762+mreraser@users.noreply.github.com> --------- Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com> Co-authored-by: Jiwook Han <33192762+mreraser@users.noreply.github.com> * Cache: new Cache format in decoder-only models (#31421) * draft bart with new cache * add cache for decoder-only models * revert utils * modify docstring * revert bart * minor fixes * fix copies (not related) * revert tests * remove enc-dec related code * remove bloom * remove opt (enc-dec) * update docstring * git, codegen, gpt_neo, gpt_neox, gpj * clean up * copied from statements * revert * tmp * update warning msg * forgot git * add more flags * run-slow git,codegen,gpt_neo,gpt_neox,gpj * add cache flag to VLMs * remove files * style * video LLMs also need a flag * style * llava will go in another PR * style * [run-slow] codegen, falcon, git, gpt_neo, gpt_neox, gptj, idefics * Update src/transformers/models/gpt_neo/modeling_gpt_neo.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * copy from * deprecate until v4.45 and warn if not training * nit * fix test * test static cache * add more tests and fix models * fix copies * return sliding window mask * run slow tests & fix + codestyle * one more falcon fix for alibi --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Gemma2: add cache warning (#32279) * gemma2 fallback to dynamic cache * Update src/transformers/models/gemma2/modeling_gemma2.py Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> * Update src/transformers/models/gemma2/modeling_gemma2.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * raise error and dont fallback to dynamic cache * prev will break most forward calls/tests * Update src/transformers/models/gemma2/modeling_gemma2.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * update * fix copies --------- Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * enable xla fsdp (#32048) * enable xla fsdp * add acceleration version check for xla fsdp * Fix typo in tokenization_utils_base.py (#32484) * Agents use grammar (#31735) * Allow optional use of grammars to constrain generation * fix broken link in docs (#32491) `https://huggingface.co/docs/transformers/en/main_classes/pipelines#transformers.TextGenerationPipeline.__call__` `generate_kwargs (dict, optional) — Additional keyword arguments to pass along to the generate method of the model (see the generate method corresponding to your framework here).` link in "here" doesnt work * Docs: alert for the possibility of manipulating logits (#32467) * logits * words * 🌐 [i18n-KO] Translated `gptq.md` to Korean (#32293) * fix: manual edits * fix: manual edits2 * fix: delete files * fix: resolve suggestions Co-authored-by: Sungmin Oh <fabxoe.kor@gmail.com> Co-authored-by: SeungYoun Lee <84276596+win2dvp21@users.noreply.github.com> Co-authored-by: 김준재 <55151385+junejae@users.noreply.github.com> * fix: resolve suggestions Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Co-authored-by: Sungmin Oh <fabxoe.kor@gmail.com> Co-authored-by: SeungYoun Lee <84276596+win2dvp21@users.noreply.github.com> Co-authored-by: 김준재 <55151385+junejae@users.noreply.github.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * 🌐 [i18n-KO] Translated `prompting.md` to Korean (#32294) * docs: ko: tasks/prompting.md * feat: nmt-draft * fix: update translation in prompting.md * fix: update toctree.yml * fix: manual edits * fix: toctree edits * fix: resolve suggestions Co-authored-by: boyunJang <gobook1234@naver.com> Co-authored-by: Harheem Kim <49297157+harheem@users.noreply.github.com> Co-authored-by: timdalxx <48753785+jeongiin@users.noreply.github.com> --------- Co-authored-by: boyunJang <gobook1234@naver.com> Co-authored-by: Harheem Kim <49297157+harheem@users.noreply.github.com> Co-authored-by: timdalxx <48753785+jeongiin@users.noreply.github.com> * 🌐 [i18n-KO] Translated `quantization/quanto.md` to Korean (#32281) * docs: ko: quantization/quanto.md * feat: nmt draft * fix: resolve suggestions Co-authored-by: SeungYoun Lee <84276596+win2dvp21@users.noreply.github.com> Co-authored-by: Minki Kim <100768622+1kmmk1@users.noreply.github.com> Co-authored-by: 김준재 <55151385+junejae@users.noreply.github.com> * fix: resolve suggestions Co-authored-by: SeungYoun Lee <84276596+win2dvp21@users.noreply.github.com> --------- Co-authored-by: SeungYoun Lee <84276596+win2dvp21@users.noreply.github.com> Co-authored-by: Minki Kim <100768622+1kmmk1@users.noreply.github.com> Co-authored-by: 김준재 <55151385+junejae@users.noreply.github.com> * 🌐 [i18n-KO] Translated `image_feature_extraction.md` to Korean (#32239) * docs: ko: tasks/images_feature_extraction.md * feat: nmt draft * fix: manual edits * fix: manual edits * fix: manual edits * fix: manual edits * feat: manual edits * Update docs/source/ko/tasks/image_feature_extraction.md Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com> * Update docs/source/ko/tasks/image_feature_extraction.md Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com> * fix: manual edits --------- Co-authored-by: Jihun Lim <31366038+heuristicwave@users.noreply.github.com> * Fix references to model google mt5 small (#32497) * Docs: Fixed WhisperModel.forward’s docstring link (#32498) Fixed WhisperModel.forward’s docstring link. * 🌐 [i18n-KO] Translated `chat_templating.md` to Korean (#32362) * docs: ko: chat_templating.md * feat: nmt draft * fix: manual edits * Update docs/source/ko/chat_templating.md Co-authored-by: Sungmin Oh <fabxoe.kor@gmail.com> * Update docs/source/ko/chat_templating.md Co-authored-by: Sungmin Oh <fabxoe.kor@gmail.com> * fix: apply suggestions from code review - anchor Co-authored-by: Sungmin Oh <fabxoe.kor@gmail.com> * fix: manual edits Co-authored-by: SeungYoun Lee <84276596+win2dvp21@users.noreply.github.com> Co-authored-by: Minki Kim <100768622+1kmmk1@users.noreply.github.com> * fix: manual edits * fix: delete 'default template' section --------- Co-authored-by: Sungmin Oh <fabxoe.kor@gmail.com> Co-authored-by: SeungYoun Lee <84276596+win2dvp21@users.noreply.github.com> Co-authored-by: Minki Kim <100768622+1kmmk1@users.noreply.github.com> * Fix link to autoclass_tutorial.md in i18n.md (#32501) * Fix typo: depracted -> deprecated (#32489) Hello! ## Pull Request overview * Fix typo ## Details This should speak for itself. cc @itazap @ArthurZucker - Tom Aarsen * Fix issue #32518: Update llm_tutorial.md (#32523) Update llm_tutorial.md remove comma re: issue 32518 https://github.com/huggingface/transformers/issues/32518 * Change Phi3 `_supports_sdpa` to True (#32457) * Change `_supports_sdpa` to True * add phi3 to sdpa support list * Uniformize kwargs for processors - GroundingDINO (#31964) * fix typo * uniform kwargs * make style * add comments * remove return_tensors * remove common_kwargs from processor since it propagates * make style * return_token_type_ids to True * revert the default imagekwargs since does not accept any value in the image processro * revert processing_utils.py * make style * add molbap's commit * fix typo * fix common processor * remain * Revert "add molbap's commit" This reverts commit a476c6ee88318ce40d73ea31e2dc2d4faa8ae410. * add unsync PR * revert * make CI happy * nit * import annotationformat * Fix add-new-model-like (#31773) * handle (processor_class, None) returned by ModelPatterns * handle (slow, fast) image processors in add model * handle old image processor case * Add Qwen2-Audio (#32137) * add qwen2audio * Update check_repo.py * fix style * fix test * fix style * add model size * Qwen2AudioEncoderModel->Qwen2AudioEncoder; add copy info * Update src/transformers/models/qwen2_audio/modeling_qwen2_audio.py Co-authored-by: Yoach Lacombe <52246514+ylacombe@users.noreply.github.com> * Update src/transformers/models/qwen2_audio/modeling_qwen2_audio.py Co-authored-by: Yoach Lacombe <52246514+ylacombe@users.noreply.github.com> * Update src/transformers/models/qwen2_audio/modeling_qwen2_audio.py Co-authored-by: Yoach Lacombe <52246514+ylacombe@users.noreply.github.com> * switch the attention_mask and the feature_attention_mask * add to PRIVATE_MODELS in check_repo.py; add to MODEL_NAMES_TO_IGNORE in check_table.py * fix initialization * update chat_template * fix consistency issue after copy * add docstrings to _merge_input_ids_with_audio_features * add copied from to prepare_inputs_for_generation * add more details to docs * rm comment * add init_std * Update src/transformers/models/qwen2_audio/modeling_qwen2_audio.py Co-authored-by: Yoach Lacombe <52246514+ylacombe@users.noreply.github.com> * Update src/transformers/models/qwen2_audio/modeling_qwen2_audio.py Co-authored-by: Yoach Lacombe <52246514+ylacombe@users.noreply.github.com> * Update src/transformers/models/qwen2_audio/modeling_qwen2_audio.py Co-authored-by: Yoach Lacombe <52246514+ylacombe@users.noreply.github.com> * Update src/transformers/models/qwen2_audio/modeling_qwen2_audio.py Co-authored-by: Yoach Lacombe <52246514+ylacombe@users.noreply.github.com> * update * Update docs/source/en/model_doc/qwen2_audio.md Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * update tests * rm ignore_index * update processor * rm ffmpeg_read * Update tests/models/qwen2_audio/test_modeling_qwen2_audio.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update docs/source/en/model_doc/qwen2_audio.md Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update docs/source/en/model_doc/qwen2_audio.md Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update docs/source/en/model_doc/qwen2_audio.md Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * update * typo * [run_slow] qwen2_audio * [run_slow] qwen2_audio * [run_slow] qwen2_audio * fix quality * [run_slow] qwen2_audio * [run_slow] qwen2_audio * [run_slow] qwen2_audio * add official model --------- Co-authored-by: Yoach Lacombe <52246514+ylacombe@users.noreply.github.com> Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * filter flash_attn optional imports loading remote code (#30954) * filter flash_attn optional imports loading remote code * improve pattern * fix code style * Update src/transformers/dynamic_module_utils.py Co-authored-by: Matt <Rocketknight1@users.noreply.github.com> --------- Co-authored-by: Matt <Rocketknight1@users.noreply.github.com> * 🌐 [i18n-KO] Translated `ko-llm_tutorial_optimization.md` to Korean (#32372) * docs: ko: llm_tutorial_optimization.md * feat: nmt draft * fix: manual edits * Update docs/source/ko/llm_tutorial_optimization.md Co-authored-by: Chaewon Song <chaewon1019@ewhain.net> * Update docs/source/ko/llm_tutorial_optimization.md Co-authored-by: Chaewon Song <chaewon1019@ewhain.net> * fix: resolve suggestions - 1 Co-authored-by: Chaewon Song <chaewon1019@ewhain.net> Co-authored-by: timdalxx <48753785+jeongiin@users.noreply.github.com> Co-authored-by: boyunJang <gobook1234@naver.com> * fix: resolve suggestions - 2 Co-authored-by: boyunJang <gobook1234@naver.com> Co-authored-by: Chaewon Song <chaewon1019@ewhain.net> Co-authored-by: timdalxx <48753785+jeongiin@users.noreply.github.com> --------- Co-authored-by: Chaewon Song <chaewon1019@ewhain.net> Co-authored-by: timdalxx <48753785+jeongiin@users.noreply.github.com> Co-authored-by: boyunJang <gobook1234@naver.com> * 🌐 [i18n-KO] Translated `trainer.md` to Korean (#32260) * docs: ko: ko-trainer * feat: nmt draft * fix: manual edits * fix: manual edits * fix: glossary * fix: glossary * Apply suggestions from code review Co-authored-by: Jinuk <45095330+JinukHong@users.noreply.github.com> Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com> --------- Co-authored-by: Jinuk <45095330+JinukHong@users.noreply.github.com> Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com> * 🌐 [i18n-KO] Translated `eetq.md` to Korean (#32352) * docs: ko: quantization/eetq.md * feat: nmt draft * fix docs: ko: quantization/eetq.md * fix docs: ko: quantization/eetq.md * fix: resolve suggestions Co-authored-by: Jiwook Han <33192762+mreraser@users.noreply.github.com> * fix: resolve suggestions * fix: resolve suggsetions --------- Co-authored-by: Jiwook Han <33192762+mreraser@users.noreply.github.com> * 🌐 [i18n-KO] Translated `fsdp.md` to Korean (#32261) * docs: ko: fsdp.md * feat: nmt draft * fix: manual edits * Apply suggestions from code review Co-authored-by: 김준재 <55151385+junejae@users.noreply.github.com> Co-authored-by: Minki Kim <100768622+1kmmk1@users.noreply.github.com> * fix: resolve suggestions * Update docs/source/ko/fsdp.md Co-authored-by: 김준재 <55151385+junejae@users.noreply.github.com> * Update docs/source/ko/fsdp.md Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Co-authored-by: 김준재 <55151385+junejae@users.noreply.github.com> Co-authored-by: Minki Kim <100768622+1kmmk1@users.noreply.github.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * 🌐 [i18n-KO] Translated `bitsandbytes.md` to Korean (#32408) * docs: ko: quantization/bitsandbytes.md * feat: nmt draft * fix: minor typos * fix: manual edits * fix: manual edits * fix: resolve suggestions Co-authored-by: wony617 <49024958+Jwaminju@users.noreply.github.com> Co-authored-by: YONGSANG <71686691+4N3MONE@users.noreply.github.com> Co-authored-by: Woojun Jung <46880056+jungnerd@users.noreply.github.com> * fix: resolve suggestions Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Co-authored-by: wony617 <49024958+Jwaminju@users.noreply.github.com> Co-authored-by: YONGSANG <71686691+4N3MONE@users.noreply.github.com> Co-authored-by: Woojun Jung <46880056+jungnerd@users.noreply.github.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Fix generate with `inputs_embeds` as input (#32493) * I think inputs_embeds has ndim == 3 * fix sequence length catch * add generate test * [run-slow]olmo, persimmon, gemma, gemma2, qwen2, llama * skip whisper * fix bart test * more fixes * Fixed test `test_static_cache_exportability` with torch 2.4.0 (#32516) Workaround the export issue in torch 2.4 Co-authored-by: Guang Yang <guangyang@fb.com> * Fix code example to load bigcode starcoder2 7b (#32474) * [docs] Translation guide (#32547) clarify * Gemma2: fix FA2 generation (#32553) fix FA2 * Fix a bug in Qwen2Audio (#32552) fix _update_model_kwargs_for_generation * fix slow integration gemma2 test (#32534) no empty revision * fix non contiguous tensor value error in save_pretrained (#32422) Signed-off-by: duzhanwei <duzhanwei@bytedance.com> Co-authored-by: duzhanwei <duzhanwei@bytedance.com> * 🌐 [i18n-KO] Translated `agent.md` to Korean (#32351) * docs: ko: main_classes/agent * feat: chatgpt draft * fix: manual edits * �fix: resolve suggestions Co-authored-by: Woojun Jung <46880056+jungnerd@users.noreply.github.com> Co-authored-by: thsamaji <60818655+thsamajiki@users.noreply.github.com> Co-authored-by: SeungAhSon <gongsoonyee@gmail.com> * fix: resolve suggestions * fix: resolve code line number --------- Co-authored-by: Woojun Jung <46880056+jungnerd@users.noreply.github.com> Co-authored-by: thsamaji <60818655+thsamajiki@users.noreply.github.com> Co-authored-by: SeungAhSon <gongsoonyee@gmail.com> * Add new model (#32615) * v1 - working version * fix * fix * fix * fix * rename to correct name * fix title * fixup * rename files * fix * add copied from on tests * rename to `FalconMamba` everywhere and fix bugs * fix quantization + accelerate * fix copies * add `torch.compile` support * fix tests * fix tests and add slow tests * copies on config * merge the latest changes * fix tests * add few lines about instruct * Apply suggestions from code review Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * fix * fix tests --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Fix: FA2 with packed training (#32487) * fix check * add tests * [run-slow] llama, gemma2 * oops, whisper actually runs but needed some special treatment * Fix sliding window attention used in Gemma2FlashAttention2 (#32522) * fix sliding window attention (flash2) in gemma2 model * [run-slow] gemma * fix slicing attention_mask for flash_attn2 * fix slicing attention_mask when flash_attn is used * add missing comment * slice the last seq_len tokens in the key, value states * revert code of slicing key, value states * fix: Fixed conditional check for `encodec` model names (#32581) * Fixed conditional check for encodec model names. * Reformatted conditional check. * Fix `.push_to_hub(..., create_pr=True, revision="my-branch")` when creating PR on not-owned repo (#32094) Fix create_pr aagainst existing revision * Bump aiohttp from 3.9.4 to 3.10.2 in /examples/research_projects/decision_transformer (#32569) Bump aiohttp in /examples/research_projects/decision_transformer Bumps [aiohttp](https://github.com/aio-libs/aiohttp) from 3.9.4 to 3.10.2. - [Release notes](https://github.com/aio-libs/aiohttp/releases) - [Changelog](https://github.com/aio-libs/aiohttp/blob/master/CHANGES.rst) - [Commits](https://github.com/aio-libs/aiohttp/compare/v3.9.4...v3.10.2) --- updated-dependencies: - dependency-name: aiohttp dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump torch from 1.13.1 to 2.2.0 in /examples/research_projects/visual_bert (#32220) Bump torch in /examples/research_projects/visual_bert Bumps [torch](https://github.com/pytorch/pytorch) from 1.13.1 to 2.2.0. - [Release notes](https://github.com/pytorch/pytorch/releases) - [Changelog](https://github.com/pytorch/pytorch/blob/main/RELEASE.md) - [Commits](https://github.com/pytorch/pytorch/compare/v1.13.1...v2.2.0) --- updated-dependencies: - dependency-name: torch dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Cleanup tool calling documentation and rename doc (#32337) * Rename "Templates for Chat Models" doc to "Chat Templates" * Small formatting fix * Small formatting fix * Small formatting fix * Cleanup tool calling docs as well * Remove unneeded 'revision' * Move tip to below main code example * Little bonus section on template editing * 🌐 [i18n-KO] Translated `deepspeed.md` to Korean (#32431) * Update _toctree.yml * docs: ko: deepspeed.md * Apply suggestions from code review Co-authored-by: wony617 <49024958+Jwaminju@users.noreply.github.com> * Apply suggestions from code review Co-authored-by: wony617 <49024958+Jwaminju@users.noreply.github.com> * Update docs/source/ko/_toctree.yml Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * Update docs/source/ko/deepspeed.md * Update docs/source/ko/deepspeed.md Co-authored-by: SeungAhSon <gongsoonyee@gmail.com> * Apply suggestions from code review Co-authored-by: wony617 <49024958+Jwaminju@users.noreply.github.com> * Update docs/source/ko/_toctree.yml --------- Co-authored-by: wony617 <49024958+Jwaminju@users.noreply.github.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> Co-authored-by: SeungAhSon <gongsoonyee@gmail.com> * 🌐 [i18n-KO] Translated `awq.md`to Korean (#32324) * fix: manual edits * Apply suggestions from code review Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com> Co-authored-by: Chulhwa (Evan) Han <cjfghk5697@ajou.ac.kr> * fix:manual edits - 잘못된 경로에 번역본 파일을 생성해서 옮김 * Delete docs/source/ko/tasks/awq.md * Update docs/source/ko/_toctree.yml Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Co-authored-by: SeongWooChoi <46990061+nuatmochoi@users.noreply.github.com> Co-authored-by: Chulhwa (Evan) Han <cjfghk5697@ajou.ac.kr> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * fix: Fixed failing `test_find_base_model_checkpoint` (#32638) Fixed failing test_find_base_model_checkpoint. * Bump tensorflow from 2.11.1 to 2.12.1 in /examples/research_projects/decision_transformer (#32341) Bump tensorflow in /examples/research_projects/decision_transformer Bumps [tensorflow](https://github.com/tensorflow/tensorflow) from 2.11.1 to 2.12.1. - [Release notes](https://github.com/tensorflow/tensorflow/releases) - [Changelog](https://github.com/tensorflow/tensorflow/blob/master/RELEASE.md) - [Commits](https://github.com/tensorflow/tensorflow/compare/v2.11.1...v2.12.1) --- updated-dependencies: - dependency-name: tensorflow dependency-type: direct:production ... Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * "to be not" -> "not to be" (#32636) * "to be not" -> "not to be" * Update sam.md * Update trainer.py * Update modeling_utils.py * Update test_modeling_utils.py * Update test_modeling_utils.py * fix: Updated the `is_torch_mps_available()` function to include `min_version` argument (#32545) * Fixed wrong argument in is_torch_mps_available() function call. * Fixed wrong argument in is_torch_mps_available() function call. * sorted the import. * Fixed wrong argument in is_torch_mps_available() function call. * Fixed wrong argument in is_torch_mps_available() function call. * Update src/transformers/utils/import_utils.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * removed extra space. * Added type hint for the min_version parameter. * Added missing import. --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Expand inputs in processors for VLMs (#30962) * let it be * draft * should not have changed * add warnings * fix & add tests * fix tests * ipnuts embeds cannot be passed with pixels * more updates * paligemma ready! * minor typos * update blip-2 * fix tests & raise error * docstring * add blip2 test * tmp * add image seq length to config * update docstring * delete * fix tests * fix blip * fix paligemma * out-of-place scatter * add llava-next-video * Update src/transformers/models/blip_2/modeling_blip_2.py Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com> * remove tmp * codestyle * nits * more nits * remove overriding in tests * comprehension when merging video * fix-copies * revert changes for embeds test * fix tests after making comprehension * Update src/transformers/models/blip_2/processing_blip_2.py Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com> * Update src/transformers/models/blip_2/processing_blip_2.py Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com> * more updates * fix tests --------- Co-authored-by: Pablo Montalvo <39954772+molbap@users.noreply.github.com> * Automatically add `transformers` tag to the modelcard (#32623) * Automatically add `transformers` tag to the modelcard * Specify library_name and test * Fix tests (#32649) * skip failing tests * [no-filter] * [no-filter] * fix wording catch in FA2 test * [no-filter] * trigger normal CI without filtering * fix tensors on different devices in `WhisperGenerationMixin` (#32316) * fix * enable on xpu * no manual remove * move to device * remove to * add move to * Add support for GrokAdamW optimizer (#32521) * add grokadamw * reformat * code review feedback, unit test * reformat * reformat * Add Depth Anything V2 Metric models (#32126) * add checkpoint and repo names * adapt head to support metric depth estimation * add max_depth output scaling * add expected logits * improve docs * fix docstring * add checkpoint and repo names * adapt head to support metric depth estimation * add max_depth output scaling * add expected logits * improve docs * fix docstring * rename depth_estimation to depth_estimation_type * add integration test * Refactored tests to include metric depth model inference test * Integration test pass when the timm backbone lines are commented (L220-L227) * address feedback * replace model path to use organization path * formatting * delete deprecated TODO * address feedback * [run_slow] depth_anything * Fix: Fixed directory path for utils folder in `test_tokenization_utils.py` (#32601) * Removed un-necessary expressions. * Fixed directory path for utils folder in test_tokenization_utils.py * Modify ProcessorTesterMixin for better generalization (#32637) * Add padding="max_length" to tokenizer kwargs and change crop_size to size for image_processor kwargs * remove crop_size argument in align processor tests to be coherent with base tests * Add pad_token when loading tokenizer if needed, change test override tokenizer kwargs, remove unnecessary test overwrites in grounding dino * TF_Deberta supporting mixed precision (#32618) * Update modeling_tf_deberta.py Corrected some codes which do not support mixed precision * Update modeling_tf_deberta_v2.py Corrected some codes which do not support mixed precision * Update modeling_tf_deberta_v2.py * Update modeling_tf_deberta.py * Add files via upload * Add files via upload * Fix tests recurrent (#32651) * add fix for recurrentgemma * [no-filter] * trigger-ci * [no-filter] * [no-filter] * attempt to fix mysterious zip error * [no-filter] * fix lookup error * [no-filter] * remove summarization hack * [no-filter] * Support MUSA (Moore Threads GPU) backend in transformers (#31913) Add accelerate version check, needs accelerate>=0.33.0 * fix: Fixed failing tests in `tests/utils/test_add_new_model_like.py` (#32678) * Fixed failing tests in tests/utils/test_add_new_model_like.py * Fixed formatting using ruff. * Small nit. * Update translation docs review (#32662) update list of people to tag * Add TorchAOHfQuantizer (#32306) * Add TorchAOHfQuantizer Summary: Enable loading torchao quantized model in huggingface. Test Plan: local test Reviewers: Subscribers: Tasks: Tags: * Fix a few issues * style * Added tests and addressed some comments about dtype conversion * fix torch_dtype warning message * fix tests * style * TorchAOConfig -> TorchAoConfig * enable offload + fix memory with multi-gpu * update torchao version requirement to 0.4.0 * better comments * add torch.compile to torchao README, add perf number link --------- Co-authored-by: Marc Sun <marc@huggingface.co> * Fix `JetMoeIntegrationTest` (#32332) JetMoeIntegrationTest Co-authored-by: ydshieh <ydshieh@users.noreply.github.com> * Update the distributed CPU training on Kubernetes documentation (#32669) * Update the Kubernetes CPU training example * Add namespace arg Signed-off-by: Dina Suehiro Jones <dina.s.jones@intel.com> --------- Signed-off-by: Dina Suehiro Jones <dina.s.jones@intel.com> * fix: Fixed unknown pytest config option `doctest_glob` (#32475) Fixed unknown config option doctest_glob. * Unpin deepspeed in Docker image/tests (#32572) Unpin deepspeed * Updated workflows to the latest versions (#32405) Updated few workflows to the latest versions. * reopen: llava-next fails to consider padding_side during Training (#32679) restore #32386 * fix: Corrected ` falcon-mamba-7b` model checkpoint name (#32837) Corrected the model checkpoint. * fix: update doc link for runhouse in README.md (#32664) * VLMs: small clean-up for cache class (#32417) * fix beam search in video llava * [run-slow] video_llava * add back the position ids (#32554) * add back the position ids * fix failing test * Use head_dim if in config for RoPE (#32495) * use head_dim if in config for RoPE * typo * simplify with getattr * Generate: unify `LogitsWarper` and `LogitsProcessor` (#32626) * [tests] make test_sdpa_equivalence device-agnostic (#32520) * fix on xpu * [run_all] * Cache: use `batch_size` instead of `max_batch_size` (#32657) * more precise name * better docstrings * Update src/transformers/cache_utils.py Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> --------- Co-authored-by: Arthur <48595927+ArthurZucker@users.noreply.github.com> * Fix AutoConfig and AutoModel support for Llava-Next-Video (#32844) * Fix: fix all model_type of Llava-Next-Video to llava_next_video * Fix doc for llava_next_video * * Fix formatting issues * Change llava-next-video.md file name into llava_next_video.md to make it compatible with implementation * Fix docs TOC for llava-next-video * improve _get_is_as_tensor_fns (#32596) * improve _get_is_as_tensor_fns * format * Revert PR 32299, flag users when Zero-3 was missed (#32851) Revert PR 32299 * fix multi-gpu with static cache (#32543) * Reduce the error log when using core models that need their weights renamed, and provide a step forward (#32656) * Fin * Modify msg * Finish up nits * Make beam_constraints.Constraint.advance() docstring more accurate (#32674) * Fix beam_constraints.Constraint.advance() docstring * Update src/transformers/generation/beam_constraints.py Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> --------- Co-authored-by: Joao Gante <joaofranciscocardosogante@gmail.com> Co-authored-by: Steven Liu <59462357+stevhliu@users.noreply.github.com> * generate: missing `to` in DoLa body, causing exceptions in multi-gpu generation (#32856) * Add Flax Dinov2 (#31960) * tfmsenv restored in main * installed flax * forward pass done and all tests passed * make fix-copies and cleaning the scripts * fixup attempt 1 * fixup attempt 2 * fixup third attempt * fixup attempt 4 * fixup attempt 5 * dinov2 doc fixed * FlaxDinov2Model + ForImageClassification added to OBJECTS_TO_IGNORE * external pos_encoding layer removed * fixup attempt 6 * fixed integration test values * fixup attempt 7 * Update src/transformers/models/dinov2/modeling_flax_dinov2.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/dinov2/modeling_flax_dinov2.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/dinov2/modeling_flax_dinov2.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/dinov2/modeling_flax_dinov2.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/dinov2/modeling_flax_dinov2.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/transformers/models/dinov2/modeling_flax_dinov2.py Co-authored-by: amyeroberts <22614925+amyeroberts@users.noreply.github.com> * Update src/tran…

ArthurZucker reviewed Jul 10, 2024

View reviewed changes

RhuiDih force-pushed the dev/fa_packing_posid branch from cf6271f to c3451db Compare July 12, 2024 07:00

ArthurZucker reviewed Jul 15, 2024

View reviewed changes

ArthurZucker approved these changes Jul 19, 2024

View reviewed changes

fxmarty reviewed Jul 19, 2024

View reviewed changes

Rhui Dih Lee added 10 commits July 23, 2024 18:42

add DataCollatorBatchFlattening

df3c9b2

Update data_collator.py

dfe08de

change name

8120b3a

new FA2 flow if position_ids is provided

0598510

add comments

1ff2343

minor fix

f97ab71

minor fix data collator

08e1f2c

add test cases for models

48ce3d2

add test case for data collator

6c6b168

remove extra code

00e7abf

RhuiDih force-pushed the dev/fa_packing_posid branch from 7573f85 to 00e7abf Compare July 23, 2024 10:42

Rhui Dih Lee added 3 commits July 23, 2024 20:00

formating for ruff check and check_repo.py

3a87293

ruff format

c60b720

ruff format tests src utils

custom_init_isort.py

9030559

ArthurZucker approved these changes Jul 23, 2024

View reviewed changes

ArthurZucker merged commit 9cf4f2a into huggingface:main Jul 23, 2024
23 checks passed

ArthurZucker mentioned this pull request Jul 24, 2024

Llama 3 - RuntimeError: shape '[-1, 0]' is invalid for input of size 41041920 #32170

Closed

4 tasks

This was referenced Jul 26, 2024

Idefics2 generation erroring with flash_attention_2 #32237

Closed

Flash-Attn: fix generation when no attention mask or no pading #32241

Merged

This was referenced Jul 30, 2024

add arg padding_free to DataCollatorForCompletionOnlyLM RhuiDih/trl#1

Closed

add arg padding_free to DataCollatorForCompletionOnlyLM huggingface/trl#1887

Merged

TJ-Solergibert mentioned this pull request Jul 30, 2024

Adding SFT training swiss-ai/nanotron#14

Open

7 tasks

This was referenced Aug 1, 2024

Introduce Padding-Free Plugin to FMS-Acceleration foundation-model-stack/fms-acceleration#57

Merged

feat: Add DataClass Arguments to Activate Padding-Free and MultiPack Plugin and FastKernels foundation-model-stack/fms-hf-tuning#280

Merged

eldarkurtic mentioned this pull request Aug 19, 2024

Add masking of different samples in a long sequence for flash-attention mechanism #32875

Closed

5 tasks

beep-bebop mentioned this pull request Aug 26, 2024

Customise the separator used for splicing in DataCollatorWithFlattening #33114

Merged

5 tasks

meliksahturker mentioned this pull request Sep 2, 2024

prepare_fa2_from_position_ids error in training with batch_size > 1 #33268

Closed

4 tasks

AlongWY mentioned this pull request Sep 18, 2024

Flatting Packing / maybe fix #5443 and #5426 hiyouga/LLaMA-Factory#5458

Closed

2 tasks

Cyrilvallez mentioned this pull request Sep 18, 2024

Flash-attn performance: remove cuda sync during inference #33570

Merged

taha-yassine mentioned this pull request Dec 18, 2024

Patch GPTNeoX to use adequate FA2 if position_ids is provided #35318

Merged

5 tasks

		@@ -59,6 +59,20 @@
		_CONFIG_FOR_DOC = "LlamaConfig"


		def prepare_fa2_from_position_ids(query, key, value, position_ids, query_length):

Enhancing SFT Training Efficiency Using Packing and FlashAttention2 with Position IDs #31629

Enhancing SFT Training Efficiency Using Packing and FlashAttention2 with Position IDs #31629

Conversation

RhuiDih commented Jun 26, 2024 • edited Loading

What does this PR do?

Before submitting

Who can review?

ArthurZucker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

RhuiDih commented Jul 12, 2024

RhuiDih commented Jul 12, 2024

ArthurZucker left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mayank31398 commented Jul 16, 2024

RhuiDih commented Jul 17, 2024

wynterl commented Jul 17, 2024

RhuiDih commented Jul 17, 2024

ArthurZucker left a comment

Choose a reason for hiding this comment

ArthurZucker commented Jul 19, 2024

fxmarty left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

HuggingFaceDocBuilderDev commented Jul 23, 2024

ArthurZucker left a comment

Choose a reason for hiding this comment

olisicky commented Aug 25, 2024 • edited Loading

RhuiDih commented Aug 26, 2024

olisicky commented Aug 27, 2024

RhuiDih commented Aug 27, 2024

olisicky commented Sep 2, 2024

MahmoudAshraf97 commented Sep 6, 2024

ArthurZucker commented Sep 6, 2024 • edited Loading

qibao77 commented Oct 15, 2024

What does this PR do?

Before submitting

Who can review?

RhuiDih commented Oct 16, 2024

RhuiDih commented Jun 26, 2024 •

edited

Loading

olisicky commented Aug 25, 2024 •

edited

Loading

ArthurZucker commented Sep 6, 2024 •

edited

Loading