Skip to content

Commit

Permalink
add values for neftune (#32399)
Browse files Browse the repository at this point in the history
I always forget what typical values are, and I have to look at the paper everytime. This will be a helpful reminder.
  • Loading branch information
nbroad1881 committed Aug 7, 2024
1 parent acc920f commit 967fb90
Showing 1 changed file with 1 addition and 1 deletion.
2 changes: 1 addition & 1 deletion src/transformers/training_args.py
Original file line number Diff line number Diff line change
Expand Up @@ -771,7 +771,7 @@ class TrainingArguments:
If not `None`, this will activate NEFTune noise embeddings. This can drastically improve model performance
for instruction fine-tuning. Check out the [original paper](https://arxiv.org/abs/2310.05914) and the
[original code](https://github.com/neelsjain/NEFTune). Support transformers `PreTrainedModel` and also
`PeftModel` from peft.
`PeftModel` from peft. The original paper used values in the range [5.0, 15.0].
optim_target_modules (`Union[str, List[str]]`, *optional*):
The target modules to optimize, i.e. the module names that you would like to train, right now this is used only for GaLore algorithm
https://arxiv.org/abs/2403.03507
Expand Down

0 comments on commit 967fb90

Please sign in to comment.