Add llama3

[ArthurZucker]

TODOs

• docs/source/en/model_doc/llama3.md

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.

[devdevgoat]

Managed to get this commit working, but it requires setting flag --llama_version 3 when calling convert_llama_wieghts_to_hf.py

python src/transformers/models/llama/convert_llama_weights_to_hf.py \
--input_dir /mnt/nvme1n1/Models/llama3/Meta-Llama-3-8B-Instruct \
--model_size 8B \
--output_dir /mnt/nvme1n1/Models/llama3/Meta-Llama-3-8B-Instruct-hf \
--llama_version 3

Additionally, it tries to delete the tmp folder before it's empty, so throws an error at the end, even though conversion is successful.

[ArthurZucker]

it requires setting flag --llama_version 3 when calling convert_llama_wieghts_to_hf.py

that is expected since you are converting version3, but should still work for earlier versions.

[kisseternity]

Hello, I notice the mask procedure ensuring self-attention does not cross document boundaries in meta's llama3 blog, as the following:

Should llama3 have an extra attention mask for this?

[ArthurZucker]

In transformers we usually let the user create the custom mask. The trainer might even support it. but no it's not a modeling code change we want 😉 you can already pass any 4d or 2d mask to Llama

[LysandreJik]

LGTM

[LysandreJik]

Good!

[ArthurZucker]

Failing tests are unrelated

[lhanchao777]

I meet this error when using the convert_llama_weights_to_hf.py. ImportError: cannot import name 'TikTokenConverter' from 'transformers.convert_slow_tokenizer', how to solve it ? The version of my transformers is 4.40.1

[LysandreJik]

Hello @lhanchao777, conversion scripts should be used with a source install of transformers. You can install from source with the following:

pip install git+https://github.com/huggingface/transformers

You could also clone the repo and add it as an editable installation:

git clone https://github.com/huggingface/transformers
pip install -e ./transformers

[ZJL0111]

I meet this error when using the convert_llama_weights_to_hf.py. ImportError: cannot import name 'TikTokenConverter' from 'transformers.convert_slow_tokenizer', how to solve it ? The version of my transformers is 4.40.1

Hi, @lhanchao777 i download model from modelscop and meet the same problem when trans model format;
To solve this, you need to instal transformer from source;
cause transformer source code uodated to 4.41.0, but transformer of huggingface updated to 4.40.1，so we need to install transformer from source;
And install from conda not pip

Transformers can be installed using conda as follows:
conda install conda-forge::transformers
NOTE: Installing transformers from the huggingface channel is deprecated.
# todo: https://github.com/huggingface/transformers/tree/main

but even thoughi solve this problem, i still get other bugs which i have no clue; so finally i still get llama3 direclly from huggingface

``import transformers
import torch
from huggingface_hub import login
login(token="your access token")

model_id = "meta-llama/Meta-Llama-3-8B-Instruct"
pipeline = transformers.pipeline("text-generation",
model=model_id,
model_kwargs={"torch_dtype": torch.float16},
device="cuda",
)
pipeline("Hey how are you doing today?")``

hope it helps!

[danieltmeta]

Hi there! Not sure if this is the right place, but I'm trying to convert the 8B model to llama2, is that possible by changing the flag in this fashion:

python src/transformers/models/llama/convert_llama_weights_to_hf.py
--input_dir /path/to/downloaded/llama/weights --model_size 8B --output_dir /output/path --llama_version 2

--llama_version 3 works fine, but that result is not what I'm looking for in my application.

I keep getting this error:

RuntimeError: Internal: could not parse ModelProto from ./Meta-Llama-3-8B/original/tokenizer.model

Thanks!

[ZJL0111]

you want llama2, but your error path is llama "./Meta-Llama-3-8B/original/tokenizer.model", check your path please

…

________________________________ 发件人: danieltmeta ***@***.***> 发送时间: 2024年4月30日 4:23 收件人: huggingface/transformers ***@***.***> 抄送: ZJL0111 ***@***.***>; Comment ***@***.***> 主题: Re: [huggingface/transformers] Add llama3 (PR #30334) Hi there! Not sure if this is the right place, but I'm trying to convert the 8B model to llama2, is that possible by changing the flag in this fashion: python src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir /path/to/downloaded/llama/weights --model_size 8B --output_dir /output/path --llama_version 2

--llama_version 3 works fine, but that result is not what I'm looking for in my application. I keep getting this error: RuntimeError: Internal: could not parse ModelProto from ./Meta-Llama-3-8B/original/tokenizer.model Thanks! ― Reply to this email directly, view it on GitHub<#30334 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AEVZCQDAT35B76KQRCX7FTTY72T2LAVCNFSM6AAAAABGOSEOTKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDAOBTGU4TKOJWHE>. You are receiving this because you commented.Message ID: ***@***.***>

[NekoMimiUnagi]

Hi there! I followed the above instructions to convert Meta-Llama-3-8B to hf format but still got errors as follows:

Traceback (most recent call last):
  File "/data/models/origin-format/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 407, in <module>
    main()
  File "/data/models/origin-format/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 394, in main
    vocab_size = len(write_tokenizer(args.output_dir, spm_path, llama_version=args.llama_version))
  File "/data/models/origin-format/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 358, in write_tokenizer
    tokenizer = Llama3Converter(input_tokenizer_path).tokenizer
  File "/data/models/origin-format/transformers/src/transformers/models/llama/convert_llama_weights_to_hf.py", line 319, in __init__
    tokenizer = self.converted()
  File "/data/models/origin-format/transformers/src/transformers/convert_slow_tokenizer.py", line 1534, in converted
    tokenizer = self.tokenizer()
  File "/data/models/origin-format/transformers/src/transformers/convert_slow_tokenizer.py", line 1527, in tokenizer
    vocab_scores, merges = self.extract_vocab_merges_from_model(self.vocab_file)
  File "/data/models/origin-format/transformers/src/transformers/convert_slow_tokenizer.py", line 1503, in extract_vocab_merges_from_model
    bpe_ranks = load_tiktoken_bpe(tiktoken_url)
  File "/home/xxx/miniconda3/envs/pytorch/lib/python3.8/site-packages/tiktoken/load.py", line 115, in load_tiktoken_bpe
    return {
  File "/home/xxx/miniconda3/envs/pytorch/lib/python3.8/site-packages/tiktoken/load.py", line 117, in <dictcomp>
    for token, rank in (line.split() for line in contents.splitlines() if line)
ValueError: too many values to unpack (expected 2)

I installed transformers from the source file under the folder /data/models/origin-format/transformers/; the version is 4.41.0.dev0. By the way, I follow the instruction to install it.

Hello @lhanchao777, conversion scripts should be used with a source install of transformers. You can install from source with the following:

pip install git+https://github.com/huggingface/transformers

You could also clone the repo and add it as an editable installation:

git clone https://github.com/huggingface/transformers
pip install -e ./transformers

[ArthurZucker]

You are most probably not using the correct original tokenizer.model. We proof tested the script many times 😉

[danieltmeta]

@ArthurZucker You are correct, I was not using this correctly. Thanks!

[ucasyouzhao1987]

Hi there! Not sure if this is the right place, but I'm trying to convert the 8B model to llama2, is that possible by changing the flag in this fashion:

python src/transformers/models/llama/convert_llama_weights_to_hf.py --input_dir /path/to/downloaded/llama/weights --model_size 8B --output_dir /output/path --llama_version 2

--llama_version 3 works fine, but that result is not what I'm looking for in my application.

I keep getting this error:

RuntimeError: Internal: could not parse ModelProto from ./Meta-Llama-3-8B/original/tokenizer.model

Thanks!

I also have the error "Internal: could not parse ModelProto from ./Meta-Llama-3-8B/original/tokenizer.model", I found the "Meta-Llama-3-8B-chat" don't have the "tokenizer.model", how to solve it ?

[ArthurZucker]

You should use PreTrainedTokenizerFast or just AutoTokenizer!

[Weiriyue]

Traceback (most recent call last):
File "C:\Users\81408\AppData\Roaming\Python\Python311\site-packages\transformers\convert_llama_weights_to_hf.py", line 407, in
main()
File "C:\Users\81408\AppData\Roaming\Python\Python311\site-packages\transformers\convert_llama_weights_to_hf.py", line 396, in main
write_model(
File "C:\Users\81408\AppData\Roaming\Python\Python311\site-packages\transformers\convert_llama_weights_to_hf.py", line 178, in write_model
f"model.layers.{layer_i}.self_attn.k_proj.weight": permute(
^^^^^^^^
File "C:\Users\81408\AppData\Roaming\Python\Python311\site-packages\transformers\convert_llama_weights_to_hf.py", line 154, in permute
return w.view(n_heads, dim1 // n_heads // 2, 2, dim2).transpose(1, 2).reshape(dim1, dim2)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
RuntimeError: shape '[32, 2, 2, 4096]' is invalid for input of size 16777216
How to solve it?

[ArthurZucker]

Could you share which version of transformers you are using? And which version of llama you are trying to convert