Add custom head_dim support to Llama#32502

Closed

suhara wants to merge 12 commits intohuggingface:mainfrom suhara:suhara/llama-kv-channels

+10-8

Commits on Aug 18, 2024

Remove head_dim * num_heads == hidden_size assertion

suhara
authored and
Yoshi Suhara
committed
Add kv_channels to LlamaConfig

suhara
authored and
Yoshi Suhara
committed
Fix o_proj

suhara
authored and
Yoshi Suhara
committed
Fix o_proj

suhara
authored and
Yoshi Suhara
committed
Rename kv_channels->head_dim

suhara
authored and
Yoshi Suhara
committed
Merge nested if into single if to address the comment

suhara
authored and
Yoshi Suhara
committed
Update src/transformers/models/llama/modeling_llama.py

authored and
Yoshi Suhara
committed
Replace config.get() with getattr() to fix AttributeError

suhara
authored and
Yoshi Suhara
committed
Update src/transformers/models/llama/modeling_llama.py

authored and
Yoshi Suhara
committed
Set default value if head_dim is not defined in config.json

suhara
authored and
Yoshi Suhara
committed
Run make fixup; make fix-copies

suhara
authored and
Yoshi Suhara
committed
Rebased; remove unnecessary assertion block
Yoshi Suhara
committed