Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Convert fixes #3967

Closed
wants to merge 2 commits into from
Closed

Convert fixes #3967

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
d6d905b
convert: Fix detection of LLAMA2
Nov 6, 2023
f36a777
convert: Fix handling of LLAMA2 vocab_size = -1
Nov 6, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 6 additions & 1 deletion convert.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -250,9 +250,14 @@ def loadOriginalParamsJson(model: LazyModel, config_path: Path) -> Params:
if config.get("rope_theta") == 1000000:
# CodeLlama
n_ctx = 16384
elif config["norm_eps"] == 1e-05:
elif config["norm_eps"] in (1e-05, 1e-06):
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Then how do we detect LLaMA-1? See #2384

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, LLAMA2 files with norm_eps equal to 1e-06.

Copy link
Collaborator

@cebtenzzre cebtenzzre Nov 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you tell me where to get an original LLaMA-2 model with norm_eps=1e-6?

Copy link
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well I got it via the LLAMA2's repository download script and utilizing the link that was sent to my email. Since it's tied to my identity and accepting the EULA I'd say it's not prudent to share it? I guess if you create a new registration just now and try to download llama 7b you'd see it?

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I just downloaded LLaMA-2 7B and got this:

{"dim": 4096, "multiple_of": 256, "n_heads": 32, "n_layers": 32, "norm_eps": 1e-05, "vocab_size": -1}

So you'll have to be more specific.

Copy link
Author

@lorddoskias lorddoskias Nov 7, 2023 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, so I have actually downloaded llama2-7b-chat :

cat llama-2-7b-chat/params.json
{"dim": 4096, "multiple_of": 256, "n_heads": 32, "n_layers": 32, "norm_eps": 1e-06, "vocab_size": -1}

cat llama-2-7b-chat/checklist.chk
0c4837f3ef97f648452f91faed308a07 consolidated.00.pth
1c39bc3c6b51079fd807cc105b86c9df params.json

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can confirm this.

@ggerganov Our "hack to determine LLaMA v1 vs v2" is apparently insufficient. Maybe we need a CLI arg to disambiguate?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ugh, this is quite messy. I guess we do need to add a CLI arg that would be required in cases where the vocab_size is -1. In such cases we abort and ask the user to specify explicitly LLaMA v1 or LLaMA v2.

# LLaMA v2
n_ctx = 4096
# For some reason FB writes -1 to vocab size for their LLAMA2 models
# simply remove this bogus value and let the return statement belo
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

spelling

# figure it out
if config["vocab_size"] == -1:
del config["vocab_size"]
else:
# LLaMA v1
n_ctx = 2048
Expand Down