Skip to content
This repository has been archived by the owner on Jun 24, 2024. It is now read-only.

Update to latest upstream BLOOM implementation / BLOOM Quantization does not work #228

Closed
danforbes opened this issue May 15, 2023 · 1 comment
Closed

Update to latest upstream BLOOM implementation / BLOOM Quantization does not work #228

danforbes opened this issue May 15, 2023 · 1 comment
Labels
issue:bug Something isn't working model:bloom BLOOM model
Milestone
0.2

Comments

@danforbes
Copy link
Contributor

I am able to perform inference using the f16 model found here https://huggingface.co/nouamanetazi/bloomz-560m-ggml/tree/main but when I use llm to quantize it to q4_0 format, the model produces gibberish.
@danforbes danforbes added issue:bug Something isn't working model:bloom BLOOM model labels May 15, 2023
@philpax
Copy link
Collaborator

philpax commented May 15, 2023

Just to confirm - the uploaded q4_0 works, right?

@danforbes danforbes mentioned this issue May 16, 2023
Closed
@philpax philpax mentioned this issue May 17, 2023
Closed
@philpax philpax added this to the 0.2 milestone May 18, 2023
@philpax philpax mentioned this issue May 20, 2023
Open
@philpax philpax changed the title BLOOM Quantization Does Not Work Update to latest upstream BLOOM implementation / BLOOM Quantization does not work May 22, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
issue:bug Something isn't working model:bloom BLOOM model
Projects
None yet
Milestone
0.2
Development

No branches or pull requests
2 participants
@philpax @danforbes