Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Neo Quantization Fixes #2724

Merged
merged 2 commits into from
Feb 6, 2025
Merged

Neo Quantization Fixes #2724

merged 2 commits into from
Feb 6, 2025

Conversation

a-ys
Copy link
Contributor

@a-ys a-ys commented Feb 5, 2025 •
edited
Loading

Description

Various fixes for neo fp8 CI:

  • Update llm/prepare.py to properly pass in option for FP8 quantization.
  • Do tokenization from FP8 on our side, like we were doing before with AutoFP8. Delegating this to llmcompressor leads to unneeded tokenization that leads to timeout in CI.
  • Use lmi-dist-venv to run awq quantization, due to hf transformers version incompatibility.

Type of change

  • Bug fix (non-breaking change which fixes an issue)

Feature/Issue validation/testing

a-ys added 2 commits February 5, 2025 19:23
@a-ys
update neo quantization test for llmcompressor
7d003c8
@a-ys
neo quantization script fixes
7596018 
- for fp8, do dataset prep and tokenization on our side rather than
  through llmcompressor. prevents timeout.
- use lmi-dist venv for awq, due to incompatible hf transformers version
@a-ys a-ys requested review from zachgk and a team as code owners February 5, 2025 23:31
@siddvenk siddvenk merged commit 772f17f into deepjavalibrary:master Feb 6, 2025
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@siddvenk siddvenk siddvenk approved these changes

@zachgk zachgk Awaiting requested review from zachgk zachgk is a code owner

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@a-ys @siddvenk