Skip to content

Commit

Permalink
Add group act order case to lm_eval test (#1080)
Browse files Browse the repository at this point in the history
SUMMARY:
- Requires next transformers release/ when the following commit is
released: huggingface/transformers#35615
- Adds group act order case which previously could not be loaded through
the AutoModel pathway due to incorrect substring replacement of the
`weight_g_idx` parameter

Testing:
- Test passes locally with transformers 4.49
  • Loading branch information
dsikka authored Feb 21, 2025
1 parent ff007b6 commit 5a82dcc
Show file tree
Hide file tree
Showing 2 changed files with 12 additions and 1 deletion.
11 changes: 11 additions & 0 deletions tests/e2e/vLLM/lm_eval_configs/w4a16_actorder_group.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,11 @@
cadence: "weekly"
model: meta-llama/Meta-Llama-3-8B-Instruct
recipe: tests/e2e/vLLM/recipes/actorder/recipe_w4a16_actorder_group.yaml
num_fewshot: 5
limit: 1000
task: "gsm8k"
dataset_id: HuggingFaceH4/ultrachat_200k
dataset_split: train_sft
exact_match,flexible-extract: 0.72
exact_match,strict-match: 0.72
scheme: W4A16_actorder_group
2 changes: 1 addition & 1 deletion tests/e2e/vLLM/lm_eval_configs/w4a16_actorder_weight.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -8,4 +8,4 @@ dataset_id: HuggingFaceH4/ultrachat_200k
dataset_split: train_sft
exact_match,flexible-extract: 0.72
exact_match,strict-match: 0.72
scheme: W4A16_actorder_group
scheme: W4A16_actorder_weight

0 comments on commit 5a82dcc

Please sign in to comment.