Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add warning message for run_qa.py #29867

Merged
merged 2 commits into from
Mar 30, 2024
Merged

Add warning message for run_qa.py #29867

Changes from 1 commit
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
dc70540
improve: error message for best model metric
jla524 Mar 26, 2024
f286b1a
update: raise warning instead of error
jla524 Mar 26, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions examples/pytorch/question-answering/run_qa.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -627,6 +627,10 @@ def post_processing_function(examples, features, predictions, stage="eval"):
references = [{"id": str(ex["id"]), "answers": ex[answer_column_name]} for ex in examples]
return EvalPrediction(predictions=formatted_predictions, label_ids=references)

accepted_best_metrics = ("exact" if data_args.version_2_with_negative else "exact_match", "f1")
if training_args.load_best_model_at_end and training_args.metric_for_best_model not in accepted_best_metrics:
raise ValueError(f"--metric_for_best_model must be set to one of {accepted_best_metrics}")

metric = evaluate.load(
"squad_v2" if data_args.version_2_with_negative else "squad", cache_dir=model_args.cache_dir
)
Expand Down