Fix device for OpenVINO Runtime in whisper evaluator (#3994)

* Fix device format for OpenVINO Runtime * Update README.md with whisper evaluator
openvinotoolkit · Jan 13, 2025 · 668e0b0 · 668e0b0
1 parent 272ffd6
commit 668e0b0
Show file tree

Hide file tree

Showing 2 changed files with 10 additions and 5 deletions.
diff --git a/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/README.md b/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/README.md
@@ -54,10 +54,15 @@ Optionally you can provide `module_config` section which contains config for cus
   Evaluator code: `<omz_dir>/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/sr_evaluator.py`.
 
 * **Tacotron2 Evaluator** demonstrates how to evaluate custom Tacotron2 model for text to speech task.
-  Evaluator code: `<omz_dir>/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/tacotron2_evaluator.py`
+  Evaluator code: `<omz_dir>/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/tacotron2_evaluator.py`.
 
 * **Open-NMT Evaluator** demonstrates how to evaluate Open-NMT model for text translation task.
-  Evaluator code: `<omz_dir>/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/opennmt_encoder_decoder_generator_evaluator.py`
+  Evaluator code: `<omz_dir>/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/opennmt_encoder_decoder_generator_evaluator.py`.
 
 * **StableDiffusion Evaluator** demonstrates how to evaluate image generation for Stable Diffusion models family.
-  Evaluator code: `<omz_dir>/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/stable_diffusion_evaluator.py`
+  Evaluator code: `<omz_dir>/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/stable_diffusion_evaluator.py`.
+
+* **Whisper Evaluator** demonstrates how to evaluate Whisper family models with various pipeline classes, including GenAIWhisperPipeline, HFWhisperPipeline, and OptimumWhisperPipeline.
+  Evaluator code: `<omz_dir>/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/whisper_evaluator.py`. 
+
+
diff --git a/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/whisper_evaluator.py b/tools/accuracy_checker/accuracy_checker/evaluators/custom_evaluators/whisper_evaluator.py
@@ -1,5 +1,5 @@
 """
-Copyright (c) 2024 Intel Corporation
+Copyright (c) 2024-2025 Intel Corporation
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.
@@ -54,7 +54,7 @@ def from_configs(cls, config, delayed_model_loading=False, orig_config=None):
         dataset_config = config["datasets"]
         pipeline_class_name = config["pipeline_class"]
         if 'device' in config['launchers'][0]:
-            config["_device"] = config['launchers'][0]['device']
+            config["_device"] = config['launchers'][0]['device'].upper()
 
         if pipeline_class_name not in cls.VALID_PIPELINE_CLASSES:
             raise ValueError(f"Invalid pipeline class name: {pipeline_class_name}. "