AudioClassificationPipelineTests::test_small_model_pt_fp16 fails for CUDA/XPU but passes for CPU

[dvrogozh]

With:

• 92c5ca9

On:

• Nvidia A10
• Intel Data Center GPU Max (PVC)

The commit f19135a introduced AudioClassificationPipelineTests::test_small_model_pt_fp16 test which passes on CPU, but fails if running on CUDA or XPU. PR was:

• fix low-precision audio classification pipeline #35435

Logs (for cuda):

python3 -m pytest tests/pipelines/test_pipelines_audio_classification.py::AudioClassificationPipelineTests::test_small_model_pt_fp16
===================================== test session starts ======================================
platform linux -- Python 3.10.12, pytest-7.4.4, pluggy-1.5.0
rootdir: /home/dvrogozh/git/huggingface/transformers
configfile: pyproject.toml
plugins: rich-0.2.0, xdist-3.6.1, asyncio-0.23.8, timeout-2.3.1
asyncio: mode=strict
collected 1 item

tests/pipelines/test_pipelines_audio_classification.py::AudioClassificationPipelineTests::test_small_model_pt_fp16 FAILED [100%]

=========================================== FAILURES ===========================================
__________________ AudioClassificationPipelineTests.test_small_model_pt_fp16 ___________________

self = <tests.pipelines.test_pipelines_audio_classification.AudioClassificationPipelineTests testMethod=test_small_model_pt_fp16>

    @require_torch
    def test_small_model_pt_fp16(self):
        model = "anton-l/wav2vec2-random-tiny-classifier"

        audio_classifier = pipeline("audio-classification", model=model, torch_dtype=torch.float16)

        audio = np.ones((8000,))
        output = audio_classifier(audio, top_k=4)

        EXPECTED_OUTPUT = [
            {"score": 0.0839, "label": "no"},
            {"score": 0.0837, "label": "go"},
            {"score": 0.0836, "label": "yes"},
            {"score": 0.0835, "label": "right"},
        ]
        EXPECTED_OUTPUT_PT_2 = [
            {"score": 0.0845, "label": "stop"},
            {"score": 0.0844, "label": "on"},
            {"score": 0.0841, "label": "right"},
            {"score": 0.0834, "label": "left"},
        ]
>       self.assertIn(nested_simplify(output, decimals=4), [EXPECTED_OUTPUT, EXPECTED_OUTPUT_PT_2])
E       AssertionError: [{'score': 0.0833, 'label': 'go'}, {'score': 0.0833, 'label': 'off'}, {'score': 0.0833, 'label': 'stop'}, {'score': 0.0833, 'label': 'on'}] not found in [[{'score': 0.0839, 'label': 'no'}, {'score': 0.0837, 'label': 'go'}, {'score': 0.0836, 'label': 'yes'}, {'score': 0.0835, 'label': 'right'}], [{'score': 0.0845, 'label': 'stop'}, {'score': 0.0844, 'label': 'on'}, {'score': 0.0841, 'label': 'right'}, {'score': 0.0834, 'label': 'left'}]]

tests/pipelines/test_pipelines_audio_classification.py:159: AssertionError
------------------------------------- Captured stderr call -------------------------------------
Device set to use cuda:0
======================================= warnings summary =======================================
tests/pipelines/test_pipelines_audio_classification.py::AudioClassificationPipelineTests::test_small_model_pt_fp16
  /home/dvrogozh/git/huggingface/transformers/src/transformers/configuration_utils.py:315: UserWarning: Passing `gradient_checkpointing` to a config initialization is deprecated and will be removed in v5 Transformers. Using `model.gradient_checkpointing_enable()` instead, or if you are using the `Trainer` API, pass `gradient_checkpointing=True` in your `TrainingArguments`.
    warnings.warn(

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html
=================================== short test summary info ====================================
FAILED tests/pipelines/test_pipelines_audio_classification.py::AudioClassificationPipelineTests::test_small_model_pt_fp16 - AssertionError: [{'score': 0.0833, 'label': 'go'}, {'score': 0.0833, 'label': 'off'}, {'sco...
================================= 1 failed, 1 warning in 1.99s =================================

Note: for XPU (upstream pytorch XPU, not IPEX) log is actually the same including same scores for labels

Does CUDA/XPU work correctly in this test? (I am confused seeing same scores 0.0833 for all 4 labels).

Overall, expectation is that test passes for CUDA/XPU or excluded for pytorch device backends if it's CPU specific.

CC: @jiqing-feng @ydshieh

[Ajinkya-25]

Hello,
I think this issue occurs because CUDA/XPU computation may lead to numerical instability when using torch.float16 due to lower precision compared to float32 or CPU computations as scores for every label is same i.e 0.833 try using mixed precision instead of torch.float16 or torch.float32.
you can also try using another model for prediction as "anton-l/wav2vec2-random-tiny-classifier" might not be optimized for torch.float16

[jiqing-feng]

Hi @dvrogozh . Thanks for your issue, could you please try this PR: 36359 to check if it fix your issue?

[ydshieh]

It's indeed failing on our T4 runners. Thank you for pointing this out 👍