Code

[Whisper] Fix whisper tokenizer#34537

Code

Merged

eustlb merged 27 commits intohuggingface:mainhuggingface/transformers:mainfrom eustlb:fix-whispertokenizereustlb/transformers:fix-whispertokenizerCopy head branch name to clipboardDec 5, 2024

+173-25

Commits

Commits on Oct 31, 2024

• handle single timestamp ending

  eustlb

  committed
  751f88a

  Copy full SHA for 751f88a

• include last timestamp token

  eustlb

  committed
  000dccd

  Copy full SHA for 000dccd

• handle single timestamp ending

  eustlb

  committed
  70c8aac

  Copy full SHA for 70c8aac

• Merge branch 'huggingface:main' into fix-whispertokenizer

  eustlb

  authored
  3f9f7ea

  Copy full SHA for 3f9f7ea

• avoid floating points arithm limitations

  eustlb

  committed
  e753206

  Copy full SHA for e753206

• ensure float64 operations

  eustlb

  committed
  c53fb2c

  Copy full SHA for c53fb2c

• new test

  eustlb

  committed
  185fb55

  Copy full SHA for 185fb55

• make fixup

  eustlb

  committed
  429904c

  Copy full SHA for 429904c

• make copies

  eustlb

  committed
  1c72244

  Copy full SHA for 1c72244

• Merge branch 'main' into fix-whispertokenizer

  eustlb

  authored
  ad4f355

  Copy full SHA for ad4f355

Commits on Nov 2, 2024

• Merge branch 'main' into fix-whispertokenizer

  eustlb

  authored
  09af9de

  Copy full SHA for 09af9de

Commits on Nov 4, 2024

• handle edge case double tokens ending with different tokens

  eustlb

  committed
  7d6f9b4

  Copy full SHA for 7d6f9b4

• handle single timestamp ending

  eustlb

  committed
  937cd2a

  Copy full SHA for 937cd2a

• make fixup

  eustlb

  committed
  9b1d51e

  Copy full SHA for 9b1d51e

Commits on Nov 5, 2024

• handle conditioning on prev segments

  eustlb

  committed
  a3cbe9f

  Copy full SHA for a3cbe9f

• fix

  eustlb

  committed
  7c0da36

  Copy full SHA for 7c0da36

Commits on Nov 21, 2024

• Update src/transformers/models/whisper/generation_whisper.py

  Show description for 4a21249
  

  eustlb

  and

  ylacombe

  authored
  4a21249

  Copy full SHA for 4a21249

Commits on Nov 22, 2024

• Merge branch 'main' into fix-whispertokenizer

  eustlb

  authored
  5b195a8

  Copy full SHA for 5b195a8

Commits on Nov 27, 2024

• [run-slow] whisper

  

  Eustache Le Bihan

  committed
  e8f2f69

  Copy full SHA for e8f2f69

• Merge branch 'main' into fix-whispertokenizer

  eustlb

  authored
  d71d40a

  Copy full SHA for d71d40a

• Merge branch 'main' into fix-whispertokenizer

  eustlb

  authored
  961d5f6

  Copy full SHA for 961d5f6

Commits on Dec 2, 2024

• Merge branch 'main' into fix-whispertokenizer

  eustlb

  authored
  e04aa92

  Copy full SHA for e04aa92

Commits on Dec 4, 2024

• Merge branch 'main' into fix-whispertokenizer

  eustlb

  authored
  69abfe0

  Copy full SHA for 69abfe0

• Merge branch 'main' into fix-whispertokenizer

  eustlb

  authored
  16b1ecb

  Copy full SHA for 16b1ecb

Commits on Dec 5, 2024

• don't call item() to avoid unnecessary sync

  eustlb

  committed
  5fba3e0

  Copy full SHA for 5fba3e0

• Merge branch 'main' into fix-whispertokenizer

  eustlb

  authored
  9624ce9

  Copy full SHA for 9624ce9

• fix

  eustlb

  committed
  88587bb

  Copy full SHA for 88587bb