Fix bug about add_special_tokens and so on#31496

Merged

amyeroberts merged 3 commits intohuggingface:mainfrom hiroshi-matsuda-rit:fix-bug-about-add_special_tokens

Jun 24, 2024

+16-3

Commits on Jun 19, 2024

fix bug about add_special_tokens and so on
hiroshi-matsuda-rit
authored

Commits on Jun 20, 2024

improve add_special_tokens and padding behavior
hiroshi-matsuda-rit
authored

Commits on Jun 22, 2024

add a test case for add_special_tokens and padding
hiroshi-matsuda-rit
committed