Fixing tokenizer when transformers is installed without tokenizers

[urialon]

What does this PR do?

This PR fixes the tokenization of the <s> and </s> tokens, when transformers is installed but tokenizers is not installed, fixing the string representation of the AddedToken class.

Using transformers without tokenizers installed results in the following problem:

import transformers
tokenizer = transformers.AutoTokenizer.from_pretrained('facebook/bart-large-mnli')

print(tokenizer.convert_tokens_to_ids(['<s>', 'a', '</s>']))
print(tokenizer.encode('a'))

Prints:

>>> [0, 102, 2]
>>> [50265, 102, 50266]

In other words, the tokenizer knows that the correct IDs for <s> and </s> are 0 and 2, but when encoding an arbitrary string, it adds the new IDs 50265 and 50266 (which are not known to the model!).

Using this solution, the tokenizer does not add additional token IDs 50265 and up, because it recognizes them as existing already in IDs 0-3.
Then, encoding a string using a tokenizer results in adding 0 and 2 as the <s> and </s> tokens.
The two lines of:

print(tokenizer.convert_tokens_to_ids(['<s>', 'a', '</s>']))
print(tokenizer.encode('a'))

result in the same output of [0, 102, 2], as expected.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

Anyone in the community is free to review the PR once the tests have passed. Feel free to tag
members/contributors who may be interested in your PR.

@ArthurZucker @ydshieh @hvaara @amyeroberts

[ArthurZucker]

Hey! Thanks for the catch! 🤗 it seems indeed that the representation does not show this. __repr__ should also be updated to show the content, single_word etc same as if tokenizers was present.

[urialon]

Thanks @ArthurZucker , I added a __repr__ function and repr=True to the dataclass definition.

[ArthurZucker]

Thanks. Let's try to match the fast implementation !

[ArthurZucker]

Nope, let's either take the default repr from the dataclass overload, or when we print the added Token we need to have: AddedToken("▁<PRE>", rstrip=True, lstrip=True, single_word=False, normalized=False, special=True)

[urialon]

Hi @ArthurZucker ,
I reverted the change to the __repr__ function and to the dataclass decorator.

I'm an not sure that the __repr__ behavior is correct. When I load a Bart tokenizer and print it:

>>> tokenizer
BartTokenizerFast(name_or_path='facebook/bart-large-mnli', vocab_size=50265, model_max_length=1024, is_fast=True, padding_side='right', truncation_side='right', special_tokens={'bos_token': '<s>', 'eos_token': '</s>', 'unk_token': '<unk>', 'sep_token': '</s>', 'pad_token': '<pad>', 'cls_token': '<s>', 'mask_token': AddedToken("<mask>", rstrip=False, lstrip=True, single_word=False, normalized=False)}, clean_up_tokenization_spaces=True)

Most of the special tokens are strings rather than AddedTokens.

So, I am keeping only the __str__ function as it fixes an existing bug.
Fixing the __repr__ function is related but not necessarily coupled with this PR.

Thanks,
Uri

[ArthurZucker]

Thanks, I'll open a follow up PR to update the __repr___ then

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint.

[hvaara]

Thanks for contributing this fix back to upstream, @urialon!