Camenbert length Tokenizer not equal config vocab_size

[Keisn1]

❓ Questions & Help

Hi there,
when I load the pretrained Camenbert model and tokenizer via

model = CamembertForMaskedLM.from_pretrained('camembert-base') tokenizer = CamembertTokenizer.from_pretrained('camembert-base')

the length of the tokenizer is 32004 but the vocab_size of the model is 32005.
print(len(tokenizer))
'print(model.config.vocab_size'

This throws me an error

Index out of range

when I try to adapt the lm_finetuning example because of
model.resize_token_embeddings(len(tokenizer))

It runs when I comment out this line. So my question is, is this the intended behaviour resp. what's the reason for the unevenness between the tokenizer and the model vocab_size?

[thomwolf]

Indeed, upon deeper investigation, it appears that the original fairseq model has a bunch of duplicate tokens in the dictionary:

import torch
camembert = torch.hub.load('pytorch/fairseq', 'camembert.v0')
list(camembert.task.source_dictionary[i] for i in range(10))
>>> ['<s>', '<pad>', '</s>', '<unk>', '<unk>', '<s>', '</s>', ',', '▁de', '.']

I'm cleaning and updating for this in #2065

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.