Fix Llama 3 TikToken conversion

[pcuenca]

What does this PR do?

Fixes a tokenizer conversion problem for Llama 3, possibly introduced in #31656

How to reproduce:

The following fails in main and works with this PR:

python src/transformers/models/llama/convert_llama_weights_to_hf.py \
--input_dir path_to_original_checkpoint \
--model_size tokenizer_only \
--output_dir converted_tokenizer \
--llama_version 3.1

The same issue may happen with Gemma models as well. I didn't test yet, but can verify once we agree on a suitable approach.

Before submitting

• This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
• Did you read the contributor guideline,
  Pull Request section?
• Was this discussed/approved via a Github issue or the forum? Please add a link
  to it if that's the case.
• Did you make sure to update the documentation with your changes? Here are the
  documentation guidelines, and
  here are tips on formatting docstrings.
• Did you write any new necessary tests?

Who can review?

@itazap @ArthurZucker

[pcuenca]

Line 348 below could possibly be removed, as the superclass will handle it.

[itazap]

Yes agreed it can be removed! Thanks

[ArthurZucker]

Good catch and indeed since now it is natively supported let's use super's functionalities!

[HuggingFaceDocBuilderDev]

The docs for this PR live here. All of your documentation changes will be reflected on that endpoint. The docs are available until 30 days after the last update.