Add convert_special_token_settings.py

[hiroshi-matsuda-rit]

@so298 Please review and test in the real model directories. After merging to main branch, please reflect the changes to llm-jp and llm-jp-nvlink clusters.

[so298]

@hiroshi-matsuda-rit

I tried to run the script for some models, but I could not successfully convert to hf models because of len(tokenizer) == model_config["vocab_size"] assertion's failure.

For example:

exp9 model and ver3.0b1 tokenizer

Phase 5: Update model config with Hugging Face Tokenizer special token settings
python convert_special_token_settings.py /model/7B_HF_RE/test_conversion_script/
Traceback (most recent call last):
  File "/model/sosuke/modelwg/convert_special_token_settings.py", line 38, in <module>
    main()
  File "/model/sosuke/modelwg/convert_special_token_settings.py", line 32, in main
    new_model_config = update_special_token_settings(tokenizer, model_config)
  File "/model/sosuke/modelwg/convert_special_token_settings.py", line 8, in update_special_token_settings
    assert len(tokenizer) == model_config["vocab_size"], f'Different vocab size definitions: {len(tokenizer)=} and {model_config["vocab_size"]=}'
AssertionError: Different vocab size definitions: len(tokenizer)=99574 and model_config["vocab_size"]=99584

exp4+ model and ver2.2 50k tokenizer

==========================================
Phase 5: Update model config with Hugging Face Tokenizer special token settings
python convert_special_token_settings.py /model/7B_HF_RE/test_conversion_script_ver2.2_50k/
Traceback (most recent call last):
  File "/model/sosuke/modelwg/convert_special_token_settings.py", line 38, in <module>
    main()
  File "/model/sosuke/modelwg/convert_special_token_settings.py", line 32, in main
    new_model_config = update_special_token_settings(tokenizer, model_config)
  File "/model/sosuke/modelwg/convert_special_token_settings.py", line 8, in update_special_token_settings
    assert len(tokenizer) == model_config["vocab_size"], f'Different vocab size definitions: {len(tokenizer)=} and {model_config["vocab_size"]=}'
AssertionError: Different vocab size definitions: len(tokenizer)=48586 and model_config["vocab_size"]=48640

While reading this issue in huggingface transformers repo, I think tokenizer's vocab size and model's vocab size can be different to make model's vocab size GPU-friendly.
(I have not looked into Megatron-LM's logic to determine vocab_size in model_config.json)

In fact, the values of vocab_size in model_config.json, such as 48640 (= 2^9 * 5 * 19) and 99584 (= 2^8 * 389), can be factored into products of large powers of two and a few prime numbers.

I think this assertion should be removed.

[so298]

https://github.com/llm-jp/Megatron-LM/blob/0f32b38c1235ef6c2272df241c701ca98600144e/megatron/tokenizer/tokenizer.py#L62

The vocab_size used in Megatron-LM may be decided here

[hiroshi-matsuda-rit]

Thanks for reviewing. I just changed the assertion to warning message. @so298

[so298]

I have confirmed that the script works correctly. I will merge it.