feat: Include HF files such as config.json etc part of recovered checkpoint

[kmehant]

Should we copy the following set of files from the source/base checkpoint to the recovered checkpoint? So that the downstream users (eval) can easily consume the checkpoint upon recovery?

1. config.json
2. generation_config.json
3. merges.txt
4. special_tokens_map.json
5. tokenizer.json
6. tokenizer_config.json
7. vocab.json

[kmehant]

@fabianlim @willmj

[fabianlim]

im not sure this is needed because it can be easily scripted

[kmehant]

We should decide on automating this somewhere may be in fms-hf-tuning controllable through a flag. We can save both dct and hf format safetensor checkpoints together. Since users were expecting a consumable model for the checkpoints flused out by our stack. WDYT?

[fabianlim]

@willmj maybe you can consult @Ssukriti or @anhuong on this since you guys are more familiar with the users.