llama : refactor unicode stuff

[ggerganov]

ref #5981

• add unicode.cpp and move codepoint tables there
• rename unicode-related functions to have prefix unicode_
• make a thin API in unicode.h

Build time when touching llama.cpp source file is now improved by ~25%

[ggerganov]

Whenever I save the unicode.h (and now unicode.cpp) file with my text editor (Neovim) it stops working on Windows:

https://github.com/ggerganov/llama.cpp/actions/runs/8230676605/job/22504415172?pr=5992#step:3:91

Does anyone have an idea how to make vim save it in the proper format?
It should prefix the file with some special bytes - I look about this sometime ago, and forgot the specifics. Any ideas how to resolve this would be appreciated, because otherwise I cannot work on this file without breaking the Windows builds

[slaren]

I don't know about VIM, but the prefix that you mention is called BOM (byte order mark). It should have an option to save with BOM.

[ggerganov]

Thanks - the command is :set bomb