Support GQA export, better run.c, Support tinyllama-1.1B

[magician-blue]

Add support to tinyllama-1.1B
Add support to convert GQA model (learned from ggerganov/llama.cpp#3364)
Better run.c

• save a little memory(same as calculate key and value inside the kv cache #400)
• make rope part a function
• hardcode to check whether there is a \n

[xefoci7612]

Current chat schemas in run.c are based on LLama 2

            // render user/system prompts into the Llama 2 Chat schema
            if (pos == 0 && system_prompt[0] != '\0') {
                char system_template[] = "[INST] <<SYS>>\n%s\n<</SYS>>\n\n%s [/INST]";
                sprintf(rendered_prompt, system_template, system_prompt, user_prompt);
            } else {
                char user_template[] = "[INST] %s [/INST]";
                sprintf(rendered_prompt, user_template, user_prompt);
            }

But you may want to use tinyllama's ones instead:

<|im_start|>user
Explain huggingface.<|im_end|>
<|im_start|>assistant

In general chat templates should be bounded to the loaded pre-trained model, so maybe they should be a configuration parameter in the .bin file

[karpathy]

?

[magician-blue]

For MHA model, the number of kv heads equals q heads.
However, for GQA model like llama2-70b, tinyllama1.1B, the number of kv heads and q head are different.

[karpathy]

why is this delta here done?

[magician-blue]

I'm not sure whether I convert the tokenizer correctly. After I convert the tinyllama-1.1B's tokenizer. The run.c gets <0x0A> instead of \n. I'm trying to figure out how to convert the tokenizer better to remove these line.
Besides, I notice that our run.c can not deal with \n in the input (for tinystory 260k,15m,110m model). They will treat it as \\ and n.
In llama.cpp, they hardcode to convert \\n to \n.

[karpathy]

This is cool, I wasn't aware of the TinyLlama 1.1B run. Sounds very nice and useful for this repo to support.
Are there any notable architectural changes in it?
This PR is a bit of a random combination of necessary differences, and a few side optimizations.

[magician-blue]

This is cool, I wasn't aware of the TinyLlama 1.1B run. Sounds very nice and useful for this repo to support. Are there any notable architectural changes in it? This PR is a bit of a random combination of necessary differences, and a few side optimizations.

There isn't notable architectural changes.