grammars : add English-only grammar

[ggerganov]

Example:

# without grammar
./bin/llama-server -m ../models/qwen2.5-32b-instruct/ggml-model-q8_0.gguf

curl --request POST --url http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer no-key" -d "$(jq -n '{ messages: [{ role: "system", content: "You are a helpful assistant." }, { role: "user", content: "How to say \"Good morning\" in Chinese?" }], "top_k": 1 }')" | jq -r .choices[0].message.content

# To say "Good morning" in Chinese, you can say:
# 
# **早晨好** (zǎo chén hǎo)
# 
# or more commonly:
# 
# **早上好** (zǎo shàng hǎo)
# 
# Both phrases are used to greet someone in the morning.

# with English-only grammar
./bin/llama-server -m ../models/qwen2.5-32b-instruct/ggml-model-q8_0.gguf --grammar-file ../grammars/english.gbnf

curl --request POST --url http://localhost:8080/v1/chat/completions -H "Content-Type: application/json" -H "Authorization: Bearer no-key" -d "$(jq -n '{ messages: [{ role: "system", content: "You are a helpful assistant." }, { role: "user", content: "How to say \"Good morning\" in Chinese?" }], "top_k": 1 }')" | jq -r .choices[0].message.content

# To say "Good morning" in Chinese, you can say " mornings" which is pronounced as "zao3 shang4" (the numbers represent the tone).

[jukofyork]

I'm on my phone now and the Unicode codepage characters doesn't seem to be showing up:

--grammar "root ::= [^一-鿿]*"

See: https://old.reddit.com/r/LocalLLaMA/comments/1cdj2uo/qwen15110b_the_first_100b_model_of_the_qwen15/

--grammar "root ::= [^一-鿿ぁ-ゟァ-ヿ가-힣]*" 

See: https://old.reddit.com/r/LocalLLaMA/comments/1bwjmoo/qwen1532b_released_with_gqa/

But is there a way to block ranges of Unicode characters like these examples? The new QwQ model seems to randomly switch to using Cyrillic and Arabic characters too?!

[ggerganov]

I'm not sure. This PR is mainly to demonstrate that the grammar works and can be used in such ways. If it does not support the specific case that you need, add a feature request and someone might implement it.

[jukofyork]

I'm not sure. This PR is mainly to demonstrate that the grammar works and can be used in such ways. If it does not support the specific case that you need, add a feature request and someone might implement it.

No problem - I'll do some experiments with the ^ option when I get back.

[ExtReMLapin]

People should keep in mind that they need to adjust the prompt when using this kind of restrictive grammar.

Because the LLM might "advance" into a direction now knowing it will not be able to output some tokens and it will lead to sub-optimal outpute.

A solution would also be to use beam search (batched beam search maybe)

[jukofyork]

I've found a problem with this as it doesn't allow the output of double newlines (which a lot of models love to use).

This is what I am using now with QwQ:

root        ::= (en-char | whitespace)*
en-char     ::= letter | digit | punctuation
letter      ::= [a-zA-Z]
digit       ::= [0-9]
punctuation ::= [!"#$%&'()*+,-./:;<=>?@[\\\]^_`{|}~]
whitespace  ::= [ \t\n]

and even when asked "to think in Chinese" it works.:

User

What is the sqrt(12654) think in Chinese.

QwQ

sqrt(12654) in Chinese is "12654" " " (square root). So, I need to find the square root of 12654. Let's see, I don't know the exact value off the top of my head, so I'll have to calculate it.

First, I should check if 12654 is a perfect square. I know that perfect squares are numbers like 1, 4, 9, 16, and so on, where they are the square of an integer. Let me see, what's the square of 100? That's 10,000. The square of 110 is 12,100, and the square of 112 is 12,544. Hmm, 12,544 is less than 12,654. What about 113 squared? Let's calculate that.

113 times 113. Well, 110 times 110 is 12,100, and then add 113 times 3, which is 339, so 12,100 plus 339 is 12,439. Wait, that doesn't seem right. Actually, 113 times 113 should be calculated differently. Let me try again.

I can use the formula (a + b)^2 = a^2 + 2ab + b^2, where a is 110 and b is 3. So, 113^2 = 110^2 + 21103 + 3^2 = 12,100 + 660 + 9 = 12,769. Okay, so 113 squared is 12,769, which is higher than 12,654. So, the square root of 12,654 should be between 112 and 113.

Let me try 112.5 squared to get a better estimate. 112.5 squared is 112.5 times 112.5. Let's calculate that.

112.5 times 110 is 12,375, and then add 112.5 times 2.5, which is 281.25, so total is 12,375 plus 281.25, which is 12,656.25. Okay, so 112.5 squared is 12,656.25, which is very close to 12,654.

So, the square root of 12,654 should be slightly less than 112.5. Maybe around 112.49 or something like that. To get a more precise value, I would need to use a calculator or a more detailed method like the long division method for square roots.

But for now, I can say that the square root of 12,654 is approximately 112.49.

Final Answer

[ \boxed{112.49} ]

[jukofyork]

User

Can you teach me some Chinese characters?

QwQ

Of course! Learning Chinese characters can be fun and rewarding. Let's start with some basic characters to get you familiar with the structure and principles behind them.

First, it's important to understand that Chinese characters, also known as Hanzi, are logograms, meaning each character represents a word or a morpheme (a meaningful unit of language). They are not purely phonetic, so learning them involves understanding both their meaning and pronunciation.

Let's begin with some fundamental characters that are commonly used in everyday language.

1. Person ( person )

   • Pinyin: REN2 (pronounced "ruhn")

   • Meaning: person, people

   • Stroke order: Start with a horizontal stroke, then a vertical stroke, and finally another horizontal stroke.

     • Horizontal (---)

     • Vertical (|)

     • Horizontal (---)

   • Example usage: people (REN2 MIN2)

2. Mountain ( mountain )

   • Pinyin: SHAN1 (pronounced "shahn")

   • Meaning: mountain

   • Stroke order: Three horizontal strokes with vertical strokes in between.

     • Horizontal (---)

     • Vertical then horizontal (| ---)

     • Vertical then horizontal (| ---)

   • Example usage: mountain (SHAN1)

3. Tree ( tree )

   • Pinyin: MU4 (pronounced "muh")

   • Meaning: tree, wood

   • Stroke order: Start with a slanted stroke, then a vertical stroke, and finish with a horizontal stroke on the bottom.

     • Slanted stroke (/)

     • Vertical stroke (|)

     • Horizontal stroke (---)

   • Example usage: forest (LIN2)

4. Sun ( sun )

   • Pinyin: RI4 (pronounced "ree")

   • Meaning: sun, day

   • Stroke order: Start with an outer square, then a horizontal stroke in the middle, and finally a horizontal stroke on the bottom.

     • Start with a horizontal stroke (---), then a vertical stroke (|), then another horizontal stroke (---) to close the top.

     • Horizontal stroke in the middle (---)

     • Horizontal stroke on the bottom (---)

   • Example usage: today (JIN1TIAN1)