Yarn rope settings only appear in deepseek and phi3 code, any reason?

[bartowski1182]

Was looking through convert_hf_to_gguf.py, particularly at Yarn rope scaling settings, and I can only find references to it in Deepseek and Phi3 code.. I know Qwen2.5 uses Yarn to extend context (but not by default on their main uploads), and was wondering - if one added the yarn settings to config.json and extended the context to 128k, this would not get saved into the GGUF metadata right? Is there any reason for that?

Could it be added more universally?

Tagging @slaren (sorry for ping, but you're the most likely to have an answer)

[slaren]

I am sorry, I don't know what's the logic for adding rope scaling settings or why it is not done more generally for all the models. @Galunid or @compilade may have the answer.

[bartowski1182]

thanks for passing it to appropriate parties, appreciate it :)

[robbiemu]

at the rather severe risk of exposing my ignorance, let me try to help a bit. :)

tl-dr;

I think we can just take the llamaforcausallm type (or whichever type it currently is) and add duplicate it for qwenforcausallm, using deepseek's approach without the extra variable (or setting it to 1.0).

details

Alibaba's team added the note:

For supported frameworks, you could add the following to config.json to enable YaRN:

{
  ...,
  "rope_scaling": {
    "factor": 4.0,
    "original_max_position_embeddings": 32768,
    "type": "yarn"
  }
}

For deployment, we recommend using vLLM. Please refer to our Documentation for usage if you are not familar with vLLM. Presently, vLLM only supports static YARN, which means the scaling factor remains constant regardless of input length, potentially impacting performance on shorter texts. We advise adding the rope_scaling configuration only when processing long contexts is required.

Structurally, this is essentially like Deepseek v2 without the n-d magnitude scaling adjustment: #7416 Deepseek's attention mechanism is unique so they found a better value for their architecture. The default qwen would use is 0.1 https://github.com/jquesnelle/yarn/blob/995db5b575e75230b3384d658f8b944c9662f775/scaled_rope/modeling_llama_together_yarn.py#L114

But the warning the alibaba team gives at the end is perhaps based on intuition from their paper: https://arxiv.org/html/2407.10671v4#S2.SS2.SSS1

To expand the context window of Qwen2, we implement Dual Chunk Attention (DCA, An et al., 2024), which segments long sequences into chunks of manageable lengths. If the input can be handled in a chunk, DCA produces the same result as the original attention. Otherwise, DCA facilitates effective capture of relative positional information between tokens within and across chunks, thereby improving long context performance. Moreover, we also employ YARN (Peng et al., 2023) to rescale the attention weights for better length extrapolation.

Moreover, we follow Qwen with the usage of SwiGLU (Dauphin et al., 2017) for activation, Rotary Positional Embeddings (RoPE, Su et al., 2024) for positional embedding, QKV bias (Su, 2023) for attention, RMSNorm (Jiang et al., 2023b) and pre-normalization for training stability.

I think applying YaRN before reaching original context length just degrades even more with DCA. Maybe similar concerns were the reasoning behind Phi's method with variable scaling factors for different length inputs, but in Qwen's case they just live with the loss.

refs: https://medium.com/@rcrajatchawla/understanding-yarn-extending-context-window-of-llms-3f21e3522465

[ggerganov]

Reading about DCA, it sounds a lot like Self-Extend. These approaches do help to extend the context, but make the KV cache management a bit painful and we lose the ability to do things like context-prefix reuse and more advanced context reuse via RoPE shifts.

YARN-only should allow context-prefix reuse to function as usual, though RoPE shifts are likely incompatible (not sure).

[robbiemu]

@bartowski1182 I'm happy to share that Qwen's use of yarn for 128k context is now supported. #10698