[Ollama] certain models are not loaded with correct n_ctx

[blakkd]

I have 2 daily driver models: qwen2.5-coder:32b-instruct-q4_K_M and qwq:32b-preview-q4_K_M.
Both are used in other applications.

The issue is that the Ollama num_ctx parameter is not always respected.
I don't know what triggers this behavior.

~ ❯❯❯ ollama show --modelfile qwq:32b-preview-q4_K_M_16k_flash_fullgpu_step_0.4                (base) 
# Modelfile generated by "ollama show"
# To build a new Modelfile based on this, replace FROM with:
# FROM qwq:32b-preview-q4_K_M_16k_flash_fullgpu_step_0.4

[...]

SYSTEM You should think step by step.
PARAMETER num_gpu 65
PARAMETER stop <|im_start|>
PARAMETER stop <|im_end|>
PARAMETER temperature 0.4
PARAMETER num_ctx 16384

Loads correctly in 16384 (ollama serve logs)

[...]
llama_new_context_with_model: n_ctx         = 16384
[...]

BUT

~ ❯❯❯ ollama show --modelfile qwen2.5-coder:32b-instruct-q4_K_M_16k_flash_fullgpu_0.6          (base) 
# Modelfile generated by "ollama show"
# To build a new Modelfile based on this, replace FROM with:
# FROM qwen2.5-coder:32b-instruct-q4_K_M_16k_flash_fullgpu_0.6

[...]

PARAMETER mirostat 0
PARAMETER num_ctx 16384
PARAMETER num_gpu 65
PARAMETER temperature 0.6

Loads in 8192 instead (ollama serve logs)

[...]
llama_new_context_with_model: n_ctx         = 8192
[...]

[lee88688]

Aider can config context. but this plugin doesn't give this to user. so I think this may cause this issue. does this influence on your result?

[blakkd]

Yes I think this is the cause cause I just noticed I had the same issue with aider alone.
Same, specifically with qwen2.5-32b-coder, I don't know what's going on.
That said, in aider yes, setting num_ctx in .aider.model.settings.yml works as intended.

- name: aider/extra_params
  extra_params:
    extra_headers:
      Custom-Header: value
    num_ctx: 16384

This correctly results in all my models being loaded with a 16384 token window.

So yeah, we should have a way to set it up in aider-composer too to have better control.

[blakkd]

And thanks for such a quick response!

[lee88688]

Yes I think this is the cause cause I just noticed I had the same issue with aider alone. Same, specifically with qwen2.5-32b-coder, I don't know what's going on. That said, in aider yes, setting num_ctx in .aider.model.settings.yml works as intended.

- name: aider/extra_params
  extra_params:
    extra_headers:
      Custom-Header: value
    num_ctx: 16384

This correctly results in all my models being loaded with a 16384 token window.

So yeah, we should have a way to set it up in aider-composer too to have better control.

this is a good advise. but currently I am working on other features. and how to design the settings may be a big problem.
by the way, is there other ways to solve this issue?

[blakkd]

Are you foreseeing problems because of the fact there is 2 separated models?
What about adding a global num_ctx field (which would set it for any model) eg. in-between Model and API Key in the sidebar? Or hidden in vscode settings, but maybe that's a bit dirty.
I unfortunately don't see other options. That said, hopefully only few models might be affected, so I think there is no urge ;)

[lee88688]

thanks, since this may not be urgent, It has more time. maybe others have a better solution.

[blakkd]

Maybe Ive been a bit hasty closing this, I'll let this up to you

[blakkd]

@lee88688 btw are you reachable somewhere through PM? Ive a little something I wanna share related to your project

[lee88688]

@blakkd you can contact me with my email in my profile.

[blakkd]

Sorry mate didn't think about the email way lol I was just wondering if you saw that Aider now have a native IDE integration? I'm you asking this because I just stumbled on it yesterday! It appears to me like a new feature but I didn't check how long it's been implemented for. So I was thinking maybe you were already aware of this but had further goals and deliberately decided to keep your project going, but afterward worried you weren't and in this case it I thought was fair to inform you just in case. That said, I've haven't been trying your project for more than 5min and so couldn't really compare the feature of your project to their in term of functionalities. I'm only reaching you in good will. I just felt like a duty to relay the info.I hope you already knew it not to bring a bad news! All the best!

…

On Thursday, January 9th, 2025 at 12:10 PM, Jimmie Lee ***@***.***> wrote: ***@***.***(https://github.com/blakkd) you can contact me with my email in my profile. — Reply to this email directly, [view it on GitHub](#58 (comment)), or [unsubscribe](https://github.com/notifications/unsubscribe-auth/BJEKR2YN2327M64MTVE5W632JZKMBAVCNFSM6AAAAABUUJIEZ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDKNZZHA3DQNRTG4). You are receiving this because you were mentioned.Message ID: ***@***.***>