Debugging model output in llama-server

[Mushoz]

When running aider's benchmark, COT models tend to get stuck at 100% load indefinitely. I am assuming these models get stuck in some infinite reasoning loops, but I am not entirely sure if that's truly the case. Perhaps it's not stuck, but just taking too long for the 1 hour hardcoded timeout to hit.

Is there some way I can see what llama-server is generating? Aider's benchmark runs with streaming disabled as it doesn't show much to the user. So I cannot inspect what is being generated. I was hoping there is some way to see what is being generated in llama-server directly.

[ggerganov]

Add -lv 1.

Most likely your are using a small context size (the default is -c 4096). Try increasing it.

[Mushoz]

This works, thanks! I was already using 32k context with context shift disabled. It isn't able to fill up the entire context size within the 1 hour timeout due to the model slowing down massively as the context increases.