Preserve thinking
Based on my experience past few days, for a Q4 27B Qwen, preserve thinking is more noise than help. It fills the context and had few instances where qwen started confusing it's plan with the actual file, claiming that I have changed the original file.
I think it would be better off to default to False. I changed it in my template because even if I set it to false in LM studio, sometimes it goes to default.
I agree that is better of, but how thumis template can override a setting that is set by llama.cpp?
Based on my experience past few days, for a Q4 27B Qwen, preserve thinking is more noise than help. It fills the context and had few instances where qwen started confusing it's plan with the actual file, claiming that I have changed the original file.
I think it would be better off to default to False. I changed it in my template because even if I set it to false in LM studio, sometimes it goes to default.
This can happen if you use KV cache quantization like K q4. Jinja templates must not override model settings. Have in mind that there are different 'best' settings for thinking and non-thinking modes.
I use KV quantization but Q8. Preserve thinking has absolutely no benefit , I have 95K context to work on and it fills it so fast it can't complete any work. Maybe is good for bigger model but for Q4 on a limited context it's very bad
Hey @Jokeru1983 ,
I actually tried defaulting preserve_thinking to false back in v20, but had to revert it because of prefix KV caching.
When historical thinking is stripped between turns, the rendered tokens of previous assistant messages change. That drops the Prefix KV Cache hit rate on llama.cpp, vLLM, and LM Studio from 100% down to 0%, forcing the engine to recompute the entire conversation history from token 0 on every turn.
Keeping it true by default guarantees fast generation and 100% cache hits. If you are on low VRAM and need to save memory tokens, you can pass "preserve_thinking": false (or "preserve_reasoning": false) in your kwargs to strip past thoughts.