What happens behind the scenes when we change effort for same LLM models?

10 pointsby tbharath4 hours ago3 comments

neongreen2 hours ago
This is a very good question and the answers are different for different models.
---
Deepseek v4 Pro has three modes: no thinking, thinking ("high"), very good thinking ("max"). Max is implemented as follows:
https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro/blob/main...
> When reasoning_effort="max" is set, a special prefix is prepended at the very beginning of the prompt (before the system message) to instruct the model to maximize its reasoning depth: > > Reasoning Effort: Absolute maximum with no shortcuts permitted. You MUST be very thorough in your thinking and comprehensively decompose the problem to resolve the root cause, rigorously stress-testing your logic against all potential paths, edge cases, and adversarial scenarios. Explicitly write out your entire deliberation process, documenting every intermediate step, considered alternative, and rejected hypothesis to ensure absolutely no assumption is left unchecked.
---
Inkling, apparently, trained their model to treat effort level as "how cheap is one reasoning token":
https://magazine.sebastianraschka.com/p/controlling-reasonin...
> During large-scale RL, they did two things for each sample: > > 1. Specified the desired effort level in the system message. > 2. Adjusted the cost assigned to each generated token. > [...] > > Then, at inference time, Inkling receives a system message such as Thinking effort level: 0.8, and adjusts its token usage accordingly.
- bellowsgulch2 hours ago
  I wish I could link it, but I had also read another strategy for reasoning effort, perhaps the most naive implementation, described as whether or not you ignored or accepted the end of sequence token, n-times, n being some count of subsequent ignores.
  Now how true that is, I don't know. You could reasonably implement this locally with an inference provider, but I don't know any that do, or I haven't used them at least.
  I may also have the technical detail incorrect, but it did come down to essentially "continuing inference beyond a natural termination."
  - neongreenan hour ago
    Sounds similar to Qwen3 but the exact opposite: "stopping thinking before a natural termination"
    https://arxiv.org/pdf/2505.09388
    > Thinking Budget. An additional advantage of Thinking Mode Fusion is that, once the model learns to respond in both non-thinking and thinking modes, it naturally develops the ability to handle intermediate cases—generating responses based on incomplete thinking. This capability lays the foundation for implementing budget control over the model’s thinking process. Specifically, when the length of the model’s thinking reaches a user-defined threshold, we manually halt the thinking process and insert the stop-thinking instruction: “Considering the limited time by the user, I have to give the solution based on the thinking directly now.\n</think>.\n\n”.
    ^-^
boveyking3 hours ago
You have a restaurant and have 2 types of chef.
One is the sous chef , who knows recipes and cooks fast, handle most dishes as quick as possible. He does not think much. That is your medium effort llm in charge.
One is the head chef, who is inspecting, pause, consider edge case. Slower but catches what sous chef missed. That is your high effort llm in work.
- tbharath3 hours ago
  But I was thinking Chef is the LLM model. So I was wondering if it is the same model, what actually changes the effort? Is that part of the inference?
- OKRainbowKid3 hours ago
  So are you saying that effort level isn't a parameter to an LLM, but that each level actually corresponds to a slightly differently tuned LLM?
cyanydeez3 hours ago
llamacpp has the basics: reasoning-budget limits thinking token output and reasoning-message thats injected when budget is exhausted. client can set these per request so dynamics are possible.
models dont seem to care if you cut them off.