Context is not free: how adding information silently shortens LLM reasoning

“Give a student the textbook during the exam. They stop deriving answers and start looking them up. They also stop checking their work.”

TL;DR

Adding context to a reasoning model’s prompt silently shortens its reasoning trace by up to 50% and reduces self-verification behaviors like double-checking (arXiv 2604.01161). Tested across multiple reasoning models including Qwen3.5-27B, GPT-OSS-120B, and Gemini 3 Flash Preview. Every RAG chunk, system prompt instruction, and tool output competes for the same budget as reasoning. Context is not free — it costs reasoning depth. For how this relates to the broader question of whether chain-of-thought reflects real reasoning, see is chain-of-thought a mirage.

What does “reasoning shift” actually mean?

When a reasoning model receives a problem with no context, it generates a long chain of thought — working through the problem step by step, checking intermediate results, backtracking when something looks wrong. Add context to the same problem — retrieved documents, system prompt instructions, prior conversation turns — and the reasoning trace shrinks. Not because the model got more efficient. Because it skipped steps.

arXiv 2604.01161 (Rodionov, 2026) tested this across multiple reasoning models including Qwen3.5-27B, GPT-OSS-120B, Gemini 3 Flash Preview, and Kimi K2 Thinking. Under six different context conditions — irrelevant padding, multi-turn conversations, subtask embedding, long reference material, needle-in-haystack, and in-context learning — reasoning traces shortened by up to 50%.

The self-verification component matters most. Models didn’t just write fewer tokens. They dropped specific behaviors: double-checking intermediate results, expressing uncertainty, and re-examining assumptions. The quality of reasoning degraded, not just the quantity.

Why does context crowd out reasoning?

Two mechanisms produce the shortening effect.

Retrieval replaces derivation. When relevant information sits in the context window, models switch from reasoning to pattern-matching. Instead of deriving an answer through multi-step logic, the model locates the answer (or something close) in the provided context and reports it. This is faster and shorter. It also skips the verification steps that catch errors. “When More is Less” (arXiv 2502.07266, Wu et al.) formalized this: task accuracy follows an inverted U-curve with chain-of-thought length. Optimal reasoning length increases with task difficulty but decreases with model capability — stronger models reason more efficiently, but context-induced shortcuts bypass even that efficiency.

Token budget competition. A model’s output budget is finite. Every token spent processing and responding to context is a token unavailable for reasoning. In production systems, system prompts alone can consume 500-2,000 tokens. RAG pipelines inject 3,000-10,000 tokens of retrieved content. By the time the model begins reasoning, a substantial fraction of its effective budget is already spent. The reasoning trace compresses to fit what remains.

graph LR
    subgraph "Without context"
        P1[Problem] --> R1["Reasoning trace<br/>(full depth)"]
        R1 --> V1["Self-verification<br/>(double-check)"]
        V1 --> A1[Answer]
    end

    subgraph "With context (RAG + system prompt)"
        SP[System prompt<br/>500-2K tokens] --> C[Context window]
        RAG[RAG chunks<br/>3-10K tokens] --> C
        P2[Problem] --> C
        C --> R2["Reasoning trace<br/>(shortened ~50%)"]
        R2 --> A2["Answer<br/>(no verification)"]
    end

Where does this hit hardest in production?

Four pipeline patterns trigger the worst degradation.

RAG with over-retrieval. Research on long-context RAG (arXiv 2410.05983) confirms the problem at scale: performance initially improves with more retrieved passages, then declines. Adding a sixth or seventh chunk to help the model often hurts it. The model spends tokens reconciling multiple sources instead of reasoning about the question. Aggressive reranking and top-2 or top-3 retrieval often outperform top-10.

Verbose system prompts. Every instruction in your system prompt is context that competes with reasoning. A 1,500-token system prompt describing output format, personality, safety rules, and few-shot examples may be displacing the model’s ability to think through the actual problem. I’ve seen teams add instructions to their system prompt without measuring the reasoning quality impact.

Multi-turn conversations. Independent tasks in conversation history still consume context. If turn 3 is a math problem and turns 1-2 were about scheduling, those scheduling turns are noise that the model processes and that reasoning must work around. Session isolation or context trimming between independent tasks preserves reasoning depth.

Subtask embedding. When a problem appears as a subtask within a larger task, models treat it as lower priority. The reasoning trace for a subtask embedded in a complex workflow can be half the length of the same problem posed in isolation. Routing subtasks to separate model calls preserves full reasoning depth.

How do you preserve reasoning depth?

Three mitigation strategies, ordered by implementation effort.

Strategy 1: Minimize context aggressively. Send only the top 2-3 most relevant retrieved chunks, not everything the retriever found. Trim conversation history to the current task. Strip system prompt to essentials. Every token of context you remove is a token returned to reasoning. This is the cheapest fix and often the most effective.

Strategy 2: Separate retrieval from reasoning. Run retrieval as a first model call that extracts relevant facts into a concise summary. Pass only that summary — not the raw documents — to a second model call focused on reasoning. This prevents the reasoning model from spending tokens on document comprehension. The two-call pattern costs more in API calls but produces deeper reasoning per call.

Strategy 3: Set thinking token budgets. NVIDIA NIM and similar inference frameworks now support minimum reasoning token allocations (arXiv 2412.18547). Set a floor below which the reasoning trace cannot compress, regardless of context size. BudgetThinker (arXiv 2508.17196) takes this further with control tokens that inform the model of its remaining reasoning budget during generation. OptimalThinkingBench (arXiv 2508.13141) shows even the best models — o3 at 71.1% on the unified metric — fail at optimal token allocation, making external budget control necessary.

Strategy	Cost	Effort	Reasoning preservation
Minimize context	Free	Low	Moderate
Separate retrieval from reasoning	2x API calls	Medium	High
Thinking token budgets	Framework support needed	High	Highest

Key takeaways

• Context is not free. Every token of context displaces a token of reasoning. The trade-off is invisible but measurable.
• Up to 50% shorter reasoning traces. Adding context causes models to skip steps, drop verification, and produce shallower chains of thought.
• Self-verification suffers most. Models stop double-checking and expressing uncertainty — the quality indicators that catch errors.
• RAG over-retrieval actively hurts. Performance follows an inverted U-curve with context volume. Top-3 often beats top-10.
• Minimize, separate, budget. Send less context, separate retrieval from reasoning, and set token budget floors.

Further reading

• Is chain-of-thought a mirage? — why CoT reflects memorized patterns, not abstract reasoning
• When LLMs stop talking to themselves — latent reasoning as an alternative to visible chain-of-thought
• Multi-step reasoning — reasoning architectures for agent systems

Want to work together?

I take on projects, advisory roles, and fractional CTO engagements in AI/ML. I also help businesses go AI-native with agentic workflows and agent orchestration.

Get in touch