You’ve spent hours tweaking a prompt. You’ve added role instructions, few-shot examples, chain-of-thought nudges. The output got better — and then it stopped getting better. You hit a ceiling that no amount of “think step by step” can break through.
That ceiling is real, and it has a name now. The AI engineering world has been shifting its attention from prompt engineering to something broader: context engineering. Gartner made it official in mid-2025 when they declared “context engineering is in, prompt engineering is out.” And based on what I’ve seen building agent-based systems over the past year, they’re right — though the relationship between the two is more nuanced than a simple replacement.
The prompt is just one line in the script
Prompt engineering is about crafting the instruction you send to the model. It’s the question you ask, the role you assign, the format you request. And it works. For single-turn tasks — summarize this, rewrite that, extract these fields — a well-crafted prompt can get you 90% of the way there.
But here’s what prompt engineering doesn’t control: what the model knows when it reads your prompt.
Think about it this way. If you ask a brilliant consultant a question, the quality of their answer depends on two things: how clearly you asked (the prompt) and what background information they have access to (the context). You can ask the clearest question in the world, but if the consultant hasn’t seen the relevant data, the answer will be generic at best.
That’s the gap context engineering fills. It’s not about what you say to the model. It’s about what the model has in front of it when you say it.
What context engineering actually means
Context engineering is the discipline of designing and managing the full information environment that surrounds an LLM at inference time. Anthropic’s engineering team defines it as providing “the right information and tools, in the right format, at the right time” so the model can accomplish a task.
That environment has multiple layers, and most of them have nothing to do with your prompt:
System prompt — The persistent instructions that shape model behavior. This is where prompt engineering and context engineering overlap. But a system prompt in a context-engineered system does more than set tone. It defines available tools, establishes output schemas, and encodes domain-specific rules.
Retrieved documents (RAG) — External knowledge pulled in at query time from vector databases, knowledge graphs, or search indices. This is the biggest lever most teams underuse. A well-designed RAG pipeline can improve LLM accuracy by 40-70% compared to relying on the model’s training data alone.
Memory — Persistent or episodic information carried across interactions. This could be user preferences from previous sessions, summaries of past conversations, or accumulated facts about a project. Without memory, every interaction starts from zero.
Tool definitions and results — The functions an agent can call and the data those functions return. When your model can query a database, check a calendar, or call an API, the results of those calls become part of the context. Tool design is context engineering.
Conversation history — The sliding window of prior messages. How you manage, compress, and summarize this history directly affects output quality, especially in long-running agent sessions.
Prompt engineering touches the first layer. Context engineering architects all five.
Why the shift is happening now
Three things converged to make context engineering the dominant frame in 2026.
Context windows got massive. Claude now handles up to 1 million tokens. Gemini offers similar capacity. GPT-5’s context window expanded significantly. When you can fit an entire codebase or a hundred-page document into a single request, the question stops being “how do I phrase this?” and becomes “what should I include?”
Agents made context the bottleneck. Single-turn chatbot interactions reward prompt engineering. But agentic systems — where models plan, use tools, iterate, and maintain state across dozens of steps — live or die by context management. An agent that can’t retrieve the right information, track its own history, or know which tools are available will fail no matter how perfect its system prompt is. Gartner projects that 40% of enterprise applications will feature task-specific AI agents by late 2026, up from under 5% in 2025. All of them need context engineering.
Diminishing returns on prompt tricks. Models have gotten smart enough that elaborate prompt gymnastics matter less than they used to. Claude, GPT-5, and Gemini all handle ambiguity, follow complex instructions, and reason through problems far better than their predecessors. The marginal gain from optimizing prompt wording keeps shrinking. The marginal gain from providing better context keeps growing.
Where prompt engineering is still enough
I don’t want to overstate the case. Prompt engineering isn’t dead — it’s just not sufficient on its own for complex systems.
For plenty of use cases, a good prompt is all you need:
- One-shot transformations. Reformat this JSON. Translate this paragraph. Summarize these notes. The model has everything it needs in the input; no external context required.
- Creative generation with clear constraints. Write a product description in this tone, at this length, for this audience. The prompt is the context.
- Classification and extraction. Given this text, categorize it. Pull out the email addresses. The task is self-contained.
If your input contains all the information the model needs to produce the output, prompt engineering is the right tool. Where it breaks down is when the model needs information that isn’t in the prompt — and that covers most real-world applications beyond simple text processing.
Five context engineering patterns that actually work
Enough theory. Here’s what context engineering looks like in practice.
1. RAG with re-ranking, not just retrieval
Most RAG implementations I see do the bare minimum: embed the query, find the nearest vectors, stuff them into the prompt. This works okay. It could work much better.
A proper RAG pipeline has three stages. Pre-retrieval: clean, chunk, and index your data thoughtfully. Chunk size matters more than people think — too small and you lose coherence, too large and you dilute relevance. In-retrieval: use hybrid search (combining semantic and keyword search) and then re-rank results before they hit the context window. Post-retrieval: assemble the context with source attribution and relevance ordering so the model can prioritize.
The difference between naive RAG and a well-engineered pipeline is dramatic. I’ve seen accuracy jump from roughly 60% to over 85% on domain-specific QA tasks just by adding re-ranking and hybrid search.
2. Structured memory injection
For agents that run across multiple sessions, memory is non-negotiable. But “memory” doesn’t mean dumping every past interaction into the context window.
Effective memory systems use tiered storage. Short-term memory (current conversation, compressed periodically) stays in the context window. Long-term memory (user preferences, project facts, past decisions) lives in a database and gets retrieved selectively based on relevance to the current task.
The key insight: memory retrieval is itself a context engineering problem. You need to decide not just what to remember, but when to recall it and how to present it. A memory entry that says “User prefers Python over JavaScript” is more useful when the model is about to suggest a code solution than when it’s drafting an email.
3. Dynamic system prompts
Static system prompts are a missed opportunity. In a context-engineered system, the system prompt changes based on the task, the user, and the current state of the workflow.
Here’s a concrete example. Say you’re building a customer support agent. Instead of one monolithic system prompt, you build a system prompt assembler that includes: base behavior rules (always applicable), product knowledge relevant to the detected issue category, the customer’s account history and past tickets, available tools for this issue type (refund tools for billing issues, technical tools for bug reports), and escalation criteria specific to this tier of customer.
Each of those components is a context layer that gets composed dynamically. The model sees a different system prompt depending on who’s asking and what they’re asking about.
4. Tool result filtering
When an agent calls a tool — say, a database query — the raw result often contains far more information than the model needs. Dumping a 500-row query result into the context window is wasteful and confusing.
Context engineering means designing a filter layer between tool execution and context insertion. Summarize large results. Extract only the fields the model needs. Add metadata about what was filtered out so the model knows it can request more detail if needed.
This sounds simple, but I’ve seen it cut token usage by 60% while actually improving output quality because the model isn’t distracted by irrelevant data.
5. Context window budgeting
With million-token context windows, it’s tempting to include everything. Don’t. More context isn’t always better context. Models can get lost in large context windows, a phenomenon researchers call “lost in the middle” — information in the middle of very long contexts gets less attention than information at the beginning or end.
Budget your context window deliberately. Allocate portions for each context layer (system prompt, memory, RAG results, tool outputs, conversation history) and set hard limits. If your RAG retrieval returns 20 relevant documents, you probably want the top 5, well-ordered, not all 20 crammed in.
The tools and protocols making this easier
The ecosystem around context engineering is maturing fast.
Model Context Protocol (MCP) is the most significant development. Introduced by Anthropic in late 2024 and donated to the Linux Foundation in December 2025, MCP standardizes how LLMs connect to external tools and data sources. Its three primitives — tools, resources, and prompts — give you a consistent interface for plugging context into any model. There are already thousands of MCP servers available for popular services like GitHub, Slack, Postgres, and dozens more.
That said, MCP is still young. The protocol relies on stateful sessions, which creates challenges for scaling across multiple instances. And there have been recent security concerns — researchers flagged a design vulnerability affecting MCP servers in April 2026 — so evaluate carefully before deploying in production.
LlamaIndex has become the go-to framework for the retrieval and indexing side of context engineering. It handles chunking strategies, embedding, vector storage, re-ranking, and context window assembly with minimal overhead (around 6ms of framework overhead per call in benchmarks). If your context engineering challenge is primarily about getting the right documents to the model, LlamaIndex is where I’d start.
LangGraph (LangChain’s evolution for production agent work) handles the orchestration side — managing agent state, routing between tools, and coordinating multi-step workflows. It’s heavier than LlamaIndex but more powerful for complex agentic systems.
Many production stacks in 2026 use both: LlamaIndex as the knowledge layer, LangGraph as the orchestration layer. Some add n8n or similar workflow tools on top for business logic.
Anthropic’s own guidance on context engineering for agents is worth reading directly. Their engineering blog has a detailed guide covering instruction hierarchy, tool design, and context management patterns specifically for Claude-based agents.
How to start if you’re still just prompting
If you’re currently getting by with prompt engineering alone, here’s a practical path into context engineering.
Start with RAG. Pick one use case where your model needs external knowledge — product docs, internal wikis, customer data — and build a basic retrieval pipeline. Even naive RAG (embed, retrieve, stuff) will show you the difference that external context makes. Then iterate: add re-ranking, experiment with chunk sizes, try hybrid search.
Add memory to one agent. If you have any agent or chatbot that handles multi-turn conversations, add a simple memory layer. Store key facts from each session and retrieve them at the start of the next one. You’ll be surprised how much this changes the user experience.
Make your system prompt dynamic. Take your current static system prompt and identify which parts could change based on the user or task. Build a simple template system that composes the prompt from components. This is low-effort, high-impact context engineering.
Audit your context window. For your most important LLM application, log the full context that gets sent to the model. Look at it critically. How much of it is actually useful? What’s missing that would help? What’s there that’s just noise? This audit alone will reveal optimization opportunities.
The shift from prompt engineering to context engineering isn’t about abandoning what you know. Your prompting skills still matter — they’re just one layer in a larger system. The engineers who thrive in 2026’s AI landscape are the ones who think beyond the prompt and design the entire information environment their models operate in.
That’s a harder problem than tweaking a few words. It’s also a much more interesting one.