📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=6_BcCthVvb8

Mastering Context Engineering: Building Resilient AI Agents

As AI agents move from simple chat interfaces to long-running autonomous systems, they face a massive “context explosion” that eventually degrades performance. This deep dive explores how the founding engineers at LangChain and Manus manage hundreds of tool calls without losing focus or accuracy through strategic data offloading and compaction.

Core Question: How can developers strategically manage the context window to maintain high performance and avoid “context rot” in complex, long-running agentic workflows?

Highlights

• The distinction between reversible “compaction” and lossy “summarization” in message history.
• Identifying the “Pre-Rot” threshold: Why performance drops significantly at 128k–200k tokens.
• Implementing a three-layered action space to prevent tool-induced context confusion.
• Why “Shared Memory” patterns often outperform traditional sub-agent communication in deep research.

⏱️ Reading time: approx. 8 minutes · Saves you about 53 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

The Rise of Context Engineering

Combatting the Context Explosion

Prompt engineering dominated the post-ChatGPT era, but 2024 has officially become the year of the agent. This shift necessitated a new discipline: context engineering, which focuses on the accumulation of tool observations in long-running loops.

Unlike static prompts, agentic context grows unboundedly, often reaching hundreds of turns in production environments.

While model windows are expanding to millions of tokens, performance rarely stays linear across that span. “Context rot” describes the phenomenon where model accuracy, speed, and instruction-following capabilities drop significantly as the message history balloons. This paradox forces a choice: do we keep feeding the model everything, or do we learn the delicate art of filling the window with only the essential signals required for the next immediate step?

💡 Digging Deeper

Q: When did the term “Context Engineering” actually emerge?
A: It gained significant traction around May 2024, coinciding with the rise of autonomous agents like Devon and the realization that long-running loops require active context management.

Q: Is a larger context window the solution to agent performance?
A: Not necessarily. Even with 1-million-token windows, models suffer from “middle-of-the-context” neglect and increased latency, making engineering more important than raw capacity.

Compaction vs. Summarization

Strategies for Information Density

Manus differentiates between compaction and summarization to preserve critical data while freeing up precious token space for reasoning.

Compaction is a reversible operation where the agent strips out verbose content—like the full body of a 50kb file it just wrote—while keeping the file path.

Because the agent can retrieve that content later via the path if needed, no information is truly lost to the “externalized” state. This prevents the model from hallucinating based on truncated summaries while significantly thinning out the message history. It essentially treats the file system as an extension of the model’s working memory.

Summarization is a more aggressive, non-reversible tactic reserved for when compaction hits its limits. Peak suggests triggering these reductions around 128k to 200k tokens, well before the theoretical million-token limit, to avoid the degradation of model reasoning.

When summarization is necessary, using a structured schema (like a form with specific fields for “Modified Files” and “Current Goals”) is far more effective than asking for a free-form summary.

💡 Digging Deeper

Q: Why summarize at 128k if the model supports 1 million tokens?
A: This is the “Pre-Rot” threshold where latency increases and the model begins to repeat itself or miss subtle instructions buried in the history.

Q: How do you keep the agent from losing its “place” after a summary?
A: Always keep the most recent 3-5 tool calls in full detail. This provides the agent with fresh “few-shot” examples of how to use its tools correctly.

The Layered Action Space

Preventing Context Confusion

To prevent “context confusion,” where a model forgets how to use its tools due to an overcrowded prompt, Manus utilizes a three-layer action space. This keeps the primary function-calling list small and manageable while maintaining the agent’s overall power.

Level one consists of atomic functions like reading files or running shell commands, limited to about 10-20 core tools.

Level two and three offload complexity to the sandbox and external APIs. Instead of defining a unique tool for every possible action, the agent writes Python scripts or uses pre-installed Linux utilities. This approach leverages the model’s inherent coding ability to interact with a vast library of capabilities without cluttering the system prompt with hundreds of individual tool schemas.

This architecture ensures that the KV cache remains stable because the core tool definitions rarely change, even as the agent performs highly complex tasks.

💡 Digging Deeper

Q: How does the agent know which shell tools are available if they aren’t in the prompt?
A: The system prompt includes a compact list of utilities and an instruction that the agent can always run a “help” flag to discover specific parameters.

Q: Why use code instead of individual tool definitions?
A: Code is composable. An agent can perform a sequence of 10 API calls in a single Python script, which is much faster and cheaper than 10 separate model turns.

Isolation and Shared Memory

Architecting Multi-Agent Communication

Multi-agent systems often fail because syncing information between isolated context windows becomes a coordination nightmare for the developer and the model.

Manus borrows from the Go programming language, utilizing two distinct patterns: “communicating” via instructions and “sharing memory” via shared context.

In the shared memory pattern, sub-agents see the full history but operate under a specialized system prompt. This is particularly useful for deep research where the final report depends on a massive trail of intermediate notes. Rather than passing files back and forth, the sub-agent simply “steps into” the existing context to perform a specialized extraction or synthesis task.

To ensure these agents actually collaborate, developers should use schemas as contracts. By enforcing structured outputs through constraint decoding, the main agent can define a specific result format that the sub-agent must adhere to, turning a messy natural language handoff into a reliable programmatic exchange.

💡 Digging Deeper

Q: When should I choose sub-agents over a single agent?
A: Use sub-agents for tasks with a clear, short instruction where only the final output matters, such as searching a codebase for a specific snippet.

Q: Isn’t sharing context expensive?
A: It increases input token costs for the pre-fill, but it often saves tokens in the long run by avoiding redundant data retrieval and multiple “handshake” turns between agents.

Key Takeaways

The transition from simple chat to autonomous agents requires a fundamental shift in how we treat the context window. It is no longer just a “message history”; it is a dynamic workspace that must be aggressively managed to prevent the “context rot” that sets in as histories exceed 100k tokens. By distinguishing between reversible compaction—which externalizes data to a file system—and lossy summarization, developers can maintain high reasoning quality over long durations.

Ultimately, the most successful agent architectures prioritize simplicity over complexity. peak emphasizes that Manus became more stable and capable when they removed unnecessary scaffolding and “anthropomorphic” roles (like “Designer Agent” or “Manager Agent”).

Instead of building complex organizational charts, focus on building robust sandboxes, structured communication contracts via schemas, and a layered action space that allows the agent to solve its own problems through code and existing system utilities.

Q&A

Q1: How does Manus handle long-term memory across different sessions?
A: Manus uses “Explicit Knowledge,” where the agent proposes information to be saved (like a preference for Excel over CSV). The user must confirm these “memory blocks” before they are persisted across sessions.

Q2: Do you recommend using vector databases for agent context?
A: For short-term sandbox tasks, simple file search tools like grep and glob are often faster and more reliable. Vector stores are better suited for massive, long-term enterprise knowledge bases that exceed what can be fit in a sandbox.

Q3: What is the biggest mistake people make with multi-agent setups?
A: Trying to mirror human organizational charts. Role-based agents (Designer, Coder, Reviewer) often waste tokens on unnecessary communication and coordination.

Q4: How do you handle planning in long tasks?
A: Manus moved away from “to-do.md” files because updating them wasted too many tokens. They now use a specialized “Planner Agent” that manages the task state through a structured tool interface.

Q5: Is fine-tuning or RL better than context engineering?
A: Not currently for most startups. Context engineering is the clearest boundary between the application and the model. If you fine-tune for a specific action space and that space changes (like with the launch of MCP), your model becomes obsolete instantly.

Q6: What file formats are best for agent context?
A: Line-based plain text is preferred over Markdown. It allows the agent to use standard Linux tools to read specific line ranges and prevents the model from over-using bullet points or complex formatting.

Q7: How do you evaluate agent performance if benchmarks are saturated?
A: Manus relies on three pillars: average user star ratings per session, internal automated tests with verifiable execution results, and human interns who judge the “taste” and visual appeal of outputs.