📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=jyuyY86GJnA

Beyond the Token Burn: Mastering “Value Maxing” with GPT-3.5

For too long, AI success was measured by the sheer volume of tokens consumed, leading to “token maxing” behaviors that inflated costs without guaranteeing results. In the era of GPT-3.5, the paradigm shifts toward “value maxing,” where intelligence per dollar and meaningful business outcomes dictate the architecture of next-generation applications.

Core Question: How can startups transition from measuring usage volume to maximizing the ROI of every token using OpenAI’s latest models and efficiency-focused API features?

Highlights

• The transition from “token maxing” (vanity metrics) to “value maxing” (outcome-driven intelligence).
• Choosing between GPT-3.5 Soul, Terra, and Luna based on the “Bang-for-Token” ratio.
• Advanced API optimizations including programmatic tool calling, prompt caching, and context compaction.
• Real-world insights from Ploy on achieving an 89% token reduction through strategic KV caching and breakpoints.

⏱️ Reading time: approx. 8 minutes · Saves you about 46 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

The Evolution of Value: From Consumption to Outcomes

Token maxing emerged as a vanity metric where progress was measured by how many prompts were sent or how many agents were managed simultaneously.

Earlier this year, the sentiment shifted as companies realized they were burning annual AI budgets in months. Enterprises began throttling spend, forcing a pivot toward measuring employees on actual value generated—such as time saved or code quality improved—rather than the volume of data processed through Large Language Models.

Value maxing asks a simple but difficult question: If you doubled your token spend tomorrow, how would you know if the investment was worth it? This involves identifying specific outcomes to improve, mapping the workflows that lead to quality artifacts, and integrating intelligence at the most impactful points. Research and engineering are now driven by rigorous evals, which define exactly what “good” looks like for autonomous systems responsible for end-to-end outcomes.

💡 Digging Deeper

Q: Is value maxing always about spending fewer tokens?
A: Not necessarily. It is about strategic spend. You might choose to use more tokens on a higher reasoning model to ensure a high-stakes output is bulletproof, effectively trading token cost for risk mitigation.

Q: How do “evals” fit into this strategy?
A: Evals are the benchmark for “good.” Without them, you cannot determine if a cheaper model or a more compact prompt is maintaining the quality level required for your specific business case.

The GPT-3.5 Ecosystem: Soul, Terra, and Luna

The release of the GPT-3.5 family offers a tiered approach to intelligence, allowing developers to match model capability to the complexity of the task.

Soul serves as the flagship model for the most complex coding and professional reasoning tasks, while Terra acts as a reliable “daily driver” for balanced workloads. Luna rounds out the family, optimized specifically for high-volume, low-latency tasks where the reasoning requirements are less demanding but cost efficiency is paramount.

OpenAI’s internal benchmarks, such as Deep Suey, demonstrate that the 5.6 model family provides the highest “bang-for-token” currently available in the market. Developers are finding that by utilizing different reasoning levels—Standard, Pro, Max, and Ultra—they can fine-tune performance. For example, a model with lower baseline intelligence can often achieve flagship-level results if the developer is willing to spend more tokens on extended reasoning or chain-of-thought processing.

💡 Digging Deeper

Q: What is the best starting point for a new coding project?
A: Start with Soul on a “Medium” reasoning level. It provides a sophisticated baseline for most professional tasks without the high token overhead of the “Extra High” or “Max” modes.

Q: When should I use the “Ultra” reasoning effort?
A: Ultra is designed for heavy-hitter use cases that require the model to natively spin up and manage multiple sub-agents. It is extremely token-intensive and should be reserved for tasks where a single agent cannot maintain the necessary state.

Architecting for Efficiency: API Strategies

Modern API features like programmatic tool calling represent a massive leap in how agents interact with external data and logic.

Programmatic tool calling gives the model access to a JavaScript sandbox where it can execute code to perform calculations or call multiple tools simultaneously. This architectural shift moves logic out of the model’s expensive chain-of-thought and into a deterministic environment, saving both tokens and time by reducing the number of round trips between the user and the API.

Prompt caching, specifically KV (Key-Value) caching, allows the system to store embeddings from prior input tokens, leading to a 10x reduction in cost for repeated context. To maximize this, developers must ensure their context window remains “append-only.” Modifying a system prompt or injecting dynamic data like the current date at the beginning of a prompt will break the cache, forcing the model to re-process every token from scratch.

💡 Digging Deeper

Q: How does context compaction work in long conversations?
A: Compaction takes a long chat history and encodes it into a condensed representation. While it takes time to process (approx. 30 seconds in some cases), it can reduce input tokens by over 80% for subsequent turns.

Q: Where should I put dynamic information like the “current date”?
A: Always append dynamic information to the very end of your prompt. This allows the massive, static block of your system prompt and tool schemas to remain cached and cost-effective.

Case Study: How Ploy Migrated to GPT-3.5 Soul

Ploy, a marketing platform for autonomous growth, migrated their production agents to GPT-3.5 Soul to leverage its superior orchestration and efficiency.

The Ploy team implemented “breakpoints” within their prompt architecture to ensure that the system prompt and core tool schemas remained cached across every single chat. By separating “always-on” tools from “on-demand” tools—which are only loaded at the end of the context when needed—they avoided cache invalidation while maintaining a robust feature set.

Beyond caching, Ploy optimized their tools to perform multiple actions in a single call, such as reading two files at once rather than making two separate requests. They also utilized “highlights” from search providers to slim down raw HTML outputs by 70%. These combined strategies resulted in an 89% reduction in the cost of the first message in new chats and an estimated $37,000 in annual savings.

Key Takeaways

The transition to value maxing is as much a shift in mindset as it is a technical implementation. By focusing on outcomes—such as the number of bugs fixed or leads qualified—startups can move away from the anxiety of token burn and toward a sustainable growth model. The GPT-3.5 family, particularly with Soul, provides the necessary reasoning depth to handle complex orchestration while Luna ensures that high-volume tasks remain economically viable.

Architecturally, the “low-hanging fruit” for optimization includes enabling prompt caching through careful breakpoint placement and adopting programmatic tool calling to reduce model turns. Developers should audit their traces to identify bloated tool responses and use compaction for long-running sessions. As demonstrated by Ploy, small adjustments to how tools are batched and how context is structured can lead to massive cumulative savings and improved user responsiveness.

Q&A

Q1: What is the difference between “Max” reasoning and “Ultra” mode?
A1: Max reasoning allows the model to use as much reasoning as it needs without a “leash,” whereas Ultra is a specific mode that pushes the model to actively use sub-agents to solve problems, making it highly parallel but very token-intensive.

Q2: Are sub-agents given the full context of the main chat?
A2: Generally, no. Sub-agents are dispatched with the specific context needed for their task to maintain efficiency, though a “side chat” in Codex functions differently by maintaining the full thread context.

Q3: Can overly relying on caching be a downside?
A3: Caching is powerful, but it can make your application brittle if you don’t have a clear grip on which parts are static. You should only introduce caching once your prompt structure is stable.

Q4: How does agent depth (nesting sub-agents) affect performance?
A4: Performance tends to degrade beyond two layers of delegation due to context loss. Sub-agents are most effective when work can be parallelized in independent lanes rather than deep, sequential hierarchies.

Q5: Should I start a new chat or use compaction for a long project?
A5: New chats are better if you have a massive amount of “initial” context (like many plugins or files) that isn’t relevant to the new task. Compaction is better for maintaining the continuity of a long-running work session.

Q6: Is programmatic tool calling available on all models?
A6: It was introduced specifically with the 5.6 family (Soul, Terra, Luna) to leverage the new JavaScript sandbox capabilities natively trained into these models.

Q7: How much faster is “Fast Mode” in Codex?
A7: Fast Mode runs outputs at approximately 1.5x speed, though it consumes usage limits faster. It is ideal for developers who prioritize their own time over token volume.