📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=AZrU6y3pUcU

The End of the Benchmark Grid: Why AI Intelligence is Now a Function of Budget

For years, AI evaluation relied on static grids comparing one model to another via a single performance score. OpenAI researcher Noam Brown argues this approach is now obsolete because a model’s true capability is no longer a fixed number—it is a scaling function of how much “thinking time” or compute budget a user is willing to allocate.

Core Question: How should we evaluate AI when performance is determined by test-time compute budget rather than just pre-training scale?

Highlights

• The traditional “benchmark grid” fails to show that newer models are often just more efficient “thinkers” rather than marginally better performers.
• Modern models can productively “reason” for weeks or months on a single problem, making standard evaluation cycles significantly harder to verify.
• Current safety policies and preparedness frameworks lack critical accounting for large-scale inference budgets, creating a dangerous blind spot.
• Recursive self-improvement is currently bottlenecked by human research taste and the temporal cost of running long-horizon experiments.

⏱️ Reading time: approx. 8 minutes · Saves you about 28 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

The Broken State of AI Evaluations

The Deception of the Grid

The industry is currently stuck in a bad equilibrium where every lab publishes a static grid of benchmark scores to prove their model’s superiority.

Noam Brown notes that during the GPT-3 era, models plateaued quickly; giving a 2020-era model a $10 million inference budget wouldn’t have yielded significantly better answers. Today, the landscape has shifted entirely because performance on complex reasoning tasks is now a direct function of the compute budget allocated at inference time, meaning a model given $1,000 to think will drastically outperform the same model given $1.

When newer models are released, initial skepticism often arises because benchmark improvements look incremental compared to predecessors. However, these grids fail to control for the “thinking time” used; newer iterations are often simply more efficient, reaching higher accuracy levels with significantly less compute than older versions, a nuance lost in a single-number score.

💡 Digging Deeper

Q: Why not just run models until their performance plateaus?
A: Because the plateau for modern models is now weeks or even months away, making it impractical for standard release cycles.

Q: What is the proposed solution for researchers and labs?
A: Models should be evaluated using an X-axis that measures tokens, cost, or time, rather than a single static Y-axis point.

Scaling Inference for Hard Reasoning

From Poker Bots to Scientific Discovery

Brown reflects on his PhD work creating poker solvers, noting that while early models struggled with basic game mechanics, current frontier models can optimize complex algorithms by 10x or 100x efficiency.

He recently used an internal model to disprove the Erdős unit distance conjecture, a feat achieved at a surprisingly low budget through iterative prompting.

The fascinating part of this mathematical breakthrough is that the capability was already latent in existing models, but it required a specific “scaffold” or steering mechanism to unlock. Had a researcher been willing to spend $100,000 in inference costs on a general-purpose scaffold months ago, they likely could have achieved the same result before the newer, specialized models were even trained. This suggests that the ceiling for current AI is much higher than we realize; we simply aren’t spending enough time or money on each individual query to see what the models are truly capable of accomplishing.

The Safety and Policy Blind Spot

Budget-Based Risk Assessment

Preparedness frameworks and responsible scaling policies were largely designed in an era before test-time compute scaling became a dominant paradigm.

If a model’s dangerous capabilities—such as the ability to assist in creating bioweapons—scale with its inference budget, then a safety evaluation performed on a $10 budget is meaningless for a bad actor willing to spend $1 million. We are entering a world where “capability” is no longer a fixed trait of the weights but a dynamic variable controlled by the user’s wallet and patience, yet current policies largely ignore this inconvenient truth.

This creates a tension in the model release cycle, where labs are pressured to release models quickly even though they haven’t had the time to fully explore the capabilities that might emerge after weeks of continuous model “thinking.”

💡 Digging Deeper

Q: Is there an “intelligence explosion” imminent?
A: Brown is skeptical of an overnight explosion because even if models become smarter, they are still bottlenecked by the physical time required to run long-horizon experiments.

Q: What is the biggest bottleneck for AI labs today?
A: Time itself remains the primary constraint, as researchers are limited by how fast they can iterate and evaluate new architectures.

Key Takeaways

The core shift in AI is the transition from model-centric intelligence to budget-centric intelligence. We must stop asking “how smart is this model” and start asking “how smart is this model given a specific amount of compute.” Evaluation methods must evolve to include a cost or token X-axis to reflect real-world utility and efficiency gains.

While recursive self-improvement is accelerating research, the “human in the loop” remains vital.

Models still lack the specific “research taste” required to identify which novel algorithmic directions are worth pursuing. The future of the frontier labs will be defined by who can most effectively balance rapid scaling with the temporal limitations of rigorous evaluation. As we move forward, the ability of models to coordinate, share knowledge, and build on accumulated findings will likely be the next major frontier, moving beyond short context windows toward a more organic, emerging property of collective machine intelligence.

Q&A

Q1: What happens if you give a model a month to think?
A: It can solve problems like entire PhD-level poker solvers, but the industry hasn’t explored these limits because release cycles are so fast.

Q2: Can inference scaling solve everything?
A: No; factual retrieval (like birth dates) doesn’t improve with more thinking time if the data isn’t in the weights.

Q3: Is routing between models the answer to efficiency?
A: It can help, but it’s still subject to the same compute-budget constraints and must be measured against a single-model scaling baseline to be proven effective.

Q4: How does Noam Brown use AI personally?
A: For high-stakes decisions like tax advice and legal paperwork, noting he trusts it more than many human experts.

Q5: Why is the industry stuck on the “grid” evaluation format?
A: It’s a bad equilibrium caused by inertia; everyone expects a grid because that’s what has always been published, even if researchers know it’s flawed.

Q6: What is the significance of the Erdős conjecture proof?
A: It proved that frontier models can solve open mathematical problems if you steer them through the right reasoning paths, even without specialized training.

Q7: Does Noam Brown believe in a “hard takeoff”?
A: Not in the sense of an overnight explosion, primarily because the scaling of intelligence via compute is still constrained by the linear progression of time.