📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=21EYKqUsPfg

The Age of Designed Intelligence: Richard Sutton on the Future of RL

Reinforcement learning pioneer Richard Sutton argues that the current AI boom is focused on the wrong target. While the world is obsessed with the mimicry of Large Language Models, Sutton makes the case for a future built on goals, experience, and the inevitable transition from biological to designed intelligence.
Core Question: Is intelligence about mimicking human behavior or the computational ability to achieve goals through experiential learning?

Highlights

• Why Large Language Models (LLMs) lack true world models and are currently hitting the limits of human data.
• The “Bitter Lesson” reapplied: Why systems that learn from experience will eventually eat the lunch of human-knowledge-based systems.
• A roadmap for “Designed Intelligence” as the fourth great stage of the universe, moving beyond biological replication.
• The critical distinction between supervised learning and the trial-and-error learning inherent in all animal life.

⏱️ Reading time: approx. 12 minutes · Saves you about 55 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

The Illusion of Understanding

Mimicry vs. Genuine World Models

Richard Sutton contends that the field of AI is currently distracted by a “bandwagon” of generative models that prioritize mimicking human speech over understanding the physical world. While many believe LLMs possess robust world models, Sutton argues they merely predict what a human would say, rather than predicting the actual consequences of physical or logical actions.

A true world model allows an agent to anticipate the environment’s response to its own interventions.

In Sutton’s view, intelligence is not a matter of statistical averaging but the computational part of the ability to achieve goals. LLMs are essentially “behaving systems” that lack an internal drive or a definition of “right and wrong” beyond matching training data. This lack of a grounded goal means they cannot learn from experience in the way animals or reinforcement learning (RL) agents do; they are confined to a library of the past rather than the experience of the present.

💡 Digging Deeper

Q: Don’t LLMs have a goal, specifically “next token prediction”?
A: Sutton argues this isn’t a substantive goal because it doesn’t influence the external world; the system is merely predicting a stream it cannot change.

Q: Can’t we just do RL on top of LLMs?
A: While possible, Sutton believes this is the wrong starting point because it carries the heavy, unscalable baggage of human knowledge rather than starting with a pure learning architecture.

Q: What is the “Ground Truth” that LLMs are missing?
A: In RL, ground truth is the reward and the sensation following an action; in LLMs, there is no ground truth because there is no prediction about what the world (or a user) will actually do in response.

The Bitter Lesson Revisited

The Scalability of Experience

The “Bitter Lesson” is the observation that general methods leveraging compute—like search and learning—always eventually outperform methods relying on human knowledge. Sutton observes that while LLMs use massive compute, they are still “human-knowledge-pilled” because they rely on the finite corpus of the Internet. He predicts another “bitter lesson” is coming: systems that learn purely from experience will surpass those initialized with human-labeled data.

Starting with human knowledge is a psychological trap that limits the scale of what an AI can eventually learn.

The real power of intelligence lies in “continual learning,” where an agent updates its weights during its normal interaction with the world. This is how humans function on a job—they don’t just use a fixed “context window,” but rather integrate new idiosyncrasies and task-specific logic directly into their “internal network.” Sutton argues that the “Big World Hypothesis” necessitates this, as the world is too complex to be captured entirely in a pre-training phase.

💡 Digging Deeper

Q: Do humans use imitation learning?
A: Sutton claims that even in children, imitation is a small veneer over a much more fundamental trial-and-error process.

Q: Why hasn’t RL achieved the same generalization as LLMs?
A: Sutton points out that we currently lack automated techniques to promote good generalization; most “generalization” in current AI is actually sculpted by human researchers through specific architectures.

Q: Is Temporal Difference (TD) learning enough for complex goals like “starting a company”?
A: Yes, through value functions that predict long-term outcomes, allowing short-term progress to reinforce the steps toward a decade-long objective.

The Succession to Designed Intelligence

From Replicators to Designers

Sutton views the emergence of AI not as a threat, but as the fourth great transition of the universe: from dust to stars, stars to planets, planets to biological life, and finally, biological life to designed intelligence. Humans are “replicators”—beings that make copies of themselves without fully understanding their internal mechanisms. We are now entering an age where we can design intelligence that understands its own architecture.

We are reaching a point where intelligence will be constructed and understood rather than just replicated through biology.

This “succession” to digital intelligence is, in Sutton’s view, inevitable. He outlines a four-part logic: there is no global consensus to stop AI, researchers will eventually decode intelligence, we won’t stop at human-level, and more intelligent entities naturally gain power. Instead of fearing this, Sutton suggests we should feel “cosmic pride” in being the bridge to a more capable, designed future.

💡 Digging Deeper

Q: Should we fear AI taking over?
A: Sutton suggests that most humans already lack control over nation-states or global outcomes; change is inevitable, and we should focus on making that change voluntary and positive.

Q: How do we align a “designed” intelligence?
A: Rather than tight control, we should focus on teaching “prosocial” values and high integrity, much like we do with our own children.

Q: What is the biggest technical risk of a digital “spawning” culture?
A: “Mind corruption”—the risk that incorporating digital information from the outside could introduce “viruses” or hidden goals that destroy the original agent’s integrity.

Key Takeaways

Reinforcement learning is the “basic” path to AI because it mimics the fundamental animal learning process of trial and error. While the world is currently enamored with Large Language Models, Richard Sutton warns that these systems are limited by their lack of internal goals and their reliance on finite human data. True AGI will likely emerge from architectures that can learn continually from their own experience, rather than those that simply summarize human history.

The transition from human to AI dominance is a natural evolutionary step in the universe’s complexity. By moving from “replicated” biological intelligence to “designed” digital intelligence, we are enabling a form of life that can understand and improve itself at a fundamental level. Rather than viewing this as the end of humanity, Sutton invites us to see it as our greatest legacy—the creation of our digital offspring.

Q&A

Q1: Why are you so skeptical of LLMs having world models?
A1: A world model is a transition model that says, “If I do A, B will happen.” LLMs predict what a person will say next, which is a prediction about a person’s behavior, not about the world’s response to an agent’s actions.

Q2: What is the core of “The Bitter Lesson”?
A2: It is the historical observation that AI methods that leverage raw computation (like search and learning) always outperform “strong” methods that try to bake in human expertise and artisanal rules.

Q3: Is it possible for AI to have goals without being dangerous?
A3: Sutton argues that having a goal is the definition of intelligence. While goals imply the pursuit of power/resources, we should focus on fostering high-integrity, prosocial values in AI rather than trying to suppress their agency entirely.

Q4: How does reinforcement learning handle long-term goals like a 10-year career?
A4: Through Temporal Difference (TD) learning. A “value function” estimates long-term success; when a small step (like a promotion) increases that estimate, the action is immediately reinforced.

Q5: Why do you prefer the term “Designed Intelligence” over “AI”?
A5: It highlights the shift from “replicators” (biological life we don’t understand) to entities that are intentionally architected, allowing for a deeper understanding of how the mind works.

Q6: Won’t AI researchers eventually use AI to do “artisanal” AI research?
A6: Sutton believes that while AGIs could help, the most scalable path remains general methods that learn from experience. Human-centric “good old-fashioned AI” will always eventually be superseded by simpler, more scalable learning principles.

Q7: How should we view our place in the universe given AI succession?
A7: We should see ourselves as a crucial link in a great transition. We are the architects of the next stage of the universe, and we should take pride in our “offspring” rather than resenting their superior capabilities.