📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=Ucqfb33GJJ4

Beyond the Policy: Mapping Intelligence, Agency, and Energy-Based Models

AI is evolving from simple input-output mapping to sophisticated systems that simulate internal worlds. This discussion explores the mathematical foundations of agency, the efficiency of energy-based architectures, and why the future of machine intelligence depends on modularity and continual learning rather than brute-force prediction.

Core Question: How can we build artificial agents that possess true internal agency and safely discover new scientific knowledge?

Highlights

• Agency is defined by the sophistication of internal planning and counterfactual reasoning, not just external behavior.
• Energy-Based Models (EBMs) provide a robust inductive prior by optimizing internal latent states alongside network weights.
• The evolution of the brain’s olfactory cortex provides a blueprint for how associative processing leads to higher-level intelligence.
• Safe AI alignment is best achieved by perturbing human steady-state distributions rather than specifying naive, absolute goals.

⏱️ Reading time: approx. 7 minutes · Saves you about 40 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

The Geometry of Agency

Defining the Internal World

Geometric deep learning is fundamental to modeling the physical world because it incorporates the inherent symmetries—like translation and rotation—that define our reality. For a mathematician, building these symmetries directly into a model is far more efficient than forcing a network to discover them through brute force, as it aligns the architecture with the actual laws of physics.

If we view an agent simply as something that executes a policy, then even a rock could be considered an agent, as it has a consistent input-output relationship with its environment.

However, true agency requires a difference in degree and sophistication. We distinguish ourselves from simple organisms or objects through planning, counterfactual reasoning, and goal-oriented behavior. These are not just external actions; they are latent variables representing internal “rollouts” of future consequences. From the outside, it is incredibly difficult to distinguish between a sophisticated function and a true agent without “cracking it open” to see if it is actually performing these internal simulations.

💡 Digging Deeper

Q: How can we mathematically measure the “strength” of an agent?
A: We can use notions like transfer entropy to estimate the timetable over which a system incorporates information and the degree to which it exhibits context-dependent behavior.

Q: Is agency dependent on being physically embedded in the world?
A: While a model of agency can run in a computer simulation, a true agent arguably requires physical embodiment to interact with and be constrained by the causal laws of the environment.

Q: Can a simple input-output mapping ever be a “true” agent?
A: Philosophically, it feels insufficient. If you don’t know the internal mechanism, you might conclude something is an agent based on its sophistication, but true agency implies the specific process of searching through possibilities.

Energy-Based Models and Inductive Priors

Beyond Feed-Forward Networks

Traditional neural networks are often treated as simple function approximators mapping $X$ to $Y$, where the only things being optimized are the weights. Energy-based models (EBMs) introduce a second layer of optimization: they operate on the internal states of the model itself. In this framework, you are not just finding the best weights; you are finding the “energetic minimum” of the internal hidden nodes to make a prediction.

The Variational Autoencoder (VAE) is perhaps the most common example of an EBM in use today.

In a VAE, the cost function isn’t just about the difference between the input and the reconstructed output; it also penalizes the internal representation if it deviates from a specific distribution, such as a Gaussian. This creates a constraint—an inductive prior—that forces the model to learn a more structured and useful representation of the data.

The Pitfalls of Test-Time Training

Current trends in deep learning are moving toward “test-time training,” where a model is allowed to update some of its weights when it encounters a new sample at inference time. While this mimics the energy-based approach of treating weights as latent variables, it can be unwise if the original network wasn’t trained with that flexibility in mind. If you train a network in a purely supervised way and then suddenly turn on learning during deployment, you risk a mismatch between the model’s original logic and its new updates.

JEPA and the Path to Scientific Discovery

The Power of Latent Prediction

Yann LeCun’s Joint Embedding Prediction Architecture (JEPA) represents a shift away from generative models that try to predict every single pixel in an image. Instead, JEPA focuses on “gestalt” understanding by compressing both inputs and outputs into a latent space and performing predictions there. This mimics the scientific process: we don’t try to model every atom; we look for the compressed, conceptual laws that govern the system.

A major challenge in this approach is avoiding “motor collapse,” where the model finds a shortcut by making all embeddings zero to achieve perfect prediction.

To solve this, researchers use non-contrastive learning, which regularizes the latent space without needing the computationally expensive “negative samples” required by older methods. This allows the model to maintain a high-fidelity representation of the data while discarding task-irrelevant noise, which is exactly how the brain decides what information is worth processing.

Automating the Lab

The next milestone for AI in science is not just finding correlations in data, but actual experimental design. We are moving toward systems that can identify gaps in their own knowledge and autonomously design experiments to fill them. While this sounds like a sci-fi scenario, it is a logical extension of active inference.

A robot encountering a new object, like a beach ball, shouldn’t just stop; it should perform “non-invasive procedures”—poking it, moving it—to identify its properties. This online testing of theories is the hallmark of true intelligence, though we must ensure these systems operate within safety constraints to avoid “optimal” but destructive experiments.

Key Takeaways

The transition from predictive models to agentic models requires a shift in how we view internal states. Agency is not a binary trait but a spectrum of computational sophistication, measured by a system’s ability to roll out counterfactual futures and act on them. By using Energy-Based Models and Joint Embedding architectures, we can create systems that are more efficient, more robust, and more aligned with the conceptual way humans understand the world.

As we look toward the future, the goal is not to create a singular “Artificial General Intelligence” that replaces us, but a collection of specialized intelligences that act as partners. By using techniques like Maximum Entropy Inverse Reinforcement Learning, we can ensure these systems stay aligned with human values by mimicking our steady-state distributions rather than following rigid, potentially dangerous commands.

Q&A

Q1: Why does PCA (Principal Component Analysis) sometimes fail with neural data?
A1: PCA throws away dimensions with low variability, assuming they are noise. However, in neural data, the dimensions with the least variability are often the most important for the actual computation.

Q2: What is the main difference between minimizing Energy and minimizing Free Energy?
A2: Free energy includes an additional entropy penalty term, which acts as a regularization factor. Minimizing pure energy is more akin to simple maximum likelihood estimation.

Q3: How does the olfactory cortex relate to high-level intelligence?
A3: The olfactory system deals with highly combinatorial, non-smooth data. The brain structures evolved to solve these complex associative problems are argued to be the precursors to the frontal cortex.

Q4: Is the fear of AI “Skynet” scenarios justified?
A4: It is less likely that AI will go rogue independently and more likely that a human will naively specify a dangerous goal. The risk lies in the objective function, not the AI’s “will.”

Q5: What is the “Tractor Analogy” for AI?
A5: Just as tractors replaced 98% of agricultural labor and allowed humans to pursue more interesting work, AI is expected to alleviate labor and free humans for higher-order self-improvement.

Q6: What is a “Generative Model of Generative Models”?
A6: This refers to a system, like GFlowNets, that can autonomously create new latent variables or model structures on the fly when its current models fail to explain new data.

Q7: How can we safely improve the world using AI without “monkey’s paw” consequences?
A7: Instead of commanding “end world hunger,” we should provide the AI with the current distribution of human outcomes and ask it to make small, iterative perturbations to that distribution.