📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=CcoGi47qD-w

Multi-Perspective Reasoning: How “The Architects” Mastered the ARC Prize

The team known as “The Architects” has achieved a rare feat in the AI competitive landscape, securing back-to-back podium finishes in the 2024 and 2025 ARC Prize. By blending classic machine learning theories like “Product of Experts” with modern masked diffusion models, they have pushed the boundaries of what LLMs can achieve in visual reasoning.

Core Question: How can researchers adapt text-based LLMs to solve 2D spatial reasoning tasks through diffusion, specialized encodings, and recursive self-correction?

Highlights

• Transitioned from 1D auto-regressive models to 2D-aware masked diffusion architectures.
• Implemented “Golden Gate” positional encodings to provide the model with diagonal and multi-angle spatial awareness.
• Rediscovered Hinton’s “Product of Experts” method to boost accuracy by aggregating multiple model perspectives.
• Discovered a “recursive refinement” technique in the final weeks of the competition that allows models to iterate on their own predictions.

⏱️ Reading time: approx. 6 minutes · Saves you about 33 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

The Evolution of a Winning Strategy

From 2024 Victories to 2025 Innovations

The Architects—Yanisov, Daniel Fransson, and David Hartman—approached the 2025 competition with the unique advantage of being the previous year’s champions. Rather than discarding their winning 2024 code, they adopted a two-pronged strategy: optimizing their existing auto-regressive pipeline while simultaneously exploring radical new architectures. This pragmatic approach allowed them to hold a top leaderboard spot using “secret sauce” models from the previous year while they spent months refining more complex experiments.

Speed was a primary focus during the early stages. The team implemented speculative decoding and prefix caching, which boosted their depth-first search and scoring speeds by nearly five times.

This efficiency was critical because it granted them a larger “test-time compute” budget, enabling the model to explore more potential solutions within the strict Kaggle submission time limits. While many teams struggled with the 12-hour constraint, the Architects were able to run more iterations and score more candidates, effectively turning compute power into reasoning accuracy.

💡 Digging Deeper

Q: How did the team balance their time during the six-month window?
A: They rotated roles across data optimization, model training, and sampling, allowing each member to follow their specific interests while maintaining overall progress.

Q: What was the impact of their 2024 win on the community?
A: Their previous solution served as a direct inspiration for other top teams, including the 2025 winners Envy Arc, who implemented parts of the Architects’ 2024 approach.

Q: Did they use massive compute resources?
A: They used a reasonable budget provided by Salt (GH200s) and a “hero run” on a 16-H100 cluster from Lambda during the final two weeks.

Masked Diffusion and Spatial Awareness

Breaking the 1D Text Barrier

Standard LLMs process information in a linear string, which is inherently ill-suited for the 2D grid puzzles found in ARC. To solve this, the team shifted to a masked diffusion LLM called “Lada” (a variant of Llama). Unlike auto-regressive models that generate one token at a time, diffusion models can predict entire chunks of a grid simultaneously, allowing the model to “fill in the blanks” of a partial solution.

This shift necessitated a total rethink of how the model “sees” the grid.

They replaced standard 1D positional encodings with “Golden Gate” positional encodings. This method uses the golden ratio to determine angles for spatial vectors, allowing the model to look not just left and right or up and down, but diagonally and across various complex axes. It effectively turned a text model into a vision-adjacent model capable of understanding geometric relationships.

💡 Digging Deeper

Q: Why use diffusion over auto-regression?
A: Diffusion allows the model to refine a temporal solution globally rather than committing to one token at a time without the ability to look ahead.

Q: What data did they use for fine-tuning?
A: They utilized the Michael Hodel “re-arch” dataset and the standard ARC2 training sets, totaling about 1,000 tasks plus augmentations.

The Power of Perspective

Product of Experts and Recursive Sampling

One of the most profound insights in their paper, Product of Experts with LLMs, is that a correct solution should look “good” from every possible angle. The team found that by rotating or flipping an ARC task and asking the model for a probability score on each variation, they could multiply these probabilities together. This “Product of Experts” (PoE) approach ensures that if a model sees a flaw in even one perspective, the entire candidate gets a low score, effectively filtering out “almost correct” but ultimately wrong answers.

The final breakthrough came just two weeks before the deadline: a recursive sampling trick.

By feeding the model’s own output back into its input—along with a “mask” token applied to every position—the model learned to act as its own editor. Even though the model was never explicitly trained on “mixed tokens” or continuous embeddings, it interpreted the mask token as a signal to “improve this area.” The team ran this loop 30 to 40 times per task until the solution converged, a discovery that led to a massive leap in their final score.

Key Takeaways

The success of The Architects underscores a critical shift in ARC research: the move from simple “next-token prediction” to iterative, multi-perspective reasoning. By treating the LLM as a flexible generator that can be corrected and refined through recursive loops, the team bypassed many of the traditional limitations of auto-regressive architectures. Their use of “Product of Experts” serves as a reminder that classical machine learning principles—some decades old—remain incredibly powerful when applied to modern transformer models.

While their experiments with synthetic “arcade game” data didn’t yield a score boost for ARC 2, they represent a forward-looking approach for the upcoming ARC 3 benchmark. The team’s journey from a “boy band” of researchers at ICML to top-tier competitors proves that consistent iteration, early submission, and community collaboration are the true requirements for moving the needle in Artificial General Intelligence research.

Q&A

Q1: What is the “Product of Experts” approach in simple terms?
A: It’s a way of combining different model “opinions.” Instead of averaging them, you multiply the probabilities. This means if one “expert” (or one perspective of the task) finds a mistake, the whole solution is rejected.

Q2: How did they handle the 2D nature of ARC tasks?
A: They used “Golden Gate” positional encodings which allow the model to perceive spatial relationships in many directions (diagonal, vertical, etc.) rather than just a linear sequence of tokens.

Q3: What was the “recursive refinement” trick?
A: They fed the model’s prediction back into itself, adding a “mask” token to every pixel. This signaled the model to rethink and improve its previous guess, iterating until the solution stopped changing.

Q4: Did the team use synthetic data?
A: Yes, David tried generating arcade-style games using Quen 2.5 and Vision LLMs, though it didn’t improve their ARC 2 score significantly. They believe it might be more useful for the upcoming ARC 3.

Q5: What was the biggest bottleneck in the Kaggle environment?
A: Debugging and compute constraints. They suggested starting submissions very early to deal with “exception” or “timeout” errors that provide very little feedback.

Q6: How did they manage to stay at the top of the leaderboard for two years?
A: A mix of optimizing last year’s successful model for speed while researching “radical” changes like masked diffusion and multi-angle encodings in parallel.

Q7: What is “speculative decoding”?
A: It’s a technique where the model tries to guess several tokens at once rather than just one, which made their inference process about five times faster.