📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=yXPPcBlcF8U

Models Are What They Eat: Why Data Curation Is the New Frontier of AI

For years, the AI research community obsessed over model architectures and parameter counts while treating training data as a static, “brute force” commodity. Ari Moros, CEO of Datalogy and former Meta researcher, argues that the real leverage for building state-of-the-art models lies in a rigorous, empirical science of data curation that moves beyond simple filtering.

Core Question: How can data-centric engineering break the “bitter lesson” of power-law scaling to create models that are 10x faster to train and significantly smaller to deploy?

Highlights

• Why Ari pivoted from “inductive biases” to data after realizing that models are almost entirely a reflection of their training distribution.
• The critical difference between basic data cleaning and a holistic curation strategy involving rebalancing, rephrasing, and curricula.
• How Datalogy achieved a 12x speedup in training by selecting only the most informative tokens from massive open-source datasets.
• Why the next frontier of AI belongs to small, specialized models (under 10B parameters) optimized for inference efficiency.

⏱️ Reading time: approx. 8 minutes · Saves you about 71 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

The Transition to Data-Centric AI

From Inductive Biases to the Bitter Lesson

AI is an empirical science where properties emerge unexpectedly rather than being proven theoretically.

Ari’s early career at Meta and DeepMind focused on “inductive biases”—trying to hard-code architectural rules to help models learn with less data. This approach aimed to build a structured science around why certain representations work, but it frequently hit a wall when tested against pure scale.

Around 2020, a series of research papers delivered a “bitter lesson”: once you scale past a million data points, architectural tweaks matter far less than the learned posterior of the data distribution itself. This realization was confronting for someone who spent years on model design, but it made one thing clear: data is the most under-invested area of research relative to its total impact on performance.

💡 Digging Deeper

Q: What is an inductive bias?
A: It is a set of assumptions built into a model’s architecture to help it generalize. For example, Convolutional Neural Networks (CNNs) assume that patterns are spatially local, which helps them process images.

Q: Why did Ari leave Meta to start Datalogy?
A: He realized that within large labs, data teams are often treated as “second-class citizens” providing the plumbing for modeling teams. He wanted to build a company where data is the end goal, not just a means to an end.

The Science of Data Curation

Beyond Basic Cleaning

Data curation is often dismissed as “plumbing” or grunt work by the research community, yet the world’s best researchers admit their secret is looking at the data. Most labs invest heavily in crawlers, but very few focus on the mathematical value of individual tokens relative to the existing training set.

Curation isn’t just about deleting “bad” data; it is about rebalancing distributions, sequencing information through curricula, and generating high-quality synthetic tokens.

A critical challenge in curation is identifying redundant concepts. While a human can spot a high-quality summary of Hamlet, they cannot keep 10 trillion tokens in their head to realize they already have 10,000 similar summaries. Effective curation requires automated systems that can determine how much redundancy a specific concept—like a “dog” versus an “elephant”—needs to enable generalization without wasting compute. Because dogs have hundreds of breeds and textures, they require more data “surface area” than the more stereotyped appearance of an elephant.

Breaking Scaling Laws and Synthetic Data

Bending the Power Law

Traditional scaling laws (such as the Kaplan or Chinchilla papers) assume data is independent and identically distributed (IID), leading to the grim reality of power-law scaling where every 10x increase in data yields diminishing marginal returns. Ari argues that we can “bend” these scaling laws. By measuring the marginal information gain of the next data point and keeping it flat rather than decaying, we can achieve performance gains that far outpace the trillion-dollar “brute force” extrapolations.

This is achieved through techniques like “rephrasing,” where a weak model reformats existing high-quality data to make it more digestible for the training model.

Unlike net-new synthetic data creation, which risks model collapse by amplifying modes and ignoring tails, rephrasing relies on the information already present in the source. This “distillation in disguise” allows even a small team to produce models that outperform much larger rivals by being smarter about what they eat. By making the training data higher quality, a developer essentially provides the model with a “compute multiplier,” getting more intelligence out of every GPU hour.

The Future of Models (Small and Specialized)

Train Faster, Better, Smaller

For enterprises, the total cost of ownership is dominated by inference, not training. Deploying a model that is twice as large as necessary can cost tens of millions in annual overhead. Training a smaller, specialized model that is “an inch wide and a mile deep” often pays for itself within months.

Ari predicts the next frontier isn’t trillion-parameter behemoths but highly optimized models under 10 billion parameters. These “cognitive cores” can use tools and browse the web to find facts, freeing up their parameters from acting as a fallible database. By purging unnecessary memorized knowledge and focusing on reasoning, we can shrink models while simultaneously improving their reliability for critical “five-nines” production environments.

The goal of Datalogy is to commoditize the data side of AI, just as companies like Mosaic and Together have commoditized the training infrastructure. When high-quality data becomes a push-button service, the barrier to training custom, private, and hyper-efficient models vanishes.

Key Takeaways

The “bitter lesson” of AI is that human intuition about model architecture is usually outperformed by raw scale, but the next evolution of that lesson is that scale alone is too expensive. We are entering an era of “data efficiency” where the marginal information gain of each token is the most important metric for a researcher. If a model is underfitting its data, the solution isn’t just more data—it’s better, more diverse data.

Curation is moving from a manual “cleaning” task to an automated “selection” science. By using models to filter, rephrase, and sequence data, we can achieve 12x gains in training speed. This allows organizations to build “sovereign” or domain-specific AI that is not only more accurate than general-purpose giants like GPT-4 but also significantly cheaper to run at scale.

Finally, the shift toward test-time compute and “cognitive cores” means that models no longer need to be massive repositories of world facts. Instead, they should be compact, high-reasoning engines that know how to fetch information. This transition will be powered by data curation that prioritizes reasoning traces over simple rote memorization.

Q&A

Q1: What is the most common mistake companies make when preparing for a training run?
A: Many spend months on hardware and architecture only to reach out two weeks before training starts to ask for help with the data. Data should be the first consideration, not the last.

Q2: Does “textbooks are all you need” mean we should only train on Wikipedia and textbooks?
A: No. While high-quality tokens are great, diversity is the most important factor for AGI. Over-indexing on a narrow distribution like textbooks leads to models that score well on benchmarks but fail “vibe checks” in the real world.

Q3: Can synthetic data cause “model collapse”?
A: Yes, if you are generating net-new data from the model’s own distribution. However, “rephrasing” (conditioning on real-world data) avoids this by keeping the information source grounded in reality.

Q4: Why does Ari think curricula (sequencing data from easy to hard) are back in style?
A: Previously, we “saturated” models with so much data that the order didn’t matter. Now that we are trying to be more efficient, the dependency between concepts (e.g., learning addition before division) matters for training speed.

Q5: What is “sovereign AI”?
A: It refers to countries or organizations wanting to own and train their own models to reflect their specific language, culture, and private data, rather than relying on a few central providers.

Q6: Is parameter pruning dead?
A: Not dead, but difficult. Unstructured pruning (removing random weights) works well for shrinking models but doesn’t run faster on modern GPUs. Data curation is a more effective way to reach smaller model sizes.

Q7: How did the RC foundation models benefit from Datalogy?
A: They started with 25 trillion tokens and curated them down to 7 trillion. The resulting 4.5B model outperformed larger rivals like Gemma while training significantly faster.