📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=7xTGNNLPyMI

From Internet Scraper to Reasoning Machine: How LLMs Actually Work

Large language models like ChatGPT are often viewed as magical black boxes, yet they are actually the result of a rigorous, three-stage engineering pipeline designed to turn raw data into digital intelligence. This guide provides a comprehensive mental model for understanding how these models are trained, why they hallucinate, and how the new era of “thinking” models is changing the landscape of AI.

Core Question: How do we transform a massive, messy scrape of the internet into a helpful, reasoning assistant through pre-training, supervised fine-tuning, and reinforcement learning?

Highlights

• The “Base Model” is simply a high-dimensional internet autocomplete engine that simulates the statistical patterns of human text.
• Supervised Fine-Tuning (SFT) is the process of “programming by example,” where human labelers teach the model to act like a helpful assistant.
• LLMs do not see text like humans; they process “tokens,” which explains why they struggle with simple tasks like counting letters in the word “strawberry.”
• Reinforcement Learning (RL) is the latest frontier, allowing models to discover “Chain of Thought” reasoning strategies that even their creators didn’t explicitly teach them.

⏱️ Reading time: approx. 15 minutes · Saves you about 196 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

The Forge of Intelligence: Pre-training

Converting the Internet into a Statistical Map

The journey begins with the pre-training stage, which is essentially the most expensive “downloading” task in history. AI companies scour the public web using tools like Common Crawl to collect trillions of words, which are then aggressively filtered to remove malware, spam, and low-quality content.

Pre-training is the foundational stage where the model learns the “gestalt” of human knowledge by predicting the next word in a sequence over and over again.

To process this data, computers convert text into “tokens”—mathematical chunks that represent groups of characters. Because the model sees tokens rather than individual letters, it often lacks a native understanding of spelling, which is why it might fail at character-level puzzles while simultaneously solving complex physics equations. This tokenization process, specifically Byte Pair Encoding (BPE), compresses the 44 terabytes of the “FineWeb” dataset into a manageable 15-trillion-token stream that the neural network can ingest.

Training these models requires massive data centers filled with thousands of Nvidia H100 GPUs, costing millions of dollars in electricity and hardware. These GPUs work in parallel to “twiddle the knobs” or parameters of the Transformer architecture—the mathematical backbone of modern AI. By the end of this stage, you have a “Base Model” which is essentially a sophisticated document simulator that can mimic any style of writing it found on the open web.

💡 Digging Deeper

Q: Why can’t we just use the Base Model as an assistant?
A: A Base Model doesn’t know it’s an assistant; if you ask it “What is the capital of France?”, it might respond with another question like “And what is its population?” because it’s just trying to complete a document, not answer a query.

Q: Is the model “memorizing” the internet?
A: It’s a lossy compression. While it can recite famous Wikipedia entries verbatim, it mostly stores a vague statistical recollection of facts within its billions of parameters.

Teaching the Assistant: Post-Training and SFT

Programming Through Conversation

Once the Base Model is ready, it undergoes Supervised Fine-Tuning (SFT) to adopt the persona of a helpful, harmless, and truthful assistant. This stage involves hiring thousands of expert human labelers to write out “ideal” conversations. Instead of the raw internet, the model is now fed a diet of structured Q&A, learning to identify the <|im_start|> and <|im_end|> tokens that define a dialogue.

You are no longer talking to the “internet”; you are talking to a statistical simulation of an expert human labeler following a specific set of instructions.

During SFT, the model learns the protocol of conversation, including how to format lists, how to admit when it doesn’t know something, and how to refuse harmful requests. This stage is computationally “cheap”—taking perhaps a few hours compared to the months required for pre-training—but it is the most critical stage for defining the model’s behavior and personality. It is here that the “Knowledge of Self” is hardcoded, telling the model it was built by OpenAI or Google.

💡 Digging Deeper

Q: How does the model know its name?
A: Its identity is either “baked in” during the SFT stage with specific Q&A pairs or provided in a “System Message” that is invisibly prepended to every conversation you have with it.

Q: Why do models still hallucinate after this training?
A: If the SFT data contains confident answers to facts the model didn’t actually learn during pre-training, it learns to prioritize “confident-sounding” prose over factual accuracy.

The Thinking Model: Reinforcement Learning

Reasoning Beyond Human Imitation

The third and most advanced stage is Reinforcement Learning (RL), which allows models to move beyond mere imitation of human experts. In this stage, the model is given a problem and allowed to “practice” thousands of different solutions. By rewarding the paths that lead to the correct answer (in verifiable domains like math or code), the model discovers “cognitive strategies” like backtracking and re-evaluation.

Reinforcement Learning is the “AlphaGo moment” for language, where the AI discovers reasoning traces that no human would have thought to write down.

This process produces what are known as “Reasoning Models” (like DeepSeek R1 or OpenAI’s o1), which exhibit an internal monologue. You will often see these models “think” for 30 seconds before answering, as they are generating “Chain of Thought” tokens. These extra tokens are the “working memory” of the model; by spreading the computation across many tokens, the model can solve problems that are too complex for a single mathematical “pass.”

💡 Digging Deeper

Q: Can RL make models smarter than humans?
A: In verifiable domains like math, yes. Just as AlphaGo beat the world’s best Go players by discovering new moves, RL allows LLMs to discover more efficient ways to solve logic puzzles.

Q: What is RLHF?
A: Reinforcement Learning from Human Feedback (RLHF) is a specific type of RL for “unverifiable” tasks like joke writing, where a “Reward Model” simulates human preferences to score the AI’s output.

Key Takeaways

Understanding LLMs requires moving away from the idea of “magic” and toward the idea of “statistical simulation.” A model is only as good as its pre-training data, its fine-tuning instructions, and the “thinking space” (tokens) it is allowed to use. When a model fails at a simple task like counting, it is often a limitation of its token-based “vision” rather than a lack of intelligence.

To get the most out of these tools, treat them as high-speed collaborators rather than infallible oracles. Use “Reasoning Models” for complex logic, but stick to faster models for general knowledge. Most importantly, remember that “Chain of Thought” is essential: if you don’t give a model the space to show its work, you are forcing it to do complex mental arithmetic in a single, error-prone leap.

Q&A

Q1: Why does the model struggle to count the ‘r’s in “strawberry”?
A: LLMs process text in “tokens” (chunks of characters). To a model, “strawberry” might be seen as the tokens straw and berry. It doesn’t natively “see” the individual letters unless it is forced to break them down via a tool or a specific reasoning step.

Q2: What is the difference between a model’s “Parameters” and its “Context Window”?
A: Parameters are like the model’s long-term memory (what it learned months ago), while the Context Window is its “working memory” (what you just typed and what it just said).

Q3: Can I run these models on my own computer?
A: Yes. While the largest models require massive server farms, “distilled” or “quantized” (compressed) versions of models like Llama 3 or DeepSeek can run locally on modern laptops using apps like LM Studio or Ollama.

Q4: Is it better to ask a model to “think step-by-step”?
A: Absolutely. This triggers a “Chain of Thought,” allowing the model to use intermediate tokens to process parts of a problem before committing to a final answer, which significantly reduces errors.

Q5: What is “Distillation” in AI?
A: Distillation is a technique where a smaller, more efficient model is trained to imitate the outputs (and sometimes the reasoning traces) of a much larger, more powerful “teacher” model.

Q6: Are “Thinking Models” always better?
A: Not necessarily. They are slower and more expensive because they generate more tokens. For basic factual retrieval or creative writing, a standard “SFT” model like GPT-4o is often faster and just as effective.