📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=aiR7F4jqjXY

The Always-Training Model: Why Ngram is Building “Neural Memory” for the Enterprise

Modern AI models are incredibly intelligent, yet they behave like “smart strangers” when dropped into a company’s private workspace. Instead of forcing LLMs to re-read your entire history every time you ask a question, Ngram co-founders Don Biederman and Jesse Lin are developing a way to bake context directly into model weights through continuous training.

Core Question: Can internalizing context into model weights via continuous learning replace the inefficient, high-cost reliance on massive context windows and RAG?

Highlights

• Ngram moves beyond the pre-training/post-training dichotomy, advocating for models that “always train” to stay current.
• Weight-based memory can offer up to a 100x reduction in token consumption compared to massive context prompts.
• The startup uses adapter fine-tuning (like LoRA) to help models “instinctually” understand bespoke company workflows.
• Human biology, specifically how the brain compresses information and “dreams” to test affordances, serves as a key inspiration.

⏱️ Reading time: approx. 7 minutes · Saves you about 38 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

The End of the “Smart Stranger”

Moving Beyond Context Engineering

Current AI models suffer from a fundamental bottleneck: they possess immense raw intelligence but lack evolving context. When you use a frontier model in a workspace like Notion or Microsoft Teams, you are essentially providing it with “externalized” memory through Retrieval-Augmented Generation (RAG). While this works for simple lookups, it forces the model to act as a stranger who must research your company’s entire culture and technical stack from scratch for every single query.

Ngram views this as a failure of architecture.

By treating training as a continuous process rather than a one-time event, they aim to move context from the “sticky note” of the prompt window into the actual “neural tissue” of the model. This allows an AI to understand company initiatives, hiring pipelines, and brand styles with the same instinctual ease that it understands the capital of France or how to write basic Python code.

💡 Digging Deeper

Q: Is Ngram trying to kill RAG entirely?
A: Not necessarily. While they describe the approach as a “RAG killer” for repetitive, high-context tasks, they acknowledge that some data—like a specific hotel room number from a year ago—is better left externalized. The goal is to decide what should be instinctual versus what should be looked up.

Q: How does this impact the “inference cost” problem?
A: Huge. By internalizing knowledge, you can reduce a 100,000-token context window down to a 100-token query because the model already “knows” the background. This can lead to a 100x reduction in inference costs.

The Architecture of Internalization

Adapters, Weights, and the Bitter Lesson

To achieve this vision, Ngram leverages various forms of adapter fine-tuning, such as LoRAs (Low-Rank Adaptation) and sparse architectures. This technical path allows them to maintain the “white-box” access required to update model weights without needing to retrain a massive 70B parameter model from scratch every time a user uploads a new document. It is a more surgical way to scale compute.

The team is “Bitter Lesson” peeled, believing that the way to solve intelligence is to burn more compute on the problem of memory.

They argue that trying to separate “algorithms” (how to think) from “databases” (facts) is a false dichotomy in deep learning. In a transformer, these two elements are inextricably mushed together; trying to force a model to be a “pure” thinker without any internalized facts makes it unnatural and less capable of complex, multi-step reasoning.

💡 Digging Deeper

Q: Why target the workspace level rather than the individual level?
A: Teams and organizations are more disciplined in how they collect data over years. It provides a “bigger deposit” of information to train on, though Ngram eventually sees this technology moving to individual phones and computers.

Q: Can this work with closed-source models like GPT-4?
A: It is difficult. The process requires “white-box” access to weights. While they can partner with providers, open-source models (like Llama) are currently the most viable targets for this kind of weight-level manipulation.

Lessons from the Human Brain

Lossy Memory and the Role of Dreams

Don Biederman, drawing on his background in neuroscience, notes that human memory is purposefully lossy. We don’t remember every single photon that hits our eyes; instead, we compress what is important to inform future decisions. Our brains have evolved to work within strict constraints of information capacity, creating fuzzy representations that allow for abstraction and connection.

Current AI systems lack this “digestive” step where information is consolidated after the initial exposure.

Ngram envisions a model that can “retreat” from interaction—much like a human dreams—to experiment with what it has learned. During this “dream” phase, the model can test its affordances, seeing how it handles extreme “tail” scenarios or social situations without the pressure of a live user interaction. This allows for a much deeper integration of skills than simple keyword searching could ever provide.

💡 Digging Deeper

Q: What was the “ChatGPT moment” for memory?
A: It hasn’t happened yet. The co-founders believe it will arrive when a user can “teach” an AI intern over a week, and by Friday, the model is noticeably smarter and different than it was on Monday.

Q: Why has language AI surpassed vision AI in the short term?
A: It may be an “electronic” advantage. In biology, vision has a massive bit-rate advantage over audio. In computers, where everything is processed as electrons on the same playing field, language’s high-dimensional abstraction might be easier for current architectures to master than the raw data of vision.

Key Takeaways

The current AI paradigm relies too heavily on “cramming” for the exam. We treat models as static entities and attempt to solve their lack of specific knowledge by providing increasingly massive, expensive, and redundant context windows. Ngram’s approach suggests that the next frontier isn’t just a bigger model, but a more “plastic” one that can learn and internalize its environment in real-time.

By shifting the focus from “retrieving” to “remembering,” companies can move away from the fragility of RAG. A model that understands your codebase or your hiring process as a set of internalized weights is faster, cheaper, and more capable of making creative associations than a model that is simply doing a high-speed search of your Google Drive.

Ultimately, Ngram envisions a future where every individual and team has a personalized “memory bank”—a neural interface to their data plane that evolves alongside them, turning years of digital work into a permanent, actionable skill set.

Q&A

Q1: What is the “bottleneck” Ngram is solving?
A: It isn’t raw intelligence; it’s the inability of models to deeply understand new and evolving contexts without being “spoon-fed” thousands of tokens in a prompt.

Q2: How does Ngram view the difference between a fact and a skill?
A: They believe there is no real distinction in a model’s weights. The same training methods that teach a model a fact (the capital of France) can also teach it a skill (how to triage a specific company’s support tickets).

Q3: Why is a KV Cache called a “monstrosity” in this context?
A: Because it is incredibly bit-inefficient. A KV cache for one long article can take up nearly as much memory as the entire model weights that “remember” the whole internet. Ngram wants to compress that information into the weights themselves.

Q4: Will I have one “Personal Model” for everything?
A: Jesse Lin suggests that we will likely want separation. You don’t necessarily want your work model suggesting personal grocery lists based on a GPU training log you wrote at the office.

Q5: What is the role of “dreams” in AI?
A: It’s a period where the model can experiment with its environment and “digest” interactions offline, improving its performance for the next day without being in a live loop.

Q6: Is this approach “Bitter Lesson” compliant?
A: Yes. It assumes the best way to improve models is to scale training compute on new context rather than relying on human-engineered search heuristics.

Q7: How does Ngram define the future of knowledge work?
A: As a world where everyone has a personalized neural interface to their data, allowing them to carry their learned “digital skills” from one job to the next.