📺 Today’s recommended deep-dive video: https://www.youtube.com/watch?v=zR9PpXWsKFQ

Where Code Meets Capital: Production Engineering at Jane Street

In high-frequency trading, a minor software bug isn’t just a site outage—it’s an existential threat that can liquidate a firm in under an hour. This article explores how Jane Street manages production environments where every single order is mission-critical and “99.9% uptime” is a failing grade.
Core Question: How does Jane Street adapt traditional SRE practices to survive the high-pressure, zero-error environment of global financial markets?
Highlights

• Why Jane Street rejects traditional SLO-based monitoring for event-based alerting.
• The “Fill Too Good” paradox: Why making too much money is a red-flag alert.
• The dangers of cause-based alerting and why you shouldn’t alert when a database goes down.
• How deep business context allows engineers to resolve complex incidents in minutes.
  ⏱️ Reading time: approx. 8 minutes · Saves you about 60 minutes vs. watching.

Want to take notes while watching? Click the image below and let AI Notebook capture the key points for you 👇

The High-Stakes Trading Environment

Why “Good Enough” Isn’t Enough

Trading is essentially connecting a computer program to a bank account and letting it run in a tight loop—a terrifying concept for any seasoned engineer.

Traditional web services often prioritize availability, assuming that a small percentage of failed requests is an acceptable trade-off for scale. In trading, however, every single order is vital. If you sell 600,000 shares at 1 yen instead of one share at 600,000 yen, the financial system will not “undo” the mistake for you. We saw this reality in 2005 with Mizuho Securities, where a single typo cost the firm 27 billion yen in minutes.

Traditional SRE focuses on Service Level Objectives where a tiny fraction of failures is acceptable, but in trading, missing the wrong 0.01% of orders can lead to catastrophic financial losses or legal fines. This necessitates a shift from statistical monitoring to exhaustive event-based tracking for every edge case. We cannot simply aggregate errors; we must understand why every single anomaly occurred.

💡 Digging Deeper

Q: Is trading just a faster version of traditional web tech?
A: Not exactly. While speed matters, the “adversarial” nature of the market means that if you make a mistake, other participants will actively “pounce” on it to extract value from your error.

Q: What is the most dangerous type of bug?
A: The “feedback loop” bug, where a system enters an infinite loop of bad trades, often accelerating its losses as it tries to correct its own faulty positions.

Reimagining Monitoring and Alerting

The Death of the Database Alert

We mostly believe that alerting on a database going down is a mistake that leads to noisy, useless notifications for the on-call staff.

The logic is simple: your users don’t actually care about your database; they care about the errors they receive when they try to use your service. If you alert on the database failure and the resulting 500 errors, you’ve created duplicate noise. If you have a backup database that takes over seamlessly, the database alert becomes “fluff” that engineers eventually learn to ignore, which is the most dangerous state for a production team.

We prefer symptom-based alerting. If a service is failing to provide value, that is the alert. The fact that the database is the cause can be attached as metadata to the alert, rather than being the alert itself. This keeps the signal-to-noise ratio high and ensures that every time a phone buzzes, it represents a genuine impact on the business.

The “Fill Too Good” Paradox

The “Fill Too Good” alert is my favorite tool because it acts as an orthogonal safety net that detects issues no specific service monitor could ever find.

Most systems are designed to alert when things fail or slow down, but we alert when things seem suspiciously successful. If we make significantly more money than our models predicted, it suggests our understanding of the world is broken. Whether the cause is stale market data, a misconfigured instrument, or a rogue algorithm, this alert catches the “unknown unknowns” by monitoring the epistemic health of the firm.

💡 Digging Deeper

Q: How do you prevent alert fatigue with so many “event-based” checks?
A: It requires a ruthless cultural obsession with the signal-to-noise ratio. If an alert is noisy, we don’t just “tweak” it; we often refactor the code to eliminate the edge case or silence it programmatically.

Q: Who receives these high-level alerts?
A: At Jane Street, the Order Engines team often acts as the “last line of defense,” so they receive many of the broadest risk alerts.

Defense in Depth and Business Context

Layers of Protection

You simply cannot build a single system so reliable that you are 100% confident it will never fail, so we rely on uncorrelated layers of defense.

We run similar risk checks in different systems, written by different teams, using different languages or metadata. If the trading strategy has a bug in its risk logic, the order entry port—maintained by a separate team—should catch the breach. We also split monitoring between technical health (monitored by engineers) and trading health (monitored by traders), ensuring two different “lenses” are always watching the system.

Engineers as Business Experts

Production engineers at Jane Street need deep business context because, during an incident, “Spoos market data is stale” must be understood instantly without a lookup table.

High-fidelity communication saves precious seconds during a market open. If an engineer knows that “Spoos” refers to S&P 500 futures on the CME exchange in Aurora, Illinois, they can find the failing order engine in seconds. Without that context, the incident response becomes a game of “telephone” where technical staff and traders struggle to translate their specific terminologies while the firm loses money.

Key Takeaways

Jane Street’s approach to production engineering is defined by a move away from the statistical “safety” of SLOs toward a culture of exhaustive, event-based accountability. By treating monitoring systems as more critical than the trading systems they watch, the firm ensures that they are never “flying blind” in a volatile market.

The integration of business knowledge into the engineering role is not a luxury; it is a core requirement for high-speed incident resolution. When engineers understand the financial impact of their code and traders understand the technical constraints of the stack, the firm can resolve complex, multi-system failures in minutes rather than hours. This synergy, combined with a “defense in depth” strategy, allows Jane Street to operate in one of the world’s most dangerous software environments with remarkable stability.

Q&A

Q1: How do you handle “Exchange-Driven Changes” (EDCs) that might break your systems?
A: We have teams dedicated to parsing the flood of emails from exchanges. However, it’s still a signal-to-noise problem. We are increasingly using LLMs and specialized processes to ensure we don’t miss a “partition split” or a protocol update buried in a routine notification.

Q2: What happens when an order engine “halts” during an incident?
A: The engine rejects further orders, and the trading strategy receives these rejects. The strategy must then have internal logic to decide whether to wait or resend the order once the engine is resumed.

Q3: Does every team at Jane Street use these event-based alerts?
A: Yes. We don’t have a separate “Ops” team that writes alerts for the “Devs.” The people who write the code are responsible for identifying every way it could fail and writing the corresponding alerts.

Q4: How do you deal with alert cascades where one failure triggers 100 alerts?
A: It’s a challenge we still face. However, symptom-based alerting helps. If we focus on the fact that “User X cannot trade” rather than “Service Y is down,” we can often prune the noise and find the root cause faster.

Q5: Do you have top-down standards for logging and alerting?
A: Surprisingly, no. Jane Street has almost zero top-down edicts. While we use common libraries, individual teams are free to develop their own conventions, though shared infrastructure usually leads to a natural convergence of styles.

Q6: Why is the “Fill Too Good” alert better than a standard “P&L” monitor?
A: Standard P&L monitors often look for losses. “Fill Too Good” catches instances where you are making money for the wrong reason, which is often the first sign of a massive technical failure that hasn’t turned into a loss yet.

Q7: How critical is the “What Changed” tool?
A: Vital. During an incident, the first question is always “What changed?” Being able to see every binary deploy, config tweak, and metadata update across the entire firm in one timeline is the cornerstone of our rapid response.