Jeff Dean: The 1% Rule for Building in AI

Why does it matter? Because the test for whether your AI startup will survive is a benchmark you can run tonight.

Jeff Dean spent 25 years building the infrastructure underlying modern AI — MapReduce, Bigtable, TensorFlow, TPU, Gemini. He still thinks most founders are running the wrong filter before they build. His test: benchmark a frontier model on your exact task before writing a line of code. If it succeeds 20% of the time, walk away. If it fails completely, you might have something.

• The only defensible starting point is a task where frontier models succeed 0–1% of the time, not 20%
• Data movement costs 1,000x more energy than computation — most "model problems" are actually IO problems in disguise
• Agents that run for days or weeks are coming; most founders are still designing for minutes
• When agents write all the code, taste — knowing what to ask them to build — becomes the only moat

A 20% frontier model success rate on your task isn't an opening — it's a countdown

A 20% success rate means the capability is already emerging in the general model. Six more months of scale will finish the job before you ship.

Dean's filter is unambiguous: "look for something where the model succeeds 0% or 1% of the time, not 20%." The reasoning is precise: partial success signals the capability is "starting to be present in those models" and will be absorbed by the next training run. Before building anything, run the benchmark.

Two shapes of opportunity survive this test. First: data the general model can't access — personal information, proprietary domain data, internal tooling. The model can't learn what it's never seen. Second: problems so hard and narrow that general-purpose training can't crack them. AlphaFold wasn't a general model; it was a highly accurate niche one for protein folding that ran circles around anything a generalist approach could deliver. Neither of those advantages gets inherited by the next Gemini release. Everything else is a race against Google's training roadmap.

Most founders are designing for a 30-step agent when the real product runs for weeks

"For some problem domains and with highly capable models underlying them, you can get them to run for days or weeks and do really, really complicated tasks." Dean followed that with a warning: "I don't think everyone has really internalized this. And that's going to be really a pretty big deal."

What breaks agents at step 30 isn't a fundamental ceiling — it's distribution shift. Once a model drifts slightly outside its training, performance degrades fast. Three countermeasures: give the model more time to plan before acting; build multi-agent systems where a separate evaluator prunes dead-end paths; and write skills that keep agents on familiar ground inside novel environments.

Dean and Sanjay built a concrete example. They wrote a skill that taught a coding agent to run microbenchmarks, measure cache footprint, and iterate on performance changes — a full optimization loop, automated, handling work that would normally require a senior Google engineer. The skill codified exactly the approach they'd use themselves, made legible enough for the model to execute.

If you're not designing for task decomposition, checkpointing, and evaluation loops from day one, you're building for a product window that's already closing.

Context engineering is the only AI layer every founder can compete on right now, regardless of compute budget

Training models requires GPU farms. Context engineering requires an API key.

Dean draws a clean line between what Google will keep compounding at (model training, hardware) and what's genuinely open to everyone. "For context engineering, everyone here could do it. You just need the API to something like Gemini." The structural asymmetry is real: training data is "trillions of tokens stirred together into a soup of hundreds of billions... of parameters" — opaque, indirect, untargetable. Context is different. It's what the model sees clearly, right now, for this specific problem. More legible than anything baked into weights.

The components that compound: retrieval design, skill libraries that encode your domain expertise, tool orchestration, and eval loops. Dean published a 30-page document called "Performance Hints" — his and Sanjay's distilled engineering judgment on system optimization. Others fed it to models in summarized form and watched those models improve at reasoning about performance issues in code. That's the pattern: institutional knowledge, made legible to agents, becoming a durable advantage.

Stop trying to out-train Google. The moat is the system around the model, and that surface is accessible to a two-person team today.

Data movement costs 1,000x more energy than computation — and founders keep misdiagnosing this

One picojoule to compute. A thousand picojoules to move the data. That gap, Dean says, "quietly decides what products are possible and how these algorithms in AI are built."

The downstream consequence most engineers overlook: this is why batching exists. Batching isn't about model efficiency — it's about amortizing data movement. You pay the thousand-X energy cost once for a large batch instead of per token. But batch size is the enemy of latency, and latency determines which product experiences are buildable at all. Dean is working on inference hardware designed around minimizing data movement as a first-order design constraint, not an afterthought.

The practical diagnostic: when your AI system is slow or expensive, the first question shouldn't be "is the model too big?" It should be "is this a compute problem or a data-movement problem?" They have completely different solutions. One calls for better hardware; the other calls for rethinking how much you're moving and when. Founders who misread IO bottlenecks as model quality problems spend months chasing the wrong fix through the wrong layer of the stack.

By 2027, AI will improve itself through fully automated experimentation loops — no human researchers required

300,000 times faster, and nearly as accurate as the original. Dean's colleagues trained a model on the inputs and outputs of an expensive density functional theory simulator — one that took a full night to run — and produced a learned approximation that returned results in seconds. When an experiment takes a night, you run a handful. When it takes seconds, you run millions. The science doesn't change; the feedback loop does.

That template is now being applied to ML itself. "You will see a lot more automation of ML systems themselves, basically getting ML systems to improve their capabilities by running lots of experiments." The loop he describes: automated problem decomposition, automated experimentation, automated evaluation, results integrated back into the next training run — all without a human researcher reviewing each step. AlphaChip and AlphaEvolve already demonstrate this in production at Google. "There's no real impediment to making that be a much more automated loop where the model itself decides it's going to explore."

The key ingredient is a fast evaluator. Slow evals bottleneck the loop regardless of how fast everything else runs. Anyone building ML tooling, evaluation infrastructure, or automated experimentation platforms is positioning in a market about to become dramatically more valuable.

When agents write all the code, knowing what to build becomes the only scarce skill

Execution is being commoditized. Problem selection is not.

Dean's framing is direct: "it's really having incredibly good taste in what you ask your agents to work on." He draws the comparison from research explicitly: most of the battle is choosing the right problem. "If you pick the problem well and you succeed in solving it, that's way better than if you delightfully execute a research investigation into a rather boring problem." Models won't solve this for you. "That high level wisdom of what to work on, I think is incredibly important. I think models are not necessarily going to be that good at it."

Taste isn't innate — it's calibrated. Dean's method: write down 10 things you think will matter in the next 12 months. Pick one to work on. After a year, score yourself on the rest — which ones actually mattered? Which did others build first? Repeated cycles of prediction and feedback are how the loop closes. A third path: thought experiments that force you to question assumptions so embedded in your domain that nobody examines them anymore. The TPU came from questioning whether compute hardware had to be general-purpose. MapReduce came from squinting at parallelization code until a cleaner abstraction emerged.

Specialized inference hardware — not bigger training chips — is the 'fits in RAM' inflection point of this era

In 2001, Dean and Sanjay realized Google's search index would soon fit in RAM instead of spinning hard drives. They rebuilt the entire search system in a few days. Speed wasn't a marginal improvement — it was a product category transformation that made Google fast enough to feel instant.

Dean sees inference hardware as the equivalent inflection: "you're going to see more and more high performance and low energy inference hardware systems." And the stakes: "imagine what you could do with something where the latency is 50x better."

The TPU delivered 30 to 80 times better energy efficiency and 20 to 30x lower latency over CPUs and GPUs of its era — not by being more powerful, but by doing less. It supported only low-precision dense linear algebra, nothing else. That specialization, which seemed dangerously narrow at the time, turned out to be exactly right because the same primitive underlies nearly every modern ML algorithm. A similar push for inference specialization is underway.

A 50x latency improvement doesn't make existing products faster. It makes product categories possible that don't yet exist — the same way instant search unlocked behaviors nobody designed for when results took seconds.

The best original ideas come from questioning assumptions the entire industry has held for 50 years

Sixty years of chip design have converged on one assumption: transistors must be nearly error-free. One error per million years, not 20 errors per day. Dean's thought experiment: what if you built from transistors that made 20 errors per day? Fabrication would look different. Signal paths would be redundant. Higher-level error correction would work differently throughout the stack. The whole design space changes — and might unlock things that aren't achievable under an absolute error constraint. He's not proposing to build it. He's demonstrating a method.

"Sometimes it's good to not take as a given things that most people seem to take as a given." MapReduce emerged from stripping the assumption that parallelization logic had to live inside business logic. The TPU came from questioning whether compute hardware had to be general-purpose. Distillation — rejected by NeurIPS as "unlikely to have significant impact" — came from questioning whether you needed a large model to serve large-model quality at scale. It now powers Gemini Flash.

The method is replicable. Pick one foundational assumption in your domain and work through, rigorously, what changes if it's wrong. Not as a parlor game — as a structured research exercise.

The age of agents rewards those who ask the right question, not those who execute fastest

The compounding force underneath everything Dean described isn't any specific technology — it's the narrowing of what humans will remain uniquely responsible for. Frontier models keep absorbing tasks where they're already partially competent. Inference hardware keeps reshaping what latency makes possible. Automated experimentation loops keep compressing the research cycle.

In that environment, the bottleneck migrates relentlessly toward judgment: which problems are worth running agents on at all? That question won't be answered by better models. It gets answered by founders who have trained their taste through repeated prediction, calibration, and the discipline to question assumptions everyone else treats as fixed.

The scarce thing was never the code.

---

Topics: AI, startups, hardware, inference, agents, context engineering, Jeff Dean, Google DeepMind, TPU, ML systems, founder advice, product strategy