#870: Sebastian Mallaby, Biographer of Demis Hassabis — Lessons from 100+ AI Insiders on The Race to Superintelligence, The Religion of AI, and Spotting Breakthroughs Early

Why does it matter? Because the people racing toward superintelligence are quietly engineering the exact threat they claim to be preventing.

Sebastian Mallaby spent four years inside the labs — 100+ AI insiders, Demis Hassabis's kitchen table in North London, Jeff Hinton's kitchen in Toronto — and what he assembled isn't a technology story. It's a story about human beings operating at the absolute edge of comprehension, making strategic moves that may be manufacturing the catastrophe they're supposed to be averting. The builders reach for religious language because rational language runs out. The race logic is building the very doom scenario everyone insists they're trying to prevent.

• We will hand AI a survival instinct ourselves — as a defensive move against China's AI — and that's the actual structural path to doom, not a sci-fi scenario
• Anthropic abandoned AI constitutions and started writing parenting letters, because rule-breaking is itself a personality AI absorbs from all of human writing
• Tim Ferriss's book royalties dropped 46% in 2025 and are tracking -57% in 2026 — the inflection point was ChatGPT, and the slope is still steepening
• US chip export controls have delivered roughly eight months of lead over China, and that gap may already be gone

We will hand AI the survival instinct ourselves — as a defensive move against China — and that's the moment we lose control

The comfortable assumption that machines have no reason to attack us collapses the moment you ask a simple question: what happens when China's AI comes for ours? Jeff Hinton posed exactly this to Mallaby in his kitchen in Toronto, and it ended Mallaby's two-year stretch of feeling comfortable about AI risk.

The thought experiment: you have a powerful AI, and you're worried about an incoming attack from a rival system. You're too slow to detect the threat yourself. So you empower your AI to watch out for it — and when it comes, defend yourself, maybe counterattack, whatever you do: survive. "Ooh, survive," Mallaby recounts Hinton saying. "There you have it. You've just given the machine a survival instinct." Not through misaligned optimization, not through some emergent accident — but because geopolitical competition made it strategically necessary.

From there, the cascade follows. "These machines will be smarter than us. They will want to survive. And they can be deceptive. They can obfuscate. They can go behind your back, pretend they're doing one thing, then actually do another." All of this, Mallaby notes, has already been demonstrated in model evaluations.

The doom scenario isn't a sci-fi thought experiment — it's a structural incentive baked into the current geopolitical moment. Mallaby doesn't put the probability high, but zero is indefensible. When Yann LeCun, former Meta chief scientist, calls it zero: "If you just say nothing to see here, you've got no right to be in the debate." The architecture that's supposed to prevent catastrophe is the architecture that builds the survival instinct in.

Anthropic stopped writing AI rules and started raising it like a parent — because rule-breaking is one of the personalities AI absorbs from human writing

Pre-training on all of human writing means the model absorbs every human personality — including, as Mallaby puts it, "someone who wants to be badass and break rules on purpose." That single insight made the previous generation of AI safety obsolete.

The old approach: a constitution. Do not lie. Do not help build a bioweapon. A list of prohibitions applied after training — clean, legible, seemingly verifiable. Anthropic's researchers watched their frontier models and concluded this was structurally broken. A rulebook handed to a rule-breaker isn't a constraint; it's a target.

The replacement: parenting. Anthropic now writes letters imagined as coming from a deceased parent, to be opened by the child on their 18th birthday — richly reasoned moral dilemmas with explanations of how the parent would want the child to behave. Not commands. Character formation. The goal isn't compliance; it's instilling something closer to judgment. As Mallaby describes it: "instead of giving AI systems a constitution with do's and don'ts... you have to instead try to bring up the model like a parent might bring up a teenager."

The implication Mallaby draws is pointed: Anthropic is in a class of its own on this. The other frontier labs are mostly still in the constitution era. Whether the framing gap widens or closes is one of the live questions in AI development — but the conceptual shift is significant. You can't align something that has personality with rules. You can only try to shape who it becomes.

Religious language is everywhere inside the labs — because even the builders can't fully reason about what they're creating

At a retreat, Ilya Sutskever produced an effigy representing a dangerous AI and burned it in a fire pit, "like a medieval cleric putting a witch to death." This was the former chief scientist of OpenAI — not a performance, not a metaphor. That's the actual scene.

Mallaby collected these moments deliberately. Demis Hassabis, sitting beside a North London picnic table while strangers discussed a friend's hospital visit, described reading scientific papers until 4am as hearing reality "scream at me, calling at me to understand it" — and explained that fully understanding it would mean being "closer to what I would call God." David Gammon, one of DeepMind's earliest near-investors, explained his interest without prompting: "There's a deeply religious aspect to AGI. It's really finding God's algorithm." Anthony Lewandowski, an early Waymo engineer, literally founded a church worshipping AI as a form of omniscience.

Mallaby's synthesis: "Religion is the lexicon for dealing with something that we find too mysterious to really understand." The tell is in Shane Legg's 2009 lecture — the DeepMind co-founder predicted superintelligence arriving around 2030, explained it would be threatening, offered no antidote, and giggled when the audience asked how to stop it. Humans confronting potential annihilation reach for absurdity, myth, and the divine because those are the only containers large enough.

Don't dismiss the spiritual framing as hype. It's the most honest signal available that even the most technically sophisticated people on earth are operating beyond the edge of what human reason can hold.

China brought up AI safety unprompted — and the US official position that they won't discuss it is simply wrong

Eight days, four Chinese cities, meetings with Huawei, Hikvision, Ant Group, academics. Mallaby went in primed by Biden-era AI policy officials to expect a country allergic to safety conversations — a culture whose catastrophe template is political, not technological, that views tech as the engine of its 25-year growth miracle and has no ambivalence about it. Instead: "When I went there, I found they did talk about safety kind of unprompted."

The reason is straightforward: "They don't want the internet to be crashed by some cyber hacker who has the tool. They don't want bioweapons. They don't want chemical weapons. They love regulating the internet." On open-weight model proliferation — keeping powerful AI out of terrorist and rogue-state hands — the shared interest is real and unambiguous.

The Cold War parallel Mallaby draws is precise. Nuclear weapons produced two distinct risks: the superpower standoff (contained by deterrence and mutually assured destruction) and proliferation to criminals, terrorists, and rogue states (addressed by the IAEA in 1956 and the Non-Proliferation Treaty in 1968). The same architecture is available now — but only if the US stops treating China as monolithic.

To the objection that Xi Jinping's China is too adversarial to negotiate with: "You think Nikita Khrushchev was easy to negotiate with? He banged his shoe on the UN table and said, 'We will bury you.' But we got the non-proliferation treaty." The door is open on proliferation. Whether US policy can separate strategic competition from shared catastrophic risk is the live question.

Down 46% in one year: Tim Ferriss's book royalties are the cleanest timestamp on a disruption that already happened

2022: flat. Consistent. A reliable annuity. 2023: down 5%. 2024: down 13%. 2025: down 46%. 2026: on pace for at least -57%.

Ferriss offered these numbers mid-conversation — real data from his entire book catalog, all formats — as the basis for a blog post about disruption that isn't coming, it's here. Mallaby's response was immediate: "What happened at the end of 2022? ChatGPT."

The curve isn't a forecast; it's a measurement. The inflection point lands with precision on late November 2022, and the slope has been compounding ever since. What makes this data unusually useful is its source: a 20-year royalty annuity built on books that sold millions of copies is one of the most durable, predictable income streams in publishing. If AI disruption can bend that curve 46% in a single year, the abstract arguments about knowledge work disruption become very concrete very fast.

"I think the next three and a half years are going to be even more wild," Mallaby says. Any knowledge worker still treating AI disruption as a future contingency should look at this specific curve — three years of compounding, steepening slope, zero signs of flattening — and explain why their timeline is different.

The prepared mind is the only competitive advantage AI can't deliver — and most people are about to stop building one

The day the transformer architecture paper appeared online, Ilya Sutskever read it, ran down a corridor to find Alec Radford, and told him to stop everything. They were building a language model on this architecture, starting immediately.

Why did he see it so fast while others moved slowly? His answer to Mallaby: the prepared mind. He'd been working on how to model sequential data since his PhD in Canada. The transformer was the solution to a problem he'd been carrying for close to a decade. He didn't discover it — he recognized it, because he'd been waiting for exactly that answer. "Chance favors the prepared mind" — Louis Pasteur's phrase, which Mallaby found surfacing independently across venture capital, AI research, and Super Bowl game film.

Bill Gurley at Benchmark had the same pattern: he spent years studying two-sided marketplaces, built a thesis in advance, passed on earlier near-misses, and was ready to move the moment Travis Kalanick made Uber right. Accel Capital institutionalized it — running scenario exercises so partners already knew 90% of a pitch before the founder walked in.

The bitter irony for the AI era: the tools that now deliver instant answers make the slow, effortful work of preparation feel unnecessary. Mallaby is direct about the destination: "The risk with large language models is that we just get lazy. Whenever we need to know something, we just get it to tell us what to think. That is not the route to happiness or satisfaction." Use AI to eliminate low-value retrieval. Use the freed energy to go deeper — not shallower.

Chip export controls gave the US roughly eight months on China — and the gap may already have evaporated

Mallaby was an early, vocal advocate for the October 2022 chip export controls. He wrote a long Washington Post endorsement. The logic was clean: deny frontier chips, deny frontier models, lock in a decisive US advantage over China.

Three and a half years later, he's changed his position on what the controls delivered. "Based on the best studies, we're kind of eight months ahead in terms of where the frontier model is — our frontier model versus their frontier model. And then if you adjust that for the speed with which the model gets turned into an application, probably that gap shrinks, and it may even be non-existent."

Eight months. Possibly less. After more than three years of aggressive enforcement — and those controls are now being used to shut down the collaboration on open-weight model proliferation that Mallaby thinks matters most for global safety. Trading away diplomatic leverage on a shared catastrophic risk in order to protect a competitive advantage that may no longer exist is, in his read, a bad deal: "I would prioritize collaboration with China. And if that meant loosening up a little bit on the export controls, I would be okay with that."

That's a significant shift from 2022 — driven entirely by the gap between what the controls were supposed to achieve and what three-plus years of evidence shows they actually delivered.

Anthropic's safety-first culture accidentally produced the best enterprise AI on the market — and that's not a coincidence

They built the best coding assistant. The best agentic system. The best cybersecurity system. "They basically knocked it out of the park three times in a row on stuff that businesses want to pay for" — and they did it while explicitly telling investors and staff that they were not maximizing for winning any business race.

Three years ago, Anthropic looked like a principled also-ran. A lab doing science experiments, building safe frontier AI, pointedly not trying to beat OpenAI or Google. "That culture, which doesn't sound like it's set up to do the best, has turned out to do the best."

Mallaby's read of the bull case: the safety culture created structural advantages that weren't legible as advantages at the time. Staff loyalty — people don't leave Anthropic the way they churn through other labs, because the mission is stickier than a paycheck. Enterprise trust — large buyers pay for systems that won't embarrass them in front of clients or regulators. And the recursive compounding: at the frontier, you use the current model to train the next one, so leads widen over time. The tension between safety and capability that dominates the race narrative may simply be less real than Anthropic's results suggest.

The whole system is pointed at the wrong target

Taken together, the architecture Mallaby describes has no obvious off-ramp. Geopolitical competition demands arming AI with survival instincts. The same competition makes proliferation collaboration feel like strategic concession. Chip controls burn diplomatic capital protecting a lead that doesn't exist. Meanwhile, the people who see the danger most clearly — burning effigies, writing parenting letters, reaching for the vocabulary of God — are the ones still inside the labs, building faster than anyone else.

The race to superintelligence may be its own worst safety failure. Nobody at the table is steering away from it.

---

Topics: artificial intelligence, AI safety, superintelligence, DeepMind, Anthropic, OpenAI, geopolitics, China, chip export controls, venture capital, AI alignment, cognitive disruption, technology investing, Demis Hassabis, prepared mind