Adaptive Markets: Financial Evolution at the Speed of Thought

By Andrew W. Lo

10 min read

Why does it matter? Because markets run on ancient survival software — and mistaking evolution for irrationality is the most expensive error in finance.

The Challenger explosion was solved in thirteen minutes by anonymous traders with no technical expertise and no insider information. A panel that included Neil Armstrong and Richard Feynman needed five months. That's a genuine advertisement for rational markets. Then consider that those same markets have produced tulip manias, dot-com euphoria, and 2008, a catastrophe that announced itself for years before anyone did anything. Both stories are completely true, and the gap between them is where this book lives.

The standard explanation — markets are mostly rational, occasionally broken — turns out to be precisely backwards. Markets are ecosystems built from primate brains running ancient survival software in an environment those brains were never designed for. Crashes aren't glitches. They're predictable outputs. Which means they can be anticipated, partially managed, and understood — just not from inside the framework that misread the hardware from the start.

The Market Named the Culprit Before Any Expert Could

At 11:39 a.m. on January 28, 1986, the Space Shuttle Challenger broke apart seventy-three seconds after launch, killing all seven crew members. Within minutes, before any official had spoken, before any investigation had begun, something extraordinary happened on the floor of the New York Stock Exchange.

By 11:52 a.m. — thirteen minutes after the explosion — trading in Morton Thiokol had to be halted. Too many sell orders, too fast. When it reopened, the stock had dropped 6 percent. By close of trading, it was down nearly 12 percent. The three other major NASA contractors (Lockheed, Martin Marietta, Rockwell International) also fell, but modestly and within statistical norms.

Five months later, a panel that included Neil Armstrong, Richard Feynman, Sally Ride, and Chuck Yeager officially concluded what had caused the disaster: the O-rings, rubber seals on the booster rocket joints made by Morton Thiokol, had stiffened in the Florida cold and failed to hold. Feynman demonstrated this at a press conference by dunking an O-ring in ice water and snapping it apart. The market had reached the same conclusion in the first hours after the explosion.

Researchers who examined the trading records in 2003 found no evidence of insider trading. The market hadn't cheated. It had done what markets do when they're working: aggregated the incomplete, scattered intuitions of thousands of traders — each acting on fragments, hunches, half-remembered engineering knowledge — into a single price signal that turned out to be right. The eventual damages Morton Thiokol incurred came to roughly $200 million. The market had already erased almost exactly that amount from its capitalization on the afternoon of the explosion.

The logic is almost coercive: any pattern in prices gets spotted and traded on until the profit disappears. Samuelson worked this out in the 1960s, and it became the Efficient Markets Hypothesis, a framework you'll keep running into. All available information is already in the price, not because investors are omniscient, but because the competitive pursuit of profit forces it there. The market is a machine for converting self-interest into collective accuracy, and on January 28, 1986, it outperformed one of the most distinguished investigation panels ever assembled.

You Think You're Reasoning — You're Actually Narrating

Antonio Damasio's patient had a brain tumor the size of a small orange pressing against his frontal lobes. Surgery removed it successfully. Most of what made the man himself stayed intact: his intelligence, his memory, his language. Within months he was unemployable anyway.

By the time Damasio encountered him at the University of Iowa, the man (pseudonym: Elliot) was trying to restore disability benefits revoked because nothing testable appeared wrong with him. What Damasio eventually noticed was what the tests missed: Elliot felt almost nothing. No sadness about his bankruptcies. No frustration when he spent three hours debating which font to put on a letter. No anxiety about two failed marriages. He described it himself — topics that once provoked strong reactions "no longer caused any reaction, positive or negative."

Damasio's conclusion was simple and devastating: emotion isn't the enemy of rationality — it's a prerequisite. Without emotional weighting, the brain has no mechanism for prioritizing, no signal distinguishing what matters from what doesn't. Elliot's infinite font deliberation and his missed deadlines weren't personality failures. They were what happens when the feedback loop between thought and feeling gets severed.

EMH built its ideal investor in Elliot's image: the person who processes information cleanly, acts on expected value, stays unmoved by fear and greed. That person can't decide which meeting to schedule.

Elliot's case shows what happens when feeling drops out. A different experiment shows what the reasoning itself is actually doing. P.S. was a split-brain patient, someone whose corpus callosum (the neural bridge between hemispheres) had been surgically cut to treat epilepsy, so each hemisphere processed information the other couldn't see. His right visual field was shown a chicken claw; his left visual field, a snow bank. His right hand picked a picture of a chicken; his left hand picked a shovel. Asked to explain, his left hemisphere (the one with language, the one that had only seen the chicken claw) said: "The chicken claw goes with the chicken, and you need a shovel to clean out the chicken shed." Plausible. Internally consistent. Completely wrong. The left hemisphere had no access to the snow bank that explained the shovel. It constructed a coherent narrative anyway, because admitting ignorance wasn't a mode available to it.

That's the brain's default operation: build a cause-and-effect story that holds together. When you think you're reasoning, you're mostly narrating, generating the most internally consistent account available, whether or not it tracks reality. Intelligence, by this account, is narrative accuracy: how well your internal story matches the world well enough to act on. A market full of investors running this system, each story shaped by fear and memory and evolutionary heuristics, looks nothing like the frictionless information-processing machine EMH describes. It looks like a population of storytellers, some of whose stories happen to be right.

Irrational Isn't a Flaw — It's a Feature Running in the Wrong Environment

Lo ran a thought experiment with imaginary creatures called tribbles. Each tribble makes one choice: nest in a valley (safe in sunshine, fatal in floods) or on a plateau (safe in rain, fatal in drought). Sunshine comes 75% of the time. The obviously rational strategy, always choosing the valley, maximizes expected offspring. The pure optimizers grow fastest in good years. Then comes one rainy season and they're wiped out entirely — extinct by generation 14. The tribbles who dominate long-term are the probability matchers: choosing valley 75% of the time and plateau 25% of the time, for no reason any individual tribble could articulate. By generation 25, they number 173 million. The "irrational" strategy won.

The mechanism is systematic risk. When rain hits, it hits all the valley-dwellers at once. One bad season, one extinction. The probability matchers survive because some of them are always on the plateau. They're hedging the extinction of their lineage, not optimizing individual payoffs. When catastrophes strike whole populations simultaneously, randomizing your behavior is the only guarantee you're never all wrong at the same time.

Psychologists found this exact pattern baffling in humans. In a lab paradigm called the Psychic Hotline game, subjects predict which of two lights will flash next. When one flashes 75% of the time, most people match their guesses to the odds rather than always picking the winner. Dismissed as cognitive failure. Lo reads it differently: an evolutionarily calibrated response to the world that built us. Floods, famines, epidemics: events that hit everyone simultaneously, for hundreds of thousands of years. Populations that diversified their behavior survived. Populations that committed to a single optimal answer got selected out the first time that answer was wrong.

We aren't rational agents with occasional malfunctions. We're collections of adaptive heuristics that function correctly in the environments that shaped them. When those heuristics migrate into a trading floor or a 401(k) decision, they produce what economists call irrational behavior. Lo uses a different word: maladaptive. A sea turtle that swallows a plastic bag isn't defective — it evolved to identify transparent objects in water as jellyfish, and for millions of years that heuristic was correct. The ocean changed faster than the turtle could adapt.

Bugs get patched. Features running in the wrong environment require something different: either a new environment, or feedback strong enough to update the heuristic. That second option is what Lo thinks markets, at their best, actually provide.

Market Crashes Are What Adaptation Produces — Not Failures of It

On August 7, 2007, Lo received a phone call from a former MIT student at a hedge fund. The student asked, with a studied casualness that didn't quite hold, whether Lo had heard anything unusual in the industry. Lo said no. The student said his fund had lost a lot of money. Not a normal amount. A lot. Then he claimed he was late for a meeting and hung up.

The next morning, two more calls arrived from different former students at different funds, both opening with nearly identical words: "Have you heard anything unusual?" By the third call, Lo knew something was happening across a whole class of funds. All of them ran statistical arbitrage, or statarb: buying recent underperformers, shorting recent overperformers, betting that prices revert to the mean.

Lo and PhD student Amir Khandani built a simulation of the simplest version of this trade and ran it through that August. Returns of −4.64%, −11.33%, −11.43% on three consecutive days. Goldman Sachs's CFO, David Viniar, called them "25-standard deviation moves, several days in a row" — statistically, such a sequence should occur once every 10¹³⁵ years, a number that dwarfs the age of the universe by a hundred orders of magnitude.

Then came Friday: +23.67%.

That rebound is the clue. A failed strategy doesn't suddenly recover. A broken model, a fundamental change in market structure: neither reverses in a single day. But a forced liquidation does. If someone sold everything they held across three days, not because they chose to but because margin calls left them no choice, then once that pressure stopped, prices would snap back exactly as they did.

Lo's hypothesis: somewhere in the system, a large institution holding market-neutral equity positions (structured to profit regardless of whether the market went up or down) had needed cash urgently. Not because anything was wrong with the stock trades, but because their mortgage-backed securities were collapsing and lenders were demanding collateral. So they sold whatever was liquid. Equities, specifically the same long/short pairs that every other statarb fund owned. One forced exit became every fund's loss, because they had all, through completely different proprietary methods, arrived at identical positions. Each had adapted to the same environment. Each had solved the same problem. Each had ended up in the same crowded trade, with no way to know the others were there.

The funds hadn't coordinated. Nobody had made an error. The crowding wasn't a coincidence or a failure — it was the inevitable result of many smart competitors independently solving the same optimization problem in the same environment. The same dynamic had killed Long-Term Capital Management a decade earlier, a hedge fund whose leverage ratio rose from 28-to-1 to 250-to-1 before the Federal Reserve had to organize a private rescue. That crowding was still running in 2007, still generating the same fragility, just wearing different clothes.

The system didn't fail. It worked exactly as adaptation would design it to, right up until it didn't.

The Financial Engineering Behind 2008 Is Also Our Best Hope for Curing Cancer

What if the lesson of 2008 isn't that financial engineering should be dismantled, but that it was aimed at the wrong target?

The tools that amplified the mortgage crisis — securitization, risk pooling across thousands of underlying assets — aren't inherently destructive. They're direction-neutral. Point them at subprime mortgages and you get 2008. The math works identically elsewhere.

Lo learned this from a single conversation. When his mother was battling lung cancer, he arranged a meeting with executives at a biotech developing treatments for her disease. He asked one question: did their source of financing influence their scientific agenda? The chief scientific officer turned to his CFO, shook his head, and said: "Influence it? Our financing drives our scientific agenda."

The problem, in concrete terms: a single cancer drug takes ten to fifteen years and $200 million, succeeds about 6% of the time, and almost no investor will touch it. Capital flows toward whatever the funding cycle tolerates, not toward the most promising science.

Diversification changes the arithmetic. Invest in 150 independent cancer programs simultaneously and the probability of at least three successes reaches 98%. Three approvals, each worth roughly $12 billion at FDA approval, more than cover the entire portfolio. The $30 billion price tag sounds impossible until you consider that the U.S. bond market holds $40 trillion — over $27 billion could be raised by issuing A-rated "cancer bonds" backed by the diversified portfolio. Same securitization mechanics, different collateral.

Harvey Lodish ran this experiment without knowing it. When he co-founded Genzyme in 1983 to treat Gaucher disease, a genetic disorder affecting 1 in 20,000 people, the first treatment required 22,000 human placentas per patient. He found a researcher who had already cloned the relevant enzyme gene and asked to use it; a recombinant treatment launched in 1994. Sanofi acquired Genzyme for $20 billion in 2011.

Lo can't replicate that — he has a finance PhD, not a biology lab. But the distinction matters less than it seems. An investor in a cancer megafund occupies the same position Lodish did in 1983: funding work whose personal consequences are invisible at the time. The same machinery of fear and greed that makes markets fragile also makes them extraordinarily effective at mobilizing collective resources. The question is only what we point them at.

When the Machinery Finally Serves the Mission

Here's the thing that doesn't let go: Harvey Lodish didn't know what he was building for in 1983. He was solving a different problem — funding a treatment for a disease so rare it barely registered on pharmaceutical radar. The engineers who built securitization weren't trying to cause a crisis either. They were moving capital across risk, efficiently, as designed. Same mathematics. Different consequences. The instrument was neutral. The direction wasn't.

That's what the Adaptive Markets Hypothesis hands you — not a verdict on markets, but a steering wheel. If evolution explains the crashes, it also explains why the same machinery could be aimed somewhere else. The fear and greed that build crowded trades is the same force that could fund a thousand cancer trials. You don't have to change human nature. You just have to point it somewhere worth adapting toward.