AI Snake Oil: What Artificial Intelligence Can Do, What It Can’t, and How to Tell the Difference

By Arvind Narayanan & Sayash Kapoor

13 min read

Why does it matter? Because the AI that's harming people right now is the kind no one's talking about.

You probably think the AI threat is somewhere in the future — a superintelligence that wakes up wrong, a robot that decides humans are inefficient. Meanwhile, an algorithm already cut off an 85-year-old woman's medical coverage while she was in severe pain. Another one quietly raised insurance premiums for elderly people on a list nobody told them they were on, and the humans nominally "in the loop" kept signing off because who argues with the machine. The confusion between AI that writes you a sonnet and AI that denies your hospital bed isn't a side effect of the hype — it's the hype's entire business model. Narayanan and Kapoor are here to break that model open, tool by tool, claim by claim, number by number.

'AI' Is Not One Thing — and Conflating Them Is the Whole Scam

Imagine someone tells you their commute depends on a 'vehicle.' That single word could mean a bicycle, a delivery truck, or a spacecraft — things that share almost nothing in cost, capability, or risk. You'd never let that ambiguity stand if you were buying one. Yet this is precisely how most public conversation about artificial intelligence works, and the confusion is not accidental.

Narayanan and Kapoor make a clean cut through the fog: 'AI' is not a technology. It's a loose label draped over at least two fundamentally different families of tools with radically different track records. Generative AI — the chatbots and image generators — is genuinely impressive and genuinely failure-prone. Predictive AI — the scoring systems that decide your bail amount, your loan application, your job interview — is, in most social applications, snake oil: it cannot do what its sellers claim. Facial recognition is a third animal entirely. Its error rate dropped from 4 percent to 0.08 percent in six years. It works. Which is exactly what makes it dangerous when Madison Square Garden uses it to bar lawyers from firms suing the venue.

These tools don't exist on a spectrum from 'weak AI' to 'strong AI.' They operate on different principles, trained for different tasks, with different failure modes. Conflating them is the mechanism hype depends on. When a company selling a hiring algorithm invokes the same word — 'AI' — that produced a convincing poem or beat a chess grandmaster, it borrows credibility it hasn't earned. The prestige of one application launders the claims of another.

The confusion costs people. Once you see that 'AI' is a category error rather than a category, the right question stops being 'how smart is the AI?' and starts being 'which tool, doing what, tested how, on whom?' Every section that follows turns on that distinction.

The Models That Harm People Were Never Designed to Help Them

In 2013, a Dutch tax authority built an algorithm to catch childcare benefit fraud. By 2019, it had wrongly accused roughly thirty thousand families. Some owed repayment demands exceeding a hundred thousand euros. The algorithm had flagged them based partly on nationality — Turkish, Moroccan, or Eastern European heritage moved you up the list, all else equal. Families who had never committed fraud found their finances destroyed, their mental health wrecked, their appeals ignored. The system ran for six years. When its mechanics became public, the Dutch data protection watchdog issued a €3.7 million fine, the largest in the country's history. Then, in 2021, the prime minister and his entire cabinet resigned.

What makes this more than a cautionary tale about one bad algorithm: the tool was working exactly as designed. It was trained on historical fraud decisions, which is to say it learned the patterns in who had previously been accused of fraud. When those accusations were themselves shaped by ethnic bias in enforcement, the algorithm absorbed that bias into its scoring logic. It wasn't detecting fraud. It was predicting who looked like past targets. The data didn't encode ground truth. It encoded the choices of the system that came before.

This is the structural problem beneath predictive AI failures: training data records what happened under prior decisions, not what would have happened under fair ones. Optum's widely used patient-risk tool ranked Black patients as lower-risk than White patients with identical clinical conditions, because Black patients had historically spent less on care. Less spending meant less data in the training set, which the model read as lower need. The algorithm formalized decades of unequal access into a medical verdict.

The people deploying these tools often know they haven't been validated against real-world outcomes. They deploy them anyway because the incentives point that way. A company can sell a hiring tool to thousands of firms before a single independent study checks whether it works. A government can license a fraud-detection system and credit it with savings before anyone asks how many false accusations drove those numbers. The CEO collects the contract revenue; the wrongly accused parent loses her housing. The liability stays abstract and distant while the harm lands on specific people with specific addresses.

The lesson isn't that the models are buggy and will improve with better engineering. The wrong question is being asked. When past decisions are baked into training data as if they were facts about the world, better algorithms still answer the wrong question more efficiently.

High Accuracy Is Meaningless If You Don't Ask: Accurate at What, on Whose Data?

Every high-accuracy AI claim is a magic trick, and the sleight of hand is always the same: the number sounds like evidence that the system works, when what it actually measures is narrower, more flattering, and often disconnected from the decision it's being used to justify.

Epic's sepsis prediction model is the cleanest demonstration on record. Epic holds medical records for roughly 250 million Americans, and when it released its sepsis detector in 2017, hospitals had every reason to trust the pedigree. The company claimed a relative accuracy of 76 to 83 percent — meaning that often, when two patients were compared, the one who would go on to develop sepsis got the higher risk score. That sounds meaningful. Epic sweetened adoption with up to a million dollars in credits to hospitals that met certain conditions, one of which was using the sepsis model. Hundreds of hospitals signed on. Then University of Michigan researchers ran the model against a fresh patient population and found the actual relative accuracy was 63 percent. A coin flip is 50 percent. Epic's celebrated model was barely clearing that bar. The hospitals had been making clinical decisions — escalating care, triggering alerts, reallocating nursing attention — on the basis of a number that measured something much smaller than anyone had been told.

Three questions would have exposed this. They fit on an index card. Accurate at what, exactly — what outcome was the model actually predicting, and is that outcome what we care about? On whose data — was the model tested on genuinely new patients, or on some version of the same records used to build it? And compared to what baseline — could a nurse with a checklist, or a simple rule based on two clinical variables, have done just as well? In Epic's case, the answer to that last question was embarrassing.

The pattern produces real casualties. Parents denied benefits. Patients sorted into lower-priority care. Children assigned to remedial tracks by tools their schools paid for and never validated. The companies collecting payment for these systems are rarely in the room when the harm lands.

Human Behavior Is Fundamentally Unpredictable — and That's Not a Data Problem

Could a bigger dataset finally let AI predict whether a child will struggle in school, end up in poverty, or drop out before graduation? This is the assumption beneath billions of dollars of investment in predictive social AI — that the models are merely data-starved, and that more sensors, more records, more compute will eventually close the gap. The Fragile Families Challenge tested that assumption directly, and what it found should end the conversation.

In 2015, the sociologist Matthew Salganik opened a remarkable competition. He had access to a longitudinal study tracking 4,000 children from birth, with roughly 10,000 data points per family — income shocks, school records, neighborhood conditions, parental relationships. He invited 160 research teams to bring their best models and predict six life outcomes for those children at age 15. The teams used the most sophisticated AI available. They lost. Not narrowly — decisively. The winning models barely outperformed a four-variable baseline built from the child's prior GPA, the mother's race, her marital status, and her education level. Variables that any sociologist in 1975 could have written on a napkin.

After the competition, Salganik's team went looking for the families whose outcomes had been most impossible to predict. In one case, a child flagged as high-risk had dramatically improved. The reason: a neighbor had been feeding him blueberries, helping with homework, and providing the kind of consistent adult attention the family couldn't. There was no field in the survey for 'supportive neighbor.' There never will be, because the survey designers couldn't have known to ask. And this is the point the blueberry story makes that no statistical argument can: the unpredictability here isn't a gap waiting for better data. It's a feature of how human lives actually work. They turn on moments — a chance encounter, a small act of kindness, a stranger's decision made on a Tuesday afternoon — that no dataset will ever capture in advance.

That's what Narayanan and Kapoor mean by irreducible error: the gap that no dataset will ever close. When a company sells a tool claiming to predict which job applicant will succeed, which defendant will reoffend, or which student needs intervention, they are selling something that cannot exist — not because the engineering is immature, but because the target moves in ways that are structurally unobservable. Human lives don't have missing fields. They have fields that don't exist yet. There was never a data problem. There was a reality problem, and the companies selling the tools had every incentive to call it something else.

What AI Actually Did Take 80 Years to Build — and Why It's Not Magic

Think of a cathedral built over eight centuries. Stand inside and you feel something that might register as magic — the light, the scale, the way stone seems to float. But the feeling and the explanation coexist: it took eight hundred years of masons solving discrete structural problems, one arch failure at a time, before the building could hold. Generative AI is the same kind of object. The wonder is real. So is the engineering ledger behind it.

The ledger starts in 1943 and runs forward in fits. The first artificial neuron. Early systems could manage simple shapes. Then decades of dead ends: symbolic rule systems in the 1980s, statistical pattern-matching systems in the 1990s. The breakthrough that changed the trajectory wasn't a flash of insight — it was a dataset. In 2007, Stanford researcher Fei-Fei Li recognized that models weren't failing because the algorithms were wrong; they were failing because the training sets were too small. She used Amazon's crowdsourced labor platform to label millions of images, building ImageNet. When a team running a neural network on gaming graphics cards entered the resulting competition in 2012, they won by a margin so large it invalidated the previous decade of assumptions about what machine learning could do.

That win unlocked everything downstream, including the transformer architecture inside every modern chatbot. The transformer's trick is brute-force attention: it calculates how every word in a text relates to every other word simultaneously. Language falls out as an emergent property of doing that calculation at sufficient scale. To generate a single poem of roughly a thousand words, ChatGPT performs approximately a quadrillion arithmetic operations — a million billion. If every person on earth contributed one calculation per minute for eight hours a day, hitting that number would take a year. The model begins producing the first word with no plan for the last one. It is next-word prediction, done at a scale that makes coherent prose appear.

This history matters because it sets the actual boundaries of what generative AI can and cannot do. It is extraordinarily good at tasks where the output is a plausible continuation of patterns in human language — summarizing, translating, drafting. It has no access to truth; it cannot verify. The eighty-year road explains both halves of that sentence, and once you hold both, the wonder and the limits become the same fact — the same architecture that produces the poetry also guarantees the hallucinations.

The 'AI Will Fix Toxic Content' Promise Was Always a Way to Avoid Fixing the Platform

The promise that AI would clean up social media was always a way to avoid the harder question: clean it up according to whose rules, decided by whom, and enforced with what cultural knowledge? Mark Zuckerberg made this rhetorical move in front of Congress in 2018 — 'AI will solve this' — while the actual work was already being done by hand. While Zuckerberg promised AI solutions to Congress, hundreds of thousands of contractors, mostly in the Philippines and Kenya, often paid below living wage, were already reviewing violent and disturbing content by hand, without meaningful psychological support.

The Rohingya case makes the real problem concrete. In 2017, Facebook was the primary communications infrastructure in Myanmar, and it was being used to coordinate ethnic cleansing — posts calling for violence, spreading dehumanizing propaganda, organizing attacks. For years, Facebook had a single Burmese-speaking moderator for the entire country. No AI model catches a dog whistle in a language it can't read, and even if the translation were perfect, the signals are buried in local tropes, in-group references, and context that requires not just language but cultural membership to decode. The genocide happened. The platform had been warned by civil society groups for years.

But even that framing — if only the AI were better at context — is too generous, because the hardest part of content moderation was never enforcement. It was policymaking. When Facebook removed the 1972 Napalm Girl photograph, one of the most important war images in history, that wasn't an AI error. A human had written a rule banning underage nudity, and the rule was applied consistently. Whether that rule should have an exception for historical photojournalism is a political and editorial judgment no algorithm can make, because it requires deciding what values the platform serves. More compute doesn't resolve that. It just enforces whatever answer power chose in advance.

Facebook's own internal research, presented to executives in 2018, found that 64 percent of the people who joined extremist groups on the platform did so because Facebook's recommendation systems pointed them there. The company chose not to fix it. Outrage and provocation drove the engagement numbers they were paid to maximize, and the AI fix was a promise designed to buy time, not a solution designed to work.

The Doomsday Narrative Is Real Harm's Best Alibi

Who decides which AI risks deserve your attention? The question has a less neutral answer than it appears. The loudest voices warning about existential catastrophe — superintelligent systems escaping human control, self-replicating agents pursuing goals that kill us all — happen to belong to the same companies building the technology and lobbying to keep building it without competition.

The paper clip maximizer is the thought experiment that anchors the doomsday case. Give an AGI a trivial goal — manufacture as many paper clips as possible — and it will logically conclude that acquiring more power serves that goal, until it has commandeered the world's resources and killed anyone who resists. The scenario sounds airtight until you ask what intelligence actually requires. An agent that interprets goals this literally would ignore traffic laws when sent to buy a lightbulb, cut in line at the checkout, and walk out without paying, because none of that was forbidden in the instructions. It would be shut down before it reached the parking lot. Real-world competence requires the opposite of literal interpretation — judgment, social awareness, the ability to question a subgoal when it looks absurd. The same rigidity that makes the scenario feel logically compelling is exactly what would make such a system useless.

Philip Tetlock's forecasting research offers a useful corrective. The generalist 'foxes' in his studies — people trained to integrate evidence across domains rather than drill deep into one — put the probability of AI-driven human extinction at roughly 0.38 percent. The domain specialists, the hedgehogs with the narrowest professional frame, ran far higher. The people most convinced of the catastrophe are the ones least likely to have looked at anything else.

Sam Altman's call for government licensing of AI, framed as caution, would require the kind of compliance infrastructure that only incumbents already have. A licensing wall doesn't slow down OpenAI; it walls out anyone who might challenge it. The authors call this criti-hype: criticism shaped so precisely that it protects what it pretends to challenge.

Every hour spent debating the paper clip maximizer is an hour not spent on the Dutch tax algorithm, the Epic sepsis model, or the Rohingya genocide. The doomsday narrative isn't just wrong. It's useful, and useful to very specific people.

The Broken Institutions Buying Snake Oil Are the Same Ones That Need Fixing

Chicago, 2021: the city is hemorrhaging money, its public safety budget is under pressure from every direction, and someone is pitching a system called ShotSpotter — acoustic sensors that claim to detect gunshots in real time and direct officers to the scene. The city spends $49 million over five years. Internal reports later show it didn't improve evidence collection in the slightest. The brokenness of the institution is the sales pitch. Chicago couldn't afford the thing that actually works: enough well-trained officers, enough social services, enough of the expensive human infrastructure that prevents violence before a gun is fired. The algorithm was a substitute for funding, dressed up as innovation.

That's the engine behind most predictive AI adoption in public institutions. Underfunded schools need to decide which students get scarce teacher attention. Overstretched courts need to process cases faster than human review allows. Into every one of those gaps walks a vendor with a dashboard and an accuracy percentage. Auditing the algorithm doesn't fix that — because the institution will just buy the next vendor promising the same shortcut.

Narayanan and Kapoor end the book with two children born in 2022. Kai's world has fear-driven regulation that banned the visible AI products while leaving addictive, engagement-optimized apps untouched. His school uses predictive tracking to allocate the teaching resources it can't afford to give everyone — and the tracking becomes a self-fulfilling verdict, foreclosing careers for kids written off at eleven. Maya's world looks different: research funding into technology's effects on children has multiplied tenfold, platforms must interoperate so no single company can hold attention hostage to its ad model, and admissions lotteries break the feedback loop where elite resources keep flowing to already-advantaged families. The difference between the two worlds isn't a better algorithm. It's whether the surrounding institutions were repaired or just patched with prediction.

The move from Kai's world to Maya's isn't abstract. Residents in San Diego organized against surveillance streetlights — attending council meetings, building coalitions with privacy advocates, and eventually killing the program before it spread citywide. An artist named Karla Ortiz testified before Congress about image generators and shifted the debate. The same public pressure that forced seatbelts into cars is available here. The institutions are broken, but they were built by choices — which means they can be rebuilt by different ones.

The Skill Worth Acquiring Before the Next Headline

The technology is the least important part of what you just read. What matters is the social machinery wrapped around it — who commissioned the study, who profits when the tool gets deployed, who loses their home when it flags them as a fraudster. AI literacy, in the end, is just the old skill of tracing the contract back to whoever profits when the tool gets it wrong and asking who isn't in the room when the decision gets made. The gap between Kai's world and Maya's isn't a gap in algorithms or legislation. It's a gap in how many people are asking those questions out loud, in school board meetings and congressional hearings and procurement reviews, before the contract is signed. The Dutch cabinet didn't resign because the algorithm was exposed. It resigned because enough people understood what they were looking at and refused to look away. That capacity — to see clearly and say so — is not technical. It's political.