Robot-Use Agents: Why General-Purpose Models May Win in Robotics

Why does it matter? Because the robotics data bottleneck may already be solved — and nobody planned it that way.

The researchers building the most capable LLM-controlled robots arrived at a destabilizing conclusion: a strong enough general-purpose model is automatically a strong robotics model, no robot data required. The unlock wasn't a breakthrough in teleoperation or hardware — it was better code generation, and decades of humans dragging cursors through CAD software. This episode traces how those two unlikely threads collapsed the case for specialized robotics foundation models, and why frontier labs now see competent-teenager-level general robots as a two-year engineering problem.

• A sufficiently strong general LLM converges to the same world representation as the best robotics model — no specialization needed • Coding agents deploy new robot tasks in one shot, with zero robot-specific training data • Decades of GUI design — Blender, SolidWorks, 3D interfaces — accidentally produced the world's largest proxy robotics dataset • In-context learning saturates at ~40 examples; the next frontier is compressing that experience back into weights, the way sleep consolidates memory

The strongest general LLM is already the strongest robotics model — specialization is a dead end

If the Platonic Representation Hypothesis holds, the entire case for building a specialized robotics foundation model collapses.

The hypothesis, drawn from MIT professor Philip Isola's viral essay, holds that as models train on increasingly large and diverse datasets, their internal representations converge — regardless of architecture, regardless of training distribution. Models learning from text, images, and code all arrive at the same geometry of the world.

Hanming from Waddle Labs applies it directly: "the very, very strong language models will have very similar representations of the world with very strong robotics models." The logic follows without much work: "if you have a really strong language model, you also have a really strong robotics model." Then the kicker: "you just need one really strong model, regardless of architecture, and they would be outperforming any specific models that is slightly weaker."

This is a direct verdict on years of VLA investment. DeepMind's RT2 was celebrated for fine-tuning a language model to output robot joint coordinates instead of text — a creative workaround for the scarcity of robot training data. The new argument is that fine-tuning was always a patch. The real value was the pre-training. And if pre-training converges to the same world model anyway, the robotics race reduces to one question: who builds the strongest general model?

The practical consequence is uncomfortable for anyone with a robotics-specific thesis. The strongest general model doesn't just beat specialized models on robotics alongside everything else — it wins by default, and no amount of domain-specific teleoperation data closes the gap if the general model is simply stronger.

Coding agents deployed new robot tasks in one shot before anyone realized coding agents worked

Give an LLM a library of robot-control functions — move_to_position(), pick_up_object(), check_grasp() — and it writes working pick-and-place policies in one shot, with zero robot-specific training data.

That was the surprise in the code-as-policies papers, which landed in late 2022 around the same week as ChatGPT, before RL-trained coding agents existed, before most people understood how they worked. "Coding agents can do this very one shot," says Hanming. "They did not need additional robot data in order to work with this code, because they're already trained on so much coding data."

The model already understood sequencing, preconditions, and failure recovery — because those are exactly what programming requires. No teleoperation demonstrations needed.

Jay draws the sharpest comparison: VLAs were architecturally constrained to output an action immediately, with no room to think. "VLAs were basically — it has to output an action. I can't allocate more compute for a more complex task. And then now I have this code chain of thought thing that I can do." Code-as-policies gave robots the equivalent of scratch-paper: plan, handle variation, express conditional logic — things raw action prediction never permitted.

What nobody at the frontier labs intended was that the all-in investment on coding ability through 2024 — Dario and Sam both pouring resources into it, as Jay describes — was also, accidentally, a massive investment in robot control. The same improvements that made frontier models better at software engineering made them better at writing robot policies. One capability explosion, applied in a direction nobody had planned for.

Dragging cursors in Blender accidentally produced the world's best robotics training data

Astra's step-function improvement in spatial intelligence — documented in RoboCurve's benchmarks — didn't come from robot training data. It came from computer-use data: recordings of humans orbiting CAD objects in Blender, spinning models in SolidWorks, navigating 3D interfaces.

Hanming's explanation: "if you feed computer use data into a big model like Astra — and by computer use data, I mean like you drag a cursor around on a screen to orbit some CAD object in order to design in Blender — this tells you how to reason about spaces. It at least tells you about top down, left, right, all these concepts that you need to control a robot."

Research from Princeton pushed this further: if you design a harness where robot tools look like computer-use tools — literally having an agent drag a cursor to control where a robot arm goes — LLM performance on physical tasks improves measurably. The GUI metaphor wasn't just similar to the physical world. It was equivalent as training signal.

The historical arc is almost darkly funny. In the 1980s, software designers built GUIs to make computers feel like the physical world — files as folders, 3D objects you can orbit, drag-and-drop spatial metaphors. The goal was human usability. What they unknowingly built, over forty years at massive scale, was a proxy robotics dataset: humans interacting with software representations of physical space, millions of hours of it.

Frontier labs trained on that data. As Jay puts it: "we literally made the graphical user interface to be more like the physical world so that we could interface with it. And we ended up building an environment that was actually helpful for robotics to learn how to use the physical world." Nobody meant to. It worked anyway.

In-context learning maxes out at 40 examples — then adding more makes performance worse

After 20 to 40 examples, in-context learning saturates — and beyond the training context window, more examples actively degrade performance.

François has been running the experiment: take a held-out task, feed examples into context one by one, measure improvement on a validation set per sample. The curve does something uncomfortable. ICL works remarkably well in low-data regimes — each example genuinely helps, cheaply, with no gradient steps. But the improvement plateaus fast. "After like 20, 30, maybe 40 examples, it is basically saturated and more examples back into the context don't improve."

Then it inverts. Once you exceed the effective context window — roughly half the training length — "you don't improve anymore, you actually just get worse because the model can't attend over everything."

The reductio: "if you're Tesla and you have all the data, infinite data, and you're learning doing self-driving car by ICL — what are you doing? Like, are you kidding?"

The answer isn't to abandon ICL. It's to use it in the right regime and then compress. François's hierarchy runs: ICL works fast and cheap in low-data conditions → RAG pulls relevant examples on demand for larger knowledge bases → experience gets compressed into callable skills → and eventually into weight updates via LoRA or full fine-tuning. "It's amazing how good in low data regimes you can do with ICL, and then even cooler, we can compress the ICL into tool use, into tools."

That compression step is what the field hasn't fully built. Without it, in-context learning is a leaky bucket — fast to fill, never holds.

Years of slow VLA progress came from chasing the wrong data

The VLA data bottleneck isn't an architecture problem. It's a data-strategy mistake — and the teams chasing teleoperation data were looking in the wrong direction.

"People have been sort of nagging at this VLA data bottleneck for years now, and we're seeing very, very slow progress." The reason: robot teleoperation data is scarce, expensive, and narrow. Every improvement required more of it, collected by humans physically guiding robot arms through specific tasks in specific environments.

What code-as-policies and computer-use results demonstrate is that the physical world can be learned from many directions at once. Coding data teaches sequencing, tool use, and conditional logic — all of which transfer directly to robot control. CAD and GUI interaction teaches spatial reasoning. Egocentric video from humans doing everyday tasks teaches manipulation priors.

The team's Bitter Lesson framing: "We want to benefit from all kinds of data. We want to pour in computer-use data into our robot models. We want to pour encoding data into our robot models." But following that logic to its end produces something that looks less like a specialized robotics model and more like a general-purpose LLM. "When you do that, you just end up getting what we think of as these general-purpose LLM agents."

That's the point, not a side effect. Treating teleoperation data as the primary training signal, while sitting on mountains of coding and computer-use data that already teach the same underlying representations, was the slower path — by years.

Frontier consensus: general-purpose robots that do anything a teenager can do with their hands — within two years

Within two years, general-purpose robots will handle any task a competent teenager can do with their hands. This is current frontier-lab consensus — and the timeline is grounded in the same capability curves this episode traces.

"There's some consensus within the Frontier Labs and also in the Robotics Foundation models companies that we will have general purpose robots within the next two years. Even earlier, and this is something that society is probably unaware of, or even under prepared for."

The definition of 'general purpose' matters: "if you give any natural language instruction, it can do what a competent teenager could do with their hands." Not a narrow industrial robot with a fixed task library. A robot that walks into an unfamiliar environment and figures it out — the way a teenager who'd never assembled IKEA furniture before would work through it.

Jay frames it as "the ChatGPT moment for robotics in terms of capabilities where it can generalize to unseen tasks and unseen environments." Most people didn't believe language models were weeks away from that inflection point either.

Two convergences are driving the timeline simultaneously: base model improvements keep delivering step-function gains in spatial reasoning and instruction following, while harness infrastructure — skill libraries, code-as-policies graphs, ICL-to-weight compression — catches up to make those models economically deployable at scale.

LLM latency is halving every month — real-time robot control could land by end of 2026

The demo shows it plainly: the arm moves, pauses, moves again. Astra thinks between each step, and thinking costs time. At current latency, deploying an LLM-controlled robot at industrial scale doesn't pencil out.

But the trend line is steep. "For Fable-class LLMs, their latency is improving by around 2x per month, which is very, very fast. If the trends continue, we could get real-time control by the end of the year."

2x per month compounds faster than intuition tracks. The bottleneck that feels structural today dissolves as a constraint within months — potentially before the harness infrastructure to exploit it is even finished.

The architectural response is a two-tier system. Use a frontier model like Astra for the first pass: contextual learning, task understanding, policy generation. Then compress that into a faster, smaller skill that runs "repeatedly at incredibly high throughput" without the frontier model in the loop for every step. The frontier model handles novelty; the compiled skill handles repetition.

The right investment now is the compression pipeline — not waiting for latency to solve itself first.

The missing piece is a sleep phase — and whoever builds it first owns the next layer of the stack

Every insight in this episode converges on the same gap: experience accumulates in context, but context is bounded and leaky. ICL saturates at 40 examples. Context windows degrade past their training length. Skill libraries grow without principled pruning. The fix — a deliberate distillation step that compresses in-context experience back into weights or callable tools — is the architectural primitive nobody has fully built yet.

François puts it in biological terms: "almost everything that is intelligent sleeps. And then during sleep, compression happens." The analogy maps cleanly onto robot-agent systems: collect experience during operation, compress it into durable weight updates during a consolidation phase, resume faster and more capable.

The teams that build the sleep phase build the moat.

---

Topics: robotics, LLMs, robot control, code-as-policies, VLAs, vision language action models, in-context learning, Platonic Representation Hypothesis, AI agents, general-purpose robots, computer use, spatial reasoning, Waddle Labs, RoboCurve, Astra, RT2, foundation models