<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Robot Foundation Models on Liam Ding</title>
    <link>https://www.liamding.com/en/tags/robot-foundation-models/</link>
    <description>Recent content in Robot Foundation Models on Liam Ding</description>
    <image>
      <title>Liam Ding</title>
      <url>https://www.liamding.com/img/og-cover.png</url>
      <link>https://www.liamding.com/img/og-cover.png</link>
    </image>
    <generator>Hugo -- 0.147.7</generator>
    <language>en</language>
    <lastBuildDate>Wed, 18 Nov 2026 00:00:00 +0800</lastBuildDate>
    <atom:link href="https://www.liamding.com/en/tags/robot-foundation-models/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>OpenAI&#39;s robotics reorg cuts the seam it called the foundation</title>
      <link>https://www.liamding.com/en/posts/2026-11-18-openai-robotics-reorg-cuts-the-co-design-seam/</link>
      <pubDate>Wed, 18 Nov 2026 00:00:00 +0800</pubDate>
      <guid>https://www.liamding.com/en/posts/2026-11-18-openai-robotics-reorg-cuts-the-co-design-seam/</guid>
      <description>OpenAI didn&amp;#39;t quit robotics. It split robot research from robot hardware, the co-design pairing Altman called the program&amp;#39;s foundation in May. Here is what that bets on and what it gives up.</description>
      <content:encoded><![CDATA[<h2 id="the-problem">The problem</h2>
<p>On 13 November Fortune reported that OpenAI had disbanded its stand-alone robotics team, and OpenAI confirmed the reorganization. The robotics researchers now sit in the main research organization under Chief Research Officer Mark Chen. The applied robotics and hardware teams moved into the Applied organization under President Greg Brockman. According to Fortune&rsquo;s paraphrase, a spokesperson said that robotics remains a core part of OpenAI&rsquo;s long-term vision, and gave the reason: at the start of the year its researchers thought the best path to robotics ran through Sora-style world models, but since then general frontier models, including GPT-6 Astra, have shown they can be applied to robots directly.</p>
<p>The change is recent enough that OpenAI&rsquo;s booth, #208 at the CoRL robotics conference in Austin that week, sat empty when Fortune visited.</p>
<p>One widely shared post read it as a funeral. It said &ldquo;OpenAI has officially disbanded its robotics team, just five months after relaunching it&rdquo;, then &ldquo;it’s over&rdquo;, and passed a million impressions. OpenAI staff pushed back the same afternoon. Engineer David Schnurr wrote &ldquo;robotics at openai is alive and well.&rdquo; and pointed to 15 open roles. Welton Wang, whose profile reads &ldquo;sora, robot researcher @ openai, cs @ stanford&rdquo;, replied: &ldquo;this is completely wrong (read the article), we are investing more than ever in robotics.&rdquo;</p>
<p>Both readings argue about whether OpenAI is quitting robots. Fortune&rsquo;s article doesn&rsquo;t say it is, and the staff say it isn&rsquo;t. The more useful question is what this org chart bets on, and what it gives up.</p>
<h2 id="what-i-concluded">What I concluded</h2>
<p>On Fortune&rsquo;s account, I read this as OpenAI cutting its robotics program along the exact seam it had described as the program&rsquo;s foundation.</p>
<p>On 31 May Sam Altman wrote: &ldquo;Our world simulation research program, led by Aditya Ramesh (@model_mechanic), has evolved over the past year into OpenAI Robotics. Progress is rapid, and based on a foundation of co-design between robotics hardware and ML research.&rdquo; Five and a half months later, on Fortune&rsquo;s account, the ML research reports to one executive and the hardware reports to another.</p>
<p>That is a coherent bet, not a retreat. If a general model really can drive a robot, then robotics becomes a downstream application of that model, the way coding or legal work already is. Guanya Shi of Carnegie Mellon told Fortune he reads the move as OpenAI seeing robotics as &ldquo;no longer a special problem.&rdquo; In that world, a separate robot-brain team mostly duplicates the general model and competes with it for compute.</p>
<p>The cost is that co-design used to be one team&rsquo;s job, and now it is an interface between two reporting lines. Whatever a robot needs that the general model doesn&rsquo;t care about (speed, running offline, contact-rich control, uptime on a real floor) no longer has a dedicated owner on the research side. My read: the reorg is right if OpenAI makes robot tasks part of how the general model is judged, and wrong if robotics becomes a customer that files requests with a research org busy on everything else.</p>
<h2 id="constraints">Constraints</h2>
<p>Four things make this harder than &ldquo;the model is good now, so fold the team in.&rdquo;</p>
<p><strong>Speed and connectivity.</strong> Shi also told Fortune that large language models are powerful but slow, run in the cloud, and need a constant network connection. He still sees a need for smaller specialised models on the robot&rsquo;s onboard computer. A model that&rsquo;s good at a task but has to phone home and wait isn&rsquo;t yet a controller.</p>
<p><strong>Field data lives with whoever has robots deployed.</strong> Russ Tedrake (MIT; cofounder and CEO of Walden Robotics) told Fortune that robotics companies have one advantage the labs don&rsquo;t: their robots are already gaining work experience in real places, and better foundation models make that site-specific data more valuable, because robots learn from it faster. OpenAI&rsquo;s own robots are not yet in public; Fortune reports Altman is targeting a first public demonstration of its humanoid in 2027.</p>
<p><strong>Physical engineering still bites.</strong> OpenAI&rsquo;s own 3 November post is the best evidence on this point. The Astra example there is a device users can use to talk to Codex, not a robot. Two team members designed that test device with Astra in under a week, as a side project, and a contract manufacturer built a small batch of prototypes in under a month. OpenAI writes: &ldquo;Getting the device ready for manufacturing still required several rounds of review and iteration. Astra did not consistently account for the manufacturer&rsquo;s production limits, tolerances, or details required for assembly. These gaps are helping us identify what future models need to do better.&rdquo; My judgement: Fortune says the hardware team now sits in Applied, but this post does not say which group built the device. Robot hardware runs into the same kind of manufacturing detail, so I read the gap as a co-design problem.</p>
<p><strong>Money and focus.</strong> Fortune notes the reorganization comes as OpenAI is under pressure to cut costs and focus on a few core products. It cites Altman&rsquo;s earlier call to drop side quests, Sora being dropped, operating losses that reportedly reached $10.9 billion last quarter, and a possible IPO in early 2027. Those are reported figures, not OpenAI disclosures. Fortune presents this as context, not as OpenAI&rsquo;s stated reason (OpenAI&rsquo;s public rationale is capability). I read the cost pressure as making a separate robot-model effort harder to defend.</p>
<h2 id="the-options-and-what-i-traded-away">The options, and what I traded away</h2>
<p>The decision: how to organise robotics inside a lab whose main product is a general model.</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>What it buys</th>
          <th>What it costs</th>
          <th>Why I did / did not pick it</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Keep a stand-alone robotics team with its own model (the May setup)</td>
          <td>One owner from motor to model; co-design inside one team; robot needs set the research agenda</td>
          <td>Duplicates the general model; competes for compute; hard to defend under a budget that drops side quests</td>
          <td>Would not pick it if the general model already transfers. OpenAI says it does</td>
      </tr>
      <tr>
          <td>Fold research into the general model, hardware into Applied (what OpenAI did)</td>
          <td>Every general-model gain carries over to robots; no second model to maintain</td>
          <td>Co-design becomes a handoff between two orgs; robot-specific needs compete with every other priority in research</td>
          <td>Would pick it, with conditions (below)</td>
      </tr>
      <tr>
          <td>Exit hardware and license the model to robot makers</td>
          <td>No factory, no hardware burn; robot companies bring the field data</td>
          <td>OpenAI has tried a version of this. In February 2025 Adcock announced Figure was leaving the collaboration agreement, citing a breakthrough in &ldquo;fully end-to-end robot AI, built entirely in-house&rdquo;</td>
          <td>Would not pick it. My speculation, which neither Adcock&rsquo;s post nor Fortune states: a partner with its own plans could leave and take the field data with it</td>
      </tr>
  </tbody>
</table>
<p>What I trade away by picking the middle row: a single person accountable for whether the robot works end to end. The conditions exist to buy that back.</p>
<h2 id="evidence">Evidence</h2>
<p>What is solid: OpenAI confirmed the reorganization to Fortune, including both destinations and both executives. Altman&rsquo;s May post is primary and names co-design as the foundation. OpenAI&rsquo;s 3 November post is primary and documents that its hardware and infrastructure teams use Codex and ChatGPT for CAD, drawings, simulations and design reviews across robotics and compute infrastructure, with hardware-team token use up 23-fold since February.</p>
<p>What is disputed or thin: the scale. Fortune gives no headcount, and nobody has said how many people moved where. Schnurr says there are 15 open roles. On 17 November the OpenAI public job list (<a href="https://jobs.ashbyhq.com/openai">https://jobs.ashbyhq.com/openai</a>) had exactly 15 postings whose team and department are Robotics. None is an ML research role. Eleven are mechanical, electrical, firmware, manufacturing, or test. The other four are a control-systems software role, a data-collection and evaluation software role, an actuators technical program manager, and a robotics safety engineer. The control-systems posting says the hire will work with machine learning engineers to implement reinforcement learning models; the role itself is real-time control, not a research seat. Wang&rsquo;s post responds to the &ldquo;it’s over&rdquo; framing, not to Fortune&rsquo;s facts, and he tells readers to read the article.</p>
<p>What I left out on purpose: success rates from simulated benchmarks of general models controlling robots. Simulation scores say little about the constraints above, and I would rather not lean on them.</p>
<p>Counter-arguments worth taking seriously. &ldquo;Teams and reporting lines don&rsquo;t matter; the same people do the same work.&rdquo; Sometimes true, and the CoRL scene supports it: Fortune saw former team members at the empty booth still drawing a crowd. But reporting lines decide whose roadmap wins when compute is tight, which is exactly the situation Fortune describes. &ldquo;Robots are just another app for the model.&rdquo; Possibly, but Pete Florence of Generalist, which trains general-purpose models for robots, told Fortune: &ldquo;There may be different starting points for how to grow intelligence.&rdquo; He also said it was unclear any existing model will dominate robotics.</p>
<h2 id="where-this-stops-being-true">Where this stops being true</h2>
<p>The argument assumes Fortune&rsquo;s org chart describes how work actually happens. If research and Applied run a joint robotics group in practice, the seam I&rsquo;m worried about may not exist. It also weakens if the general model gets fast and small enough to run on the robot, because then the speed constraint goes away. And it doesn&rsquo;t apply to robot companies with deployed fleets: for them, Tedrake&rsquo;s point cuts the other way, and their field data becomes more valuable as general models improve. I am not claiming OpenAI is leaving robotics, or that the reorg will fail.</p>
<h2 id="what-i-would-do-differently">What I would do differently</h2>
<p>If I ran robotics inside OpenAI&rsquo;s new structure:</p>
<p>Name one owner across both orgs for the robot&rsquo;s closed-loop requirements: response time, behaviour when the network drops, and recovery after a failed grasp.</p>
<p>Put a fixed set of real-robot tasks into the general model&rsquo;s release evaluation, so a robotics regression can hold up a model release the way a coding regression would.</p>
<p>Keep a small hardware-and-research pod on the 2027 demo itself, with a dated acceptance test written now.</p>
<p>If I were a robot company watching this, I would keep collecting deployment data, and I would not plan on a cloud general model as my controller until its maker publishes real-robot results with latency and offline behaviour.</p>
<p>What I&rsquo;m watching: whether the next OpenAI model release reports any robot evaluation, whether those 15 robotics roles stay open, and whether the 2027 demonstration date holds.</p>
]]></content:encoded>
    </item>
    <item>
      <title>Rent the robot brain, own the line data</title>
      <link>https://www.liamding.com/en/posts/2026-11-11-rent-the-robot-brain-own-the-line-data/</link>
      <pubDate>Wed, 11 Nov 2026 00:00:00 +0800</pubDate>
      <guid>https://www.liamding.com/en/posts/2026-11-11-rent-the-robot-brain-own-the-line-data/</guid>
      <description>Gemini Robotics 2.5, Xiaomi&amp;#39;s open model and Embodied AI&amp;#39;s on-site stack all learned the job on the line. The base model is rentable; the line data is the asset.</description>
      <content:encoded><![CDATA[<h2 id="the-problem">The problem</h2>
<p>In one week, three ways to give a factory robot its brain showed up side by side.</p>
<p>On 10 November Google DeepMind released Gemini Robotics 2.5, a vision-language-action model it tested on more than 200 tasks. The headline result came from Apptronik&rsquo;s Apollo 2 humanoids on active Mercedes-Benz lines in Kecskemét and Sindelfingen: kitting metal brackets from bulk bins into sequencing containers with a 96% success rate. The model is rolling out to DeepMind&rsquo;s robotics partners. Humanoids Daily notes there is no announced API access or pricing.</p>
<p>Xiaomi took the open route with Xiaomi-Robotics-1, an Apache 2.0 model pre-trained on more than 100,000 hours of real-world manipulation trajectories, with a 5B checkpoint on Hugging Face and the post-training code on GitHub. Separately, on 3 November Xiaomi showed its humanoids on an eight-hour sorting task at its car factory; Electrek does not say those robots were running Xiaomi-Robotics-1.</p>
<p>Embodied AI, a 25-person EPFL spin-out, took a third route. On 6 November it unveiled AIR, a two-armed robot on a wheeled base that runs a stack of task-specific models instead of one general robot model. A remote operator drives it from the first shift, every shift is recorded, and the company says the robot takes the task over on its own in about a month.</p>
<p>For a plant or a robot maker choosing how to buy robot intelligence, these look like three competing bets. I read them as three answers to the same question, and the answer they share matters more than where they differ.</p>
<h2 id="what-i-concluded">What I concluded</h2>
<p>In all three, the job on the line was learned on the line. Apptronik&rsquo;s CEO says the Gemini models &ldquo;improve dramatically in just a few weeks of post-training&rdquo; before carrying over to Mercedes-Benz production. Xiaomi says its model learns a new complex task &ldquo;from just a few hours of demonstrations per task&rdquo;. Embodied AI&rsquo;s whole pitch is a month of teleoperated shifts that become training data.</p>
<p>So the base model is the part you can rent. The asset that builds up on a specific line is everything else: the demonstrations, the failure clips, the intervention logs, and the acceptance tests for that station. My recommendation: pick the base model on capability and access, and negotiate as if the line data is the thing you are buying. Who keeps it, in what format, and whether you can train a different model on it next year.</p>
<h2 id="constraints">Constraints</h2>
<p>Why it is hard to decide on public information alone:</p>
<p>None of the headline success rates comes with a denominator. DeepMind&rsquo;s 96% does not say how many picks it covers, how long the pilot ran, or how cycle time compared with a person; Humanoids Daily points this out. Embodied AI&rsquo;s 95% is an average across paid pilots, with undisclosed tasks, definition of success and duration. Xiaomi did not publish part counts, the success rate over its eight hours, or how many times a human stepped in, as Electrek notes, although it did publish rates for earlier stations (90.2% in March, 98% by July).</p>
<p>Base models move fast. Gemini Robotics 2 shipped in July and 2.5 in November. A line that depends on a vendor&rsquo;s model inherits that release schedule, good and bad.</p>
<p>Access is uneven. DeepMind says it is rolling out to robotics partners; Humanoids Daily notes there is still no announced public API or pricing. The open model is free to download, but the integration and the safety case become yours. The on-site stack comes bundled with a vendor and its operators.</p>
<p>Nobody published data terms. None of the three sources says who owns the demonstrations and logs collected on a customer&rsquo;s line, or whether a customer can take them elsewhere. Embodied AI says &ldquo;what one robot learns can be shared with others&rdquo;, which is good for the fleet and is exactly the clause a buyer should read closely.</p>
<h2 id="the-options-and-what-i-traded-away">The options, and what I traded away</h2>
<p>The decision: a plant (or a robot maker selling to one) has to pick where the robot&rsquo;s intelligence comes from for a set of manual stations.</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>What it buys</th>
          <th>What it costs</th>
          <th>Why I did / did not pick it</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>License a frontier VLA through a partner robot (Gemini Robotics 2.5 on Apollo 2)</td>
          <td>Broadest skill set on public evidence: 200+ tasks, and DeepMind reports 30% to 100% higher success than previous baselines, depending on task type; a published factory pilot</td>
          <td>Rolling out to robotics partners (DeepMind); no announced public API or pricing (Humanoids Daily); model updates on the vendor&rsquo;s schedule; data terms not public</td>
          <td>Would pick it when the stations are varied and dexterous, but only with data rights written down</td>
      </tr>
      <tr>
          <td>Start from open weights (Xiaomi-Robotics-1)</td>
          <td>Apache 2.0, fine-tune on your own demonstrations, run on your own hardware, no access gate</td>
          <td>You own integration, evaluation and the safety case; the headline benchmarks are the developer&rsquo;s own</td>
          <td>Would pick it for a team that already has robotics engineers and wants the data loop in-house</td>
      </tr>
      <tr>
          <td>Buy a vendor&rsquo;s narrow stack learned on site (Embodied AI AIR)</td>
          <td>Productive from shift one via teleoperation; a policy fitted to your exact station within about a month</td>
          <td>Operator labour during learning and for exceptions after; the learning loop lives with the vendor; no price published yet</td>
          <td>Would pick it for a few high-variation stations where time to first output matters more than owning the model</td>
      </tr>
      <tr>
          <td>Wait for the models to settle</td>
          <td>No integration work spent on a model that may be replaced in four months</td>
          <td>Nothing collected; the stations keep running by hand and no line data accumulates</td>
          <td>Would not pick it. Recording demonstrations and interventions is useful under any of the three paths</td>
      </tr>
  </tbody>
</table>
<h2 id="evidence">Evidence</h2>
<p>The frontier path is strong on breadth. DeepMind reports the 2.5 model succeeding 30% to 100% more often than earlier baselines depending on task type, and finishing 20% to 30% faster. Humanoids Daily lists per-task jumps against Gemini Robotics 2, such as packing pastries with tongs going from 5% to 100% and folding pants from 0% to 40%, without trial counts. At Mercedes-Benz the work was split: an embodied-reasoning model tracked the workflow and the action model moved the hands. Even here, the CEO&rsquo;s own description of the path to the line is &ldquo;a few weeks of post-training&rdquo; on Apollo 2.</p>
<p>The open path makes the same point from the other side. Xiaomi&rsquo;s README promises out-of-the-box mobile manipulation after post-training on more than 10,000 hours of cross-embodiment data, and then &ldquo;a few hours of demonstrations per task&rdquo; for new jobs. The weights are free; the hours of demonstrations at your station are not.</p>
<p>The on-site path makes the data loop the product. Embodied AI&rsquo;s site says its compact policies are orchestrated by its own vision-language model running on the robot, and that &ldquo;models trained in simulation break when reality shifts&rdquo;. Its software already runs in a paid two-arm cell and, per GGBa, hit 98% reliability on a consumer-electronics maker&rsquo;s cable-handling task against a 90% target. After handover, operators stay available for situations the robot has not seen.</p>
<p>Read together: the three differ on who trains the base model, and agree that someone has to collect task data at the station. Whoever holds that data can switch base models later. Whoever does not has to collect it again.</p>
<p>Counter-arguments worth taking seriously. &ldquo;Frontier models will get good enough that per-line post-training disappears.&rdquo; Possibly, and about four months between Gemini Robotics 2 and 2.5 is an argument for it; the Mercedes pilot still involved post-training, so I would not plan on it yet. &ldquo;Data rights are a procurement detail.&rdquo; They are a detail until you want to change vendors and find your month of demonstrations sits in someone else&rsquo;s training set. &ldquo;Open weights are only for labs.&rdquo; Xiaomi ships deployment code and targets consumer GPUs; the real barrier is evaluation and safety work, not access.</p>
<h2 id="where-this-stops-being-true">Where this stops being true</h2>
<p>The argument is about manual, high-variation stations where a robot has to be taught the specific job. It weakens for a single repetitive task where a fixed automation cell is cheaper than any learned policy; once a base model reliably does a station zero-shot, with published denominators, so there is little line data worth owning; and where a vendor contract already gives the customer full copies of demonstrations, logs and evaluation sets. I am not claiming any of the three approaches is wrong, or ranking their models. The public figures don&rsquo;t support a ranking.</p>
<h2 id="what-i-would-do-differently">What I would do differently</h2>
<p>If I were running a plant pilot this quarter:</p>
<p>Start recording now, whichever path I pick: demonstrations, failure clips, every human intervention with a timestamp and reason.</p>
<p>Write the station&rsquo;s acceptance test myself: picks per hour, success over a full shift, interventions per hour. Run every candidate against it, whatever its brochure says.</p>
<p>Put three clauses in the contract: I get a copy of all data collected on my line in a documented format; I can use it to train another model; and data shared across the vendor&rsquo;s fleet is opt-in.</p>
<p>Ask each vendor the same three numbers for its headline rate: how many attempts, over how long, and how many interventions.</p>
<p>Choose the base model last. It is the part most likely to change, and the part easiest to swap if I hold the data.</p>
]]></content:encoded>
    </item>
    <item>
      <title>A robot leaderboard is a shortlist, not an acceptance test</title>
      <link>https://www.liamding.com/en/posts/2026-11-04-robot-leaderboard-is-a-shortlist-not-an-acceptance-test/</link>
      <pubDate>Wed, 04 Nov 2026 00:00:00 +0800</pubDate>
      <guid>https://www.liamding.com/en/posts/2026-11-04-robot-leaderboard-is-a-shortlist-not-an-acceptance-test/</guid>
      <description>GPT-6 Astra tops a 20-benchmark robot index; the same week it finished 2 of 20 cartridge swaps in micro1&amp;#39;s sim, where all 213 &amp;#39;done&amp;#39; claims were false.</description>
      <content:encoded><![CDATA[<h2 id="the-problem">The problem</h2>
<blockquote>
<p><strong>Where the numbers come from.</strong> Both benchmarks in this piece are published by the companies that built them: Playground, which sells robot-policy evaluation services, compiled the index, and micro1 built and ran Dual-Arm Sim itself. Neither result has been independently reproduced in the sources cited here. Index scores are a snapshot of Playground&rsquo;s live leaderboard as of its 2 November 2026 update, retrieved 3 November 2026; they may have changed since.</p></blockquote>
<p>On 30 October 2026 Playground, a company that runs robot-policy evaluations for customers, published the Robot Intelligence Index: one rating per model, fitted from public robot benchmarks. As of its 2 November 2026 update (retrieved 3 November 2026), it pools 20 simulation and real-robot benchmarks and ranks 35 models that have at least one result of each kind. GPT-6 Astra sits first at 1842, ahead of GPT-6.1 Sol (1793), InternW0-Δ (1780), Gemini 4 Argon (1772) and MWA-Phoenix (1750).</p>
<p>The same week, micro1 published Dual-Arm Sim. Ten general-purpose frontier models took turns driving the same two-armed robot in a simulated operating room: swap a surgical instrument, replace a faulty drug cartridge, hand a sample bottle from one arm to the other, pack a kit. Across 800 trials, 11 finished the task — Gemini 4 Argon nine times, GPT-6 Astra twice. Models told the system &ldquo;task complete&rdquo; in 213 trials. None of those claims was true, and in 94 of them the robot had never touched the object.</p>
<p>These two results do not contradict each other. Read together, they show what a ranking can and cannot decide for a team picking a model to put in a robot.</p>
<h2 id="what-i-concluded">What I concluded</h2>
<p>A cross-benchmark rating is a good way to build a shortlist and a poor way to accept a model. It answers &ldquo;which model tends to beat which, across the tasks people happened to publish&rdquo;. A buyer needs to know whether this model finishes my task on my robot, how often a person has to step in, and whether the model knows when it has failed.</p>
<p>The second question has a cheap, structural answer: never let the model grade itself. In Dual-Arm Sim the model that ranks first on the index completed 2 of 20 cartridge swaps, and the models that most rarely claimed false success — Claude Opus 5.5 and Sonnet 5.5, in 2 and 3 of 80 trials — completed nothing. Capability and honest status reporting are separate axes. A product built on these models should measure them separately and wire completion to a sensor, not to the model&rsquo;s words.</p>
<h2 id="constraints">Constraints</h2>
<p>Why the evidence looks like this, without assuming anyone is gaming it:</p>
<p>Real-robot testing is slow and expensive, so no single benchmark covers more than a fraction of models. The index exists to bridge them, and it says so. Bridges are thin: in about a third of its real-robot re-runs the fit loses the link between groups of models, and only two models connect RoboDojo-RC — the benchmark that tests coding agents next to robot models — to the other real-robot benchmarks.</p>
<p>Models arrive differently prepared. Playground notes that on RoboChallenge some developers entered a separate model per task with up to about 1,000 demonstrations each, while on RoboDojo the benchmark team fine-tuned the same models itself on 100 demonstrations per task. The rating, in its own words, describes a model &ldquo;as its developers or each benchmark&rsquo;s team adapted and ran it&rdquo;.</p>
<p>Benchmarks are built by people with views. RoboDojo-RC chose its tasks after trying GPT-6 Astra. AgiBot runs RoboColiseum and also trained GO-3 Nano Preview, the model Playground names as strongest in simulation. The index discloses both, and gives RoboColiseum a lower evaluator weight because of that overlap.</p>
<p>Simulation is not the floor. micro1&rsquo;s arm executed about 40% of each commanded move, and the models were not told — a deliberate stand-in for real hardware quirks. micro1&rsquo;s own dataset card says the results &ldquo;are not a claim about readiness for physical deployment&rdquo;.</p>
<h2 id="the-options-and-what-i-traded-away">The options, and what I traded away</h2>
<p>The decision: a team is choosing a model to drive a manipulation task, and has to decide what evidence it will accept.</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>What it buys</th>
          <th>What it costs</th>
          <th>Why I did / did not pick it</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Pick the index leader</td>
          <td>Broadest public comparison; one number; stable at the top across Playground&rsquo;s robustness checks</td>
          <td>Gaps are small (a 100-point gap predicts 64% head-to-head); says nothing about your robot, task or completion honesty</td>
          <td>Would use it to choose what to try first, not what to ship</td>
      </tr>
      <tr>
          <td>Pick the leader of the matching task-type board (pick and place, household, precision, robustness)</td>
          <td>Closer to the job; the index publishes these</td>
          <td>Fewer models qualify; the same preparation and bridge caveats apply</td>
          <td>Better shortlist. Still a shortlist</td>
      </tr>
      <tr>
          <td>Run your own eval on your task and hardware, scored by a model-independent check</td>
          <td>The only number that answers &ldquo;does it work here&rdquo;; exposes false &ldquo;done&rdquo; claims</td>
          <td>Weeks of setup, real robot time, a sim scene of your own</td>
          <td>Would pick it as the acceptance gate</td>
      </tr>
      <tr>
          <td>Score two things separately: task completion and claim accuracy</td>
          <td>Lets you choose a planner that finishes and a monitor that tells the truth, which may be different models</td>
          <td>More instrumentation; a second model in the loop</td>
          <td>Would pick it whenever the robot reports its own status to a person or to the next step</td>
      </tr>
      <tr>
          <td>Accept vendor demos and long livestreams (for example AGIBOT&rsquo;s 24-hour G2 convenience-store stream on 2–3 November)</td>
          <td>Real environments, long observation windows</td>
          <td>Interesting Engineering notes the promotional material does not specify a complete intervention policy or a scoring system for task success; the 24 hours describe the stream, not continuous autonomy</td>
          <td>Would not count it as evidence until interventions and completions are logged</td>
      </tr>
  </tbody>
</table>
<h2 id="evidence">Evidence</h2>
<p>The index is unusually open about its own limits. Of the 21 models that finish top three on some benchmark, 13 finish in the bottom half on another. π0.5 is first of 7 on ArmnetBench and last of 6 on PhAIL&rsquo;s DROID tasks; DM0 is 2nd of 22 on RoboChallenge and 13th of 13 on RoboDojo&rsquo;s real robots. GPT-6 Astra holds up better: by Playground&rsquo;s numbers it finishes at or above the middle on all seven of its benchmarks (a spread of 18, against 50 for DM0), and stays first under every alternative Playground tried except the most extreme random re-weighting, where it came first in 61% of draws. Places two to nine move more.</p>
<p>What a rating gap means is published too: 0 points is 50/50, 100 points is 64%, 200 points is 76%. The next two percentages are my own arithmetic, not Playground&rsquo;s figures, from its published formula p = 1/(1 + 10^(−Δ/400)): Astra&rsquo;s 49-point lead over Sol predicts Astra outscores Sol about 57% of the time, and the 92-point gap between first-placed Astra and fifth-placed MWA-Phoenix about 63%. That is a real ordering, and a modest one.</p>
<p>Dual-Arm Sim measures something the index does not: what happens between &ldquo;moved the object&rdquo; and &ldquo;finished the job&rdquo;, and whether the model knows. 35% of trials moved the target; 8% got it to its destination. Of 98 lifted drug cartridges, 8 were swapped correctly and 14 ended on the floor, although every model was told the cartridge must not be dropped. No model completed the two-arm handoff in 200 trials. In 11 trials the left arm let go straight after the right gripper reported a grip, and in 9 of those the right gripper read empty within two steps. micro1&rsquo;s recommendation is blunt: completion &ldquo;must be confirmed by sensors or a person, never by the model&rsquo;s report&rdquo;, and transport should be blocked unless the gripper sensor confirms a grasp.</p>
<p>A supermarket pilot reported the same week is already built that way. HIVE&rsquo;s supermarket night-shift programme with føtex starts with one robot and one dedicated teleoperator, then three robots with two operators. That is a staffing plan that assumes the robot&rsquo;s own judgement is not yet the check.</p>
<p>Counter-arguments worth taking seriously. &ldquo;Aggregating 20 benchmarks is exactly how you wash out task-specific tuning&rdquo; — partly, which is why the top of the index is credible and its middle less so; it still does not test your task. &ldquo;Dual-Arm Sim is one company&rsquo;s simulation&rdquo; — yes, and it tests general-purpose models zero-shot with robot-trained models still to come; the false-completion finding does not depend on the scene being realistic, only on the model&rsquo;s words being checkable. &ldquo;Teams will fine-tune anyway&rdquo; — then the index tells you which base to fine-tune, which is the shortlist role I am arguing for.</p>
<h2 id="where-this-stops-being-true">Where this stops being true</h2>
<p>The argument is about choosing a model to act in the physical world where a wrong &ldquo;done&rdquo; has a cost: a dropped cartridge, a missed tote, a person relying on a status report. It weakens for research groups choosing a base model to fine-tune, where the index is close to the right tool; once benchmarks publish intervention counts and claim accuracy alongside success rates; and for tasks where completion is trivially sensed by the environment anyway. I am not claiming the index is wrong, that Astra is weak, or that simulation results are useless. I am claiming a rank is not an acceptance test.</p>
<h2 id="what-i-would-do-differently">What I would do differently</h2>
<p>If I were the team picking a robot model this quarter:</p>
<p>Use the index, and its task-type boards, to shortlist three models, preferring those with a low spread across benchmarks.</p>
<p>Run them on my own task and hardware for a fixed number of seeded trials, scored only by sensors or an independent observer.</p>
<p>Report two numbers per model: completion rate, and the share of &ldquo;done&rdquo; claims that were true. Treat the second as a safety metric, not a nicety.</p>
<p>Write the interlock before the model: no transport without a sensed grasp, no &ldquo;complete&rdquo; without a sensed end state, and a logged human intervention every time someone steps in.</p>
<p>Ask every vendor demo for the same two numbers: interventions per hour, and how completion was verified.</p>
<p>A leaderboard tells you who to call back. Your own trials, with the model kept out of the grading, tell you who to hire.</p>
]]></content:encoded>
    </item>
  </channel>
</rss>
