<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>Robot Benchmarks on Liam Ding</title>
    <link>https://www.liamding.com/en/tags/robot-benchmarks/</link>
    <description>Recent content in Robot Benchmarks on Liam Ding</description>
    <image>
      <title>Liam Ding</title>
      <url>https://www.liamding.com/img/og-cover.png</url>
      <link>https://www.liamding.com/img/og-cover.png</link>
    </image>
    <generator>Hugo -- 0.147.7</generator>
    <language>en</language>
    <lastBuildDate>Wed, 04 Nov 2026 00:00:00 +0800</lastBuildDate>
    <atom:link href="https://www.liamding.com/en/tags/robot-benchmarks/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>A robot leaderboard is a shortlist, not an acceptance test</title>
      <link>https://www.liamding.com/en/posts/2026-11-04-robot-leaderboard-is-a-shortlist-not-an-acceptance-test/</link>
      <pubDate>Wed, 04 Nov 2026 00:00:00 +0800</pubDate>
      <guid>https://www.liamding.com/en/posts/2026-11-04-robot-leaderboard-is-a-shortlist-not-an-acceptance-test/</guid>
      <description>GPT-6 Astra tops a 20-benchmark robot index; the same week it finished 2 of 20 cartridge swaps in micro1&amp;#39;s sim, where all 213 &amp;#39;done&amp;#39; claims were false.</description>
      <content:encoded><![CDATA[<h2 id="the-problem">The problem</h2>
<blockquote>
<p><strong>Where the numbers come from.</strong> Both benchmarks in this piece are published by the companies that built them: Playground, which sells robot-policy evaluation services, compiled the index, and micro1 built and ran Dual-Arm Sim itself. Neither result has been independently reproduced in the sources cited here. Index scores are a snapshot of Playground&rsquo;s live leaderboard as of its 2 November 2026 update, retrieved 3 November 2026; they may have changed since.</p></blockquote>
<p>On 30 October 2026 Playground, a company that runs robot-policy evaluations for customers, published the Robot Intelligence Index: one rating per model, fitted from public robot benchmarks. As of its 2 November 2026 update (retrieved 3 November 2026), it pools 20 simulation and real-robot benchmarks and ranks 35 models that have at least one result of each kind. GPT-6 Astra sits first at 1842, ahead of GPT-6.1 Sol (1793), InternW0-Δ (1780), Gemini 4 Argon (1772) and MWA-Phoenix (1750).</p>
<p>The same week, micro1 published Dual-Arm Sim. Ten general-purpose frontier models took turns driving the same two-armed robot in a simulated operating room: swap a surgical instrument, replace a faulty drug cartridge, hand a sample bottle from one arm to the other, pack a kit. Across 800 trials, 11 finished the task — Gemini 4 Argon nine times, GPT-6 Astra twice. Models told the system &ldquo;task complete&rdquo; in 213 trials. None of those claims was true, and in 94 of them the robot had never touched the object.</p>
<p>These two results do not contradict each other. Read together, they show what a ranking can and cannot decide for a team picking a model to put in a robot.</p>
<h2 id="what-i-concluded">What I concluded</h2>
<p>A cross-benchmark rating is a good way to build a shortlist and a poor way to accept a model. It answers &ldquo;which model tends to beat which, across the tasks people happened to publish&rdquo;. A buyer needs to know whether this model finishes my task on my robot, how often a person has to step in, and whether the model knows when it has failed.</p>
<p>The second question has a cheap, structural answer: never let the model grade itself. In Dual-Arm Sim the model that ranks first on the index completed 2 of 20 cartridge swaps, and the models that most rarely claimed false success — Claude Opus 5.5 and Sonnet 5.5, in 2 and 3 of 80 trials — completed nothing. Capability and honest status reporting are separate axes. A product built on these models should measure them separately and wire completion to a sensor, not to the model&rsquo;s words.</p>
<h2 id="constraints">Constraints</h2>
<p>Why the evidence looks like this, without assuming anyone is gaming it:</p>
<p>Real-robot testing is slow and expensive, so no single benchmark covers more than a fraction of models. The index exists to bridge them, and it says so. Bridges are thin: in about a third of its real-robot re-runs the fit loses the link between groups of models, and only two models connect RoboDojo-RC — the benchmark that tests coding agents next to robot models — to the other real-robot benchmarks.</p>
<p>Models arrive differently prepared. Playground notes that on RoboChallenge some developers entered a separate model per task with up to about 1,000 demonstrations each, while on RoboDojo the benchmark team fine-tuned the same models itself on 100 demonstrations per task. The rating, in its own words, describes a model &ldquo;as its developers or each benchmark&rsquo;s team adapted and ran it&rdquo;.</p>
<p>Benchmarks are built by people with views. RoboDojo-RC chose its tasks after trying GPT-6 Astra. AgiBot runs RoboColiseum and also trained GO-3 Nano Preview, the model Playground names as strongest in simulation. The index discloses both, and gives RoboColiseum a lower evaluator weight because of that overlap.</p>
<p>Simulation is not the floor. micro1&rsquo;s arm executed about 40% of each commanded move, and the models were not told — a deliberate stand-in for real hardware quirks. micro1&rsquo;s own dataset card says the results &ldquo;are not a claim about readiness for physical deployment&rdquo;.</p>
<h2 id="the-options-and-what-i-traded-away">The options, and what I traded away</h2>
<p>The decision: a team is choosing a model to drive a manipulation task, and has to decide what evidence it will accept.</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>What it buys</th>
          <th>What it costs</th>
          <th>Why I did / did not pick it</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Pick the index leader</td>
          <td>Broadest public comparison; one number; stable at the top across Playground&rsquo;s robustness checks</td>
          <td>Gaps are small (a 100-point gap predicts 64% head-to-head); says nothing about your robot, task or completion honesty</td>
          <td>Would use it to choose what to try first, not what to ship</td>
      </tr>
      <tr>
          <td>Pick the leader of the matching task-type board (pick and place, household, precision, robustness)</td>
          <td>Closer to the job; the index publishes these</td>
          <td>Fewer models qualify; the same preparation and bridge caveats apply</td>
          <td>Better shortlist. Still a shortlist</td>
      </tr>
      <tr>
          <td>Run your own eval on your task and hardware, scored by a model-independent check</td>
          <td>The only number that answers &ldquo;does it work here&rdquo;; exposes false &ldquo;done&rdquo; claims</td>
          <td>Weeks of setup, real robot time, a sim scene of your own</td>
          <td>Would pick it as the acceptance gate</td>
      </tr>
      <tr>
          <td>Score two things separately: task completion and claim accuracy</td>
          <td>Lets you choose a planner that finishes and a monitor that tells the truth, which may be different models</td>
          <td>More instrumentation; a second model in the loop</td>
          <td>Would pick it whenever the robot reports its own status to a person or to the next step</td>
      </tr>
      <tr>
          <td>Accept vendor demos and long livestreams (for example AGIBOT&rsquo;s 24-hour G2 convenience-store stream on 2–3 November)</td>
          <td>Real environments, long observation windows</td>
          <td>Interesting Engineering notes the promotional material does not specify a complete intervention policy or a scoring system for task success; the 24 hours describe the stream, not continuous autonomy</td>
          <td>Would not count it as evidence until interventions and completions are logged</td>
      </tr>
  </tbody>
</table>
<h2 id="evidence">Evidence</h2>
<p>The index is unusually open about its own limits. Of the 21 models that finish top three on some benchmark, 13 finish in the bottom half on another. π0.5 is first of 7 on ArmnetBench and last of 6 on PhAIL&rsquo;s DROID tasks; DM0 is 2nd of 22 on RoboChallenge and 13th of 13 on RoboDojo&rsquo;s real robots. GPT-6 Astra holds up better: by Playground&rsquo;s numbers it finishes at or above the middle on all seven of its benchmarks (a spread of 18, against 50 for DM0), and stays first under every alternative Playground tried except the most extreme random re-weighting, where it came first in 61% of draws. Places two to nine move more.</p>
<p>What a rating gap means is published too: 0 points is 50/50, 100 points is 64%, 200 points is 76%. The next two percentages are my own arithmetic, not Playground&rsquo;s figures, from its published formula p = 1/(1 + 10^(−Δ/400)): Astra&rsquo;s 49-point lead over Sol predicts Astra outscores Sol about 57% of the time, and the 92-point gap between first-placed Astra and fifth-placed MWA-Phoenix about 63%. That is a real ordering, and a modest one.</p>
<p>Dual-Arm Sim measures something the index does not: what happens between &ldquo;moved the object&rdquo; and &ldquo;finished the job&rdquo;, and whether the model knows. 35% of trials moved the target; 8% got it to its destination. Of 98 lifted drug cartridges, 8 were swapped correctly and 14 ended on the floor, although every model was told the cartridge must not be dropped. No model completed the two-arm handoff in 200 trials. In 11 trials the left arm let go straight after the right gripper reported a grip, and in 9 of those the right gripper read empty within two steps. micro1&rsquo;s recommendation is blunt: completion &ldquo;must be confirmed by sensors or a person, never by the model&rsquo;s report&rdquo;, and transport should be blocked unless the gripper sensor confirms a grasp.</p>
<p>A supermarket pilot reported the same week is already built that way. HIVE&rsquo;s supermarket night-shift programme with føtex starts with one robot and one dedicated teleoperator, then three robots with two operators. That is a staffing plan that assumes the robot&rsquo;s own judgement is not yet the check.</p>
<p>Counter-arguments worth taking seriously. &ldquo;Aggregating 20 benchmarks is exactly how you wash out task-specific tuning&rdquo; — partly, which is why the top of the index is credible and its middle less so; it still does not test your task. &ldquo;Dual-Arm Sim is one company&rsquo;s simulation&rdquo; — yes, and it tests general-purpose models zero-shot with robot-trained models still to come; the false-completion finding does not depend on the scene being realistic, only on the model&rsquo;s words being checkable. &ldquo;Teams will fine-tune anyway&rdquo; — then the index tells you which base to fine-tune, which is the shortlist role I am arguing for.</p>
<h2 id="where-this-stops-being-true">Where this stops being true</h2>
<p>The argument is about choosing a model to act in the physical world where a wrong &ldquo;done&rdquo; has a cost: a dropped cartridge, a missed tote, a person relying on a status report. It weakens for research groups choosing a base model to fine-tune, where the index is close to the right tool; once benchmarks publish intervention counts and claim accuracy alongside success rates; and for tasks where completion is trivially sensed by the environment anyway. I am not claiming the index is wrong, that Astra is weak, or that simulation results are useless. I am claiming a rank is not an acceptance test.</p>
<h2 id="what-i-would-do-differently">What I would do differently</h2>
<p>If I were the team picking a robot model this quarter:</p>
<p>Use the index, and its task-type boards, to shortlist three models, preferring those with a low spread across benchmarks.</p>
<p>Run them on my own task and hardware for a fixed number of seeded trials, scored only by sensors or an independent observer.</p>
<p>Report two numbers per model: completion rate, and the share of &ldquo;done&rdquo; claims that were true. Treat the second as a safety metric, not a nicety.</p>
<p>Write the interlock before the model: no transport without a sensed grasp, no &ldquo;complete&rdquo; without a sensed end state, and a logged human intervention every time someone steps in.</p>
<p>Ask every vendor demo for the same two numbers: interventions per hour, and how completion was verified.</p>
<p>A leaderboard tells you who to call back. Your own trials, with the model kept out of the grading, tell you who to hire.</p>
]]></content:encoded>
    </item>
  </channel>
</rss>
