<?xml version="1.0" encoding="utf-8" standalone="yes"?>
<rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom" xmlns:content="http://purl.org/rss/1.0/modules/content/">
  <channel>
    <title>VLA on Liam Ding</title>
    <link>https://www.liamding.com/en/tags/vla/</link>
    <description>Recent content in VLA on Liam Ding</description>
    <image>
      <title>Liam Ding</title>
      <url>https://www.liamding.com/img/og-cover.png</url>
      <link>https://www.liamding.com/img/og-cover.png</link>
    </image>
    <generator>Hugo -- 0.147.7</generator>
    <language>en</language>
    <lastBuildDate>Wed, 11 Nov 2026 00:00:00 +0800</lastBuildDate>
    <atom:link href="https://www.liamding.com/en/tags/vla/index.xml" rel="self" type="application/rss+xml" />
    <item>
      <title>Rent the robot brain, own the line data</title>
      <link>https://www.liamding.com/en/posts/2026-11-11-rent-the-robot-brain-own-the-line-data/</link>
      <pubDate>Wed, 11 Nov 2026 00:00:00 +0800</pubDate>
      <guid>https://www.liamding.com/en/posts/2026-11-11-rent-the-robot-brain-own-the-line-data/</guid>
      <description>Gemini Robotics 2.5, Xiaomi&amp;#39;s open model and Embodied AI&amp;#39;s on-site stack all learned the job on the line. The base model is rentable; the line data is the asset.</description>
      <content:encoded><![CDATA[<h2 id="the-problem">The problem</h2>
<p>In one week, three ways to give a factory robot its brain showed up side by side.</p>
<p>On 10 November Google DeepMind released Gemini Robotics 2.5, a vision-language-action model it tested on more than 200 tasks. The headline result came from Apptronik&rsquo;s Apollo 2 humanoids on active Mercedes-Benz lines in Kecskemét and Sindelfingen: kitting metal brackets from bulk bins into sequencing containers with a 96% success rate. The model is rolling out to DeepMind&rsquo;s robotics partners. Humanoids Daily notes there is no announced API access or pricing.</p>
<p>Xiaomi took the open route with Xiaomi-Robotics-1, an Apache 2.0 model pre-trained on more than 100,000 hours of real-world manipulation trajectories, with a 5B checkpoint on Hugging Face and the post-training code on GitHub. Separately, on 3 November Xiaomi showed its humanoids on an eight-hour sorting task at its car factory; Electrek does not say those robots were running Xiaomi-Robotics-1.</p>
<p>Embodied AI, a 25-person EPFL spin-out, took a third route. On 6 November it unveiled AIR, a two-armed robot on a wheeled base that runs a stack of task-specific models instead of one general robot model. A remote operator drives it from the first shift, every shift is recorded, and the company says the robot takes the task over on its own in about a month.</p>
<p>For a plant or a robot maker choosing how to buy robot intelligence, these look like three competing bets. I read them as three answers to the same question, and the answer they share matters more than where they differ.</p>
<h2 id="what-i-concluded">What I concluded</h2>
<p>In all three, the job on the line was learned on the line. Apptronik&rsquo;s CEO says the Gemini models &ldquo;improve dramatically in just a few weeks of post-training&rdquo; before carrying over to Mercedes-Benz production. Xiaomi says its model learns a new complex task &ldquo;from just a few hours of demonstrations per task&rdquo;. Embodied AI&rsquo;s whole pitch is a month of teleoperated shifts that become training data.</p>
<p>So the base model is the part you can rent. The asset that builds up on a specific line is everything else: the demonstrations, the failure clips, the intervention logs, and the acceptance tests for that station. My recommendation: pick the base model on capability and access, and negotiate as if the line data is the thing you are buying. Who keeps it, in what format, and whether you can train a different model on it next year.</p>
<h2 id="constraints">Constraints</h2>
<p>Why it is hard to decide on public information alone:</p>
<p>None of the headline success rates comes with a denominator. DeepMind&rsquo;s 96% does not say how many picks it covers, how long the pilot ran, or how cycle time compared with a person; Humanoids Daily points this out. Embodied AI&rsquo;s 95% is an average across paid pilots, with undisclosed tasks, definition of success and duration. Xiaomi did not publish part counts, the success rate over its eight hours, or how many times a human stepped in, as Electrek notes, although it did publish rates for earlier stations (90.2% in March, 98% by July).</p>
<p>Base models move fast. Gemini Robotics 2 shipped in July and 2.5 in November. A line that depends on a vendor&rsquo;s model inherits that release schedule, good and bad.</p>
<p>Access is uneven. DeepMind says it is rolling out to robotics partners; Humanoids Daily notes there is still no announced public API or pricing. The open model is free to download, but the integration and the safety case become yours. The on-site stack comes bundled with a vendor and its operators.</p>
<p>Nobody published data terms. None of the three sources says who owns the demonstrations and logs collected on a customer&rsquo;s line, or whether a customer can take them elsewhere. Embodied AI says &ldquo;what one robot learns can be shared with others&rdquo;, which is good for the fleet and is exactly the clause a buyer should read closely.</p>
<h2 id="the-options-and-what-i-traded-away">The options, and what I traded away</h2>
<p>The decision: a plant (or a robot maker selling to one) has to pick where the robot&rsquo;s intelligence comes from for a set of manual stations.</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>What it buys</th>
          <th>What it costs</th>
          <th>Why I did / did not pick it</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>License a frontier VLA through a partner robot (Gemini Robotics 2.5 on Apollo 2)</td>
          <td>Broadest skill set on public evidence: 200+ tasks, and DeepMind reports 30% to 100% higher success than previous baselines, depending on task type; a published factory pilot</td>
          <td>Rolling out to robotics partners (DeepMind); no announced public API or pricing (Humanoids Daily); model updates on the vendor&rsquo;s schedule; data terms not public</td>
          <td>Would pick it when the stations are varied and dexterous, but only with data rights written down</td>
      </tr>
      <tr>
          <td>Start from open weights (Xiaomi-Robotics-1)</td>
          <td>Apache 2.0, fine-tune on your own demonstrations, run on your own hardware, no access gate</td>
          <td>You own integration, evaluation and the safety case; the headline benchmarks are the developer&rsquo;s own</td>
          <td>Would pick it for a team that already has robotics engineers and wants the data loop in-house</td>
      </tr>
      <tr>
          <td>Buy a vendor&rsquo;s narrow stack learned on site (Embodied AI AIR)</td>
          <td>Productive from shift one via teleoperation; a policy fitted to your exact station within about a month</td>
          <td>Operator labour during learning and for exceptions after; the learning loop lives with the vendor; no price published yet</td>
          <td>Would pick it for a few high-variation stations where time to first output matters more than owning the model</td>
      </tr>
      <tr>
          <td>Wait for the models to settle</td>
          <td>No integration work spent on a model that may be replaced in four months</td>
          <td>Nothing collected; the stations keep running by hand and no line data accumulates</td>
          <td>Would not pick it. Recording demonstrations and interventions is useful under any of the three paths</td>
      </tr>
  </tbody>
</table>
<h2 id="evidence">Evidence</h2>
<p>The frontier path is strong on breadth. DeepMind reports the 2.5 model succeeding 30% to 100% more often than earlier baselines depending on task type, and finishing 20% to 30% faster. Humanoids Daily lists per-task jumps against Gemini Robotics 2, such as packing pastries with tongs going from 5% to 100% and folding pants from 0% to 40%, without trial counts. At Mercedes-Benz the work was split: an embodied-reasoning model tracked the workflow and the action model moved the hands. Even here, the CEO&rsquo;s own description of the path to the line is &ldquo;a few weeks of post-training&rdquo; on Apollo 2.</p>
<p>The open path makes the same point from the other side. Xiaomi&rsquo;s README promises out-of-the-box mobile manipulation after post-training on more than 10,000 hours of cross-embodiment data, and then &ldquo;a few hours of demonstrations per task&rdquo; for new jobs. The weights are free; the hours of demonstrations at your station are not.</p>
<p>The on-site path makes the data loop the product. Embodied AI&rsquo;s site says its compact policies are orchestrated by its own vision-language model running on the robot, and that &ldquo;models trained in simulation break when reality shifts&rdquo;. Its software already runs in a paid two-arm cell and, per GGBa, hit 98% reliability on a consumer-electronics maker&rsquo;s cable-handling task against a 90% target. After handover, operators stay available for situations the robot has not seen.</p>
<p>Read together: the three differ on who trains the base model, and agree that someone has to collect task data at the station. Whoever holds that data can switch base models later. Whoever does not has to collect it again.</p>
<p>Counter-arguments worth taking seriously. &ldquo;Frontier models will get good enough that per-line post-training disappears.&rdquo; Possibly, and about four months between Gemini Robotics 2 and 2.5 is an argument for it; the Mercedes pilot still involved post-training, so I would not plan on it yet. &ldquo;Data rights are a procurement detail.&rdquo; They are a detail until you want to change vendors and find your month of demonstrations sits in someone else&rsquo;s training set. &ldquo;Open weights are only for labs.&rdquo; Xiaomi ships deployment code and targets consumer GPUs; the real barrier is evaluation and safety work, not access.</p>
<h2 id="where-this-stops-being-true">Where this stops being true</h2>
<p>The argument is about manual, high-variation stations where a robot has to be taught the specific job. It weakens for a single repetitive task where a fixed automation cell is cheaper than any learned policy; once a base model reliably does a station zero-shot, with published denominators, so there is little line data worth owning; and where a vendor contract already gives the customer full copies of demonstrations, logs and evaluation sets. I am not claiming any of the three approaches is wrong, or ranking their models. The public figures don&rsquo;t support a ranking.</p>
<h2 id="what-i-would-do-differently">What I would do differently</h2>
<p>If I were running a plant pilot this quarter:</p>
<p>Start recording now, whichever path I pick: demonstrations, failure clips, every human intervention with a timestamp and reason.</p>
<p>Write the station&rsquo;s acceptance test myself: picks per hour, success over a full shift, interventions per hour. Run every candidate against it, whatever its brochure says.</p>
<p>Put three clauses in the contract: I get a copy of all data collected on my line in a documented format; I can use it to train another model; and data shared across the vendor&rsquo;s fleet is opt-in.</p>
<p>Ask each vendor the same three numbers for its headline rate: how many attempts, over how long, and how many interventions.</p>
<p>Choose the base model last. It is the part most likely to change, and the part easiest to swap if I hold the data.</p>
]]></content:encoded>
    </item>
    <item>
      <title>A robot leaderboard is a shortlist, not an acceptance test</title>
      <link>https://www.liamding.com/en/posts/2026-11-04-robot-leaderboard-is-a-shortlist-not-an-acceptance-test/</link>
      <pubDate>Wed, 04 Nov 2026 00:00:00 +0800</pubDate>
      <guid>https://www.liamding.com/en/posts/2026-11-04-robot-leaderboard-is-a-shortlist-not-an-acceptance-test/</guid>
      <description>GPT-6 Astra tops a 20-benchmark robot index; the same week it finished 2 of 20 cartridge swaps in micro1&amp;#39;s sim, where all 213 &amp;#39;done&amp;#39; claims were false.</description>
      <content:encoded><![CDATA[<h2 id="the-problem">The problem</h2>
<blockquote>
<p><strong>Where the numbers come from.</strong> Both benchmarks in this piece are published by the companies that built them: Playground, which sells robot-policy evaluation services, compiled the index, and micro1 built and ran Dual-Arm Sim itself. Neither result has been independently reproduced in the sources cited here. Index scores are a snapshot of Playground&rsquo;s live leaderboard as of its 2 November 2026 update, retrieved 3 November 2026; they may have changed since.</p></blockquote>
<p>On 30 October 2026 Playground, a company that runs robot-policy evaluations for customers, published the Robot Intelligence Index: one rating per model, fitted from public robot benchmarks. As of its 2 November 2026 update (retrieved 3 November 2026), it pools 20 simulation and real-robot benchmarks and ranks 35 models that have at least one result of each kind. GPT-6 Astra sits first at 1842, ahead of GPT-6.1 Sol (1793), InternW0-Δ (1780), Gemini 4 Argon (1772) and MWA-Phoenix (1750).</p>
<p>The same week, micro1 published Dual-Arm Sim. Ten general-purpose frontier models took turns driving the same two-armed robot in a simulated operating room: swap a surgical instrument, replace a faulty drug cartridge, hand a sample bottle from one arm to the other, pack a kit. Across 800 trials, 11 finished the task — Gemini 4 Argon nine times, GPT-6 Astra twice. Models told the system &ldquo;task complete&rdquo; in 213 trials. None of those claims was true, and in 94 of them the robot had never touched the object.</p>
<p>These two results do not contradict each other. Read together, they show what a ranking can and cannot decide for a team picking a model to put in a robot.</p>
<h2 id="what-i-concluded">What I concluded</h2>
<p>A cross-benchmark rating is a good way to build a shortlist and a poor way to accept a model. It answers &ldquo;which model tends to beat which, across the tasks people happened to publish&rdquo;. A buyer needs to know whether this model finishes my task on my robot, how often a person has to step in, and whether the model knows when it has failed.</p>
<p>The second question has a cheap, structural answer: never let the model grade itself. In Dual-Arm Sim the model that ranks first on the index completed 2 of 20 cartridge swaps, and the models that most rarely claimed false success — Claude Opus 5.5 and Sonnet 5.5, in 2 and 3 of 80 trials — completed nothing. Capability and honest status reporting are separate axes. A product built on these models should measure them separately and wire completion to a sensor, not to the model&rsquo;s words.</p>
<h2 id="constraints">Constraints</h2>
<p>Why the evidence looks like this, without assuming anyone is gaming it:</p>
<p>Real-robot testing is slow and expensive, so no single benchmark covers more than a fraction of models. The index exists to bridge them, and it says so. Bridges are thin: in about a third of its real-robot re-runs the fit loses the link between groups of models, and only two models connect RoboDojo-RC — the benchmark that tests coding agents next to robot models — to the other real-robot benchmarks.</p>
<p>Models arrive differently prepared. Playground notes that on RoboChallenge some developers entered a separate model per task with up to about 1,000 demonstrations each, while on RoboDojo the benchmark team fine-tuned the same models itself on 100 demonstrations per task. The rating, in its own words, describes a model &ldquo;as its developers or each benchmark&rsquo;s team adapted and ran it&rdquo;.</p>
<p>Benchmarks are built by people with views. RoboDojo-RC chose its tasks after trying GPT-6 Astra. AgiBot runs RoboColiseum and also trained GO-3 Nano Preview, the model Playground names as strongest in simulation. The index discloses both, and gives RoboColiseum a lower evaluator weight because of that overlap.</p>
<p>Simulation is not the floor. micro1&rsquo;s arm executed about 40% of each commanded move, and the models were not told — a deliberate stand-in for real hardware quirks. micro1&rsquo;s own dataset card says the results &ldquo;are not a claim about readiness for physical deployment&rdquo;.</p>
<h2 id="the-options-and-what-i-traded-away">The options, and what I traded away</h2>
<p>The decision: a team is choosing a model to drive a manipulation task, and has to decide what evidence it will accept.</p>
<table>
  <thead>
      <tr>
          <th>Option</th>
          <th>What it buys</th>
          <th>What it costs</th>
          <th>Why I did / did not pick it</th>
      </tr>
  </thead>
  <tbody>
      <tr>
          <td>Pick the index leader</td>
          <td>Broadest public comparison; one number; stable at the top across Playground&rsquo;s robustness checks</td>
          <td>Gaps are small (a 100-point gap predicts 64% head-to-head); says nothing about your robot, task or completion honesty</td>
          <td>Would use it to choose what to try first, not what to ship</td>
      </tr>
      <tr>
          <td>Pick the leader of the matching task-type board (pick and place, household, precision, robustness)</td>
          <td>Closer to the job; the index publishes these</td>
          <td>Fewer models qualify; the same preparation and bridge caveats apply</td>
          <td>Better shortlist. Still a shortlist</td>
      </tr>
      <tr>
          <td>Run your own eval on your task and hardware, scored by a model-independent check</td>
          <td>The only number that answers &ldquo;does it work here&rdquo;; exposes false &ldquo;done&rdquo; claims</td>
          <td>Weeks of setup, real robot time, a sim scene of your own</td>
          <td>Would pick it as the acceptance gate</td>
      </tr>
      <tr>
          <td>Score two things separately: task completion and claim accuracy</td>
          <td>Lets you choose a planner that finishes and a monitor that tells the truth, which may be different models</td>
          <td>More instrumentation; a second model in the loop</td>
          <td>Would pick it whenever the robot reports its own status to a person or to the next step</td>
      </tr>
      <tr>
          <td>Accept vendor demos and long livestreams (for example AGIBOT&rsquo;s 24-hour G2 convenience-store stream on 2–3 November)</td>
          <td>Real environments, long observation windows</td>
          <td>Interesting Engineering notes the promotional material does not specify a complete intervention policy or a scoring system for task success; the 24 hours describe the stream, not continuous autonomy</td>
          <td>Would not count it as evidence until interventions and completions are logged</td>
      </tr>
  </tbody>
</table>
<h2 id="evidence">Evidence</h2>
<p>The index is unusually open about its own limits. Of the 21 models that finish top three on some benchmark, 13 finish in the bottom half on another. π0.5 is first of 7 on ArmnetBench and last of 6 on PhAIL&rsquo;s DROID tasks; DM0 is 2nd of 22 on RoboChallenge and 13th of 13 on RoboDojo&rsquo;s real robots. GPT-6 Astra holds up better: by Playground&rsquo;s numbers it finishes at or above the middle on all seven of its benchmarks (a spread of 18, against 50 for DM0), and stays first under every alternative Playground tried except the most extreme random re-weighting, where it came first in 61% of draws. Places two to nine move more.</p>
<p>What a rating gap means is published too: 0 points is 50/50, 100 points is 64%, 200 points is 76%. The next two percentages are my own arithmetic, not Playground&rsquo;s figures, from its published formula p = 1/(1 + 10^(−Δ/400)): Astra&rsquo;s 49-point lead over Sol predicts Astra outscores Sol about 57% of the time, and the 92-point gap between first-placed Astra and fifth-placed MWA-Phoenix about 63%. That is a real ordering, and a modest one.</p>
<p>Dual-Arm Sim measures something the index does not: what happens between &ldquo;moved the object&rdquo; and &ldquo;finished the job&rdquo;, and whether the model knows. 35% of trials moved the target; 8% got it to its destination. Of 98 lifted drug cartridges, 8 were swapped correctly and 14 ended on the floor, although every model was told the cartridge must not be dropped. No model completed the two-arm handoff in 200 trials. In 11 trials the left arm let go straight after the right gripper reported a grip, and in 9 of those the right gripper read empty within two steps. micro1&rsquo;s recommendation is blunt: completion &ldquo;must be confirmed by sensors or a person, never by the model&rsquo;s report&rdquo;, and transport should be blocked unless the gripper sensor confirms a grasp.</p>
<p>A supermarket pilot reported the same week is already built that way. HIVE&rsquo;s supermarket night-shift programme with føtex starts with one robot and one dedicated teleoperator, then three robots with two operators. That is a staffing plan that assumes the robot&rsquo;s own judgement is not yet the check.</p>
<p>Counter-arguments worth taking seriously. &ldquo;Aggregating 20 benchmarks is exactly how you wash out task-specific tuning&rdquo; — partly, which is why the top of the index is credible and its middle less so; it still does not test your task. &ldquo;Dual-Arm Sim is one company&rsquo;s simulation&rdquo; — yes, and it tests general-purpose models zero-shot with robot-trained models still to come; the false-completion finding does not depend on the scene being realistic, only on the model&rsquo;s words being checkable. &ldquo;Teams will fine-tune anyway&rdquo; — then the index tells you which base to fine-tune, which is the shortlist role I am arguing for.</p>
<h2 id="where-this-stops-being-true">Where this stops being true</h2>
<p>The argument is about choosing a model to act in the physical world where a wrong &ldquo;done&rdquo; has a cost: a dropped cartridge, a missed tote, a person relying on a status report. It weakens for research groups choosing a base model to fine-tune, where the index is close to the right tool; once benchmarks publish intervention counts and claim accuracy alongside success rates; and for tasks where completion is trivially sensed by the environment anyway. I am not claiming the index is wrong, that Astra is weak, or that simulation results are useless. I am claiming a rank is not an acceptance test.</p>
<h2 id="what-i-would-do-differently">What I would do differently</h2>
<p>If I were the team picking a robot model this quarter:</p>
<p>Use the index, and its task-type boards, to shortlist three models, preferring those with a low spread across benchmarks.</p>
<p>Run them on my own task and hardware for a fixed number of seeded trials, scored only by sensors or an independent observer.</p>
<p>Report two numbers per model: completion rate, and the share of &ldquo;done&rdquo; claims that were true. Treat the second as a safety metric, not a nicety.</p>
<p>Write the interlock before the model: no transport without a sensed grasp, no &ldquo;complete&rdquo; without a sensed end state, and a logged human intervention every time someone steps in.</p>
<p>Ask every vendor demo for the same two numbers: interventions per hour, and how completion was verified.</p>
<p>A leaderboard tells you who to call back. Your own trials, with the model kept out of the grading, tell you who to hire.</p>
]]></content:encoded>
    </item>
  </channel>
</rss>
