---
title: "Human in the loop is not a robot safety specification, and ADAS already proved it"
title_zh: "「人在回路」不是机器人安全规格，要问的是接管延迟与并发"
url: https://www.liamding.com/posts/2026-10-14-human-in-the-loop-is-not-a-robot-safety-spec/
markdown_url: https://www.liamding.com/posts/2026-10-14-human-in-the-loop-is-not-a-robot-safety-spec/index.md
html_url: https://www.liamding.com/posts/2026-10-14-human-in-the-loop-is-not-a-robot-safety-spec/
language: en
date: 2026-10-14
lastmod: 2026-10-14
description: "Anthropic's usage policy treats autonomous physical operation as high risk. My reading from ADAS: 'human in the loop' is a compliance condition, not a safety specification. Spec the numbers instead."
categories: ["Robotics"]
tags: ["robotics", "human-in-the-loop", "teleoperation", "ADAS", "embodied AI", "safety policy"]
professional: true
sources:
  - title: "Anthropic 研究：小型 AI 模型可能接近在 IoT 设备上自主复制"
    url: "https://aihot.news/items/emc0djmqlomn6b5whge285x7l"
  - title: "Epic Games 转向 AI 先行开发模式，与迪士尼合作射击游戏无限期延期"
    url: "https://aihot.news/items/kcwro19vl7dr5jgcjgec7mive"
  - title: "QGIS 通过 QEP428 禁止 AI 辅助贡献"
    url: "https://aihot.news/items/t0gnehm6pcbj4xyn5f3txv1ge"
  - title: "用户报告 Codex CLI 0.157.0 默认启动持久化自更新后台守护进程"
    url: "https://aihot.news/items/amjmxvrvj57q7752kgihp9p9q"
author: "Liam Ding（丁洪）"
canonical: https://www.liamding.com/posts/2026-10-14-human-in-the-loop-is-not-a-robot-safety-spec/
---

## The problem

Anthropic's usage policy puts autonomous physical operation in a high-risk category that requires a human in the loop, and separately bars using its models to arm drones. My conclusion is that the risk tier is right and the boolean is wrong: what belongs in that sentence is a set of numbers, not a person's presence. The drone clause is uncontroversial. The human-in-the-loop clause looks like a bright line and is not one, because it carries no engineering content. It does not say how fast the human must respond, how many machines one human may hold, what the robot does when the human is unreachable, or how any of it is verified.

Automotive went through this phase. "The driver is supervising" was both the regulatory and the marketing answer to level-2 assistance, and it satisfied nobody — not the regulator who had to define supervision, not the insurer pricing it, not the engineers implementing a sentence with no threshold in it. What moved the industry was quantification: a declared ODD, a measured takeover time, a published operational number, argued inside a taxonomy everyone had agreed on.

The timing is the interesting part. The same week, Anthropic published research showing a 9B model running inference on eight of the nine hardware classes it tested, with 30B-and-below models completing most individual steps of autonomous replication; Epic Games moved to an AI-first pipeline where Claude writes all the code and directs all the art on a project tied to a $1.5bn Disney partnership; and QGIS merged QEP428 on 2026-10-07 declaring AI-assisted contributions NOT ALLOWED. Three policy postures toward machine autonomy in one week — and only one concerns a domain where a wrong action can be undone.

## What I concluded

Naming physical action as its own risk surface is correct and overdue. Expressing the control as a boolean is the mistake. The line that matters is not human versus no human; it is reversible versus irreversible error. A software agent that leaves a background daemon running after you close the terminal — Codex CLI 0.157.0 does exactly this by default, with `codex app-server --managed-daemon` — is an annoyance you end with a process kill. An arm moving through a shared aisle is not undoable, and "the operator was watching" does not reverse the outcome.

What should be written instead is a quantified human-in-the-loop specification with four numbers: teleoperation ratio (share of runtime or task attempts that genuinely needs a remote human), takeover latency as a distribution with a stated P95, operator concurrency (robots per trained operator per shift), and the minimum risk state the robot enters when the human link fails. A fifth, if allowed: the authority model — what the robot may do when it cannot reach a human and stopping is the worse option.

That reframes the question. "A human must be in the loop" asks whether someone is watching. "Disclose your teleop ratio and your takeover P95" asks whether the deployment is actually supervised. Only the second is auditable.

## Constraints

Four constraints decide how far this gets.

Vocabulary. ADAS inherited a coarse but shared taxonomy of levels and a shared concept of ODD. Mobile manipulators and humanoids have no equivalent: no agreed definition of a task, no ODD language for mixed-traffic warehouses, no rule for when a model version change makes it a different deployment. Without that, a teleop ratio is not comparable across vendors.

Measurement culture. Automotive has a reporting habit for disengagements and mileage. Robotics has demos. There is no public benchmark for takeover latency in manipulation and no cross-vendor reporting of operator concurrency, so the numbers I am asking for have to be built by each deployer rather than compared against a norm.

Split incentives. Model providers inherit the compliance obligation, integrators carry the deployment cost, the site carries the physical risk. Nobody in that chain is naturally funded to measure.

The constraint that decides the outcome: auditability asymmetry. A boolean is cheap to check, and therefore attractive to a regulator with no robotics headcount. A P95 requires telemetry, definitions and a competent auditor. Policy gets written at the level of what can be verified — which is exactly why the boolean survives unless deployers bring the numbers first.

## The options, and what I traded away

| Option | What it buys | What it costs | Why I did / did not pick it |
|---|---|---|---|
| Treat the clause as a checkbox: name a remote supervisor, define nothing | Fastest route to a signed deployment; lowest headcount | Zero risk reduction — the failure mode is unchanged, only documented; will not survive the first incident or a serious procurement audit | Not picked. It turns a safety question into a documentation question, and it is why "supervision" became a dirty word in ADAS |
| One full-time teleoperator per robot | Real fallback authority; the only thing that works today on the hardest tasks | Fleet size becomes a function of headcount; unit economics does not close; fatigue degrades it into the checkbox within a shift | Not picked as the target; kept as the data-collection mode |
| Publish a quantified human-in-the-loop spec — teleop ratio, takeover P95, robots per operator, minimum risk state | An acceptance-testable product claim; forces takeover-latency telemetry and a minimum-risk-state definition before concurrency is raised; gives procurement something to contract on | Expensive to instrument; you will miss your own numbers early; it slows the roadmap | Picked. A number is the only thing that can be argued with |
| Wait for a standard, or lobby the clause away | Avoids failing your own numbers in public | Nothing ships meanwhile; buyers default to the checkbox, which then becomes the de facto standard | Not picked |

I would deliberately run the second option during data collection and contract to reach the third. What I traded away is the comfort of pretending the checkbox was sufficient.

## Evidence

The reversibility asymmetry is directly observable. Codex CLI 0.157.0 opening a persistent managed daemon when an interactive session starts is a real agent-containment failure, recoverable in seconds. QGIS can declare AI-assisted contributions NOT ALLOWED (QEP428, merged 2026-10-07) because its worst case is a bad patch in a review queue. Epic's Claude-driven pipeline can iterate because the artefact is code and art. Those domains can afford blunt policy because a wrong action is reversible. Physical action is the one category where that assumption fails — which is why I accept the risk tier and reject the boolean.

Capability is documented too: Anthropic's own research shows 9B models running on eight of nine tested hardware classes, with 30B and below completing most single steps of autonomous replication. Read that as a forecast about cheap autonomy attempts, not about robots doing useful work.

What I am not treating as evidence: the automotive record, which I can describe structurally but will not quote as a benchmark here. And no public dataset exists for manipulation takeover latency, so my four numbers are a proposal, not a standard.

## Where this stops being true

Inside a fenced cell with hard interlocks and one repeated task, "no human inside the fence while the cell runs" is correct and sufficient, because the fence is the control and the human is not. The boolean is right there.

Fleet size: below a handful of units — order of one to ten — the operator is effectively the integration layer and a takeover P95 is theatre. The argument bites once the operator-to-robot ratio reaches a hiring plan.

Segment: this is about unstructured, multi-task manipulation in spaces shared with people. A fixed-track AGV in a closed aisle is a different problem with a defensible fixed rule. And the numbers themselves move: if a coming VLA generation pushes interventions down to exception handling, the four metrics are still the right specification while their acceptable values shift a long way. Today's numbers are not the target.

Company size and region: a small team cannot fund the measurement stack, and what a deployer must file differs by jurisdiction. Proportion the spec to the fleet and to who owns the risk. The argument also stops being true in the good way if standards bodies put the numbers into the rule — then a boolean attestation is fine, because the standard carries the engineering content.

## What I would do differently

If I were drafting the clause, I would keep the risk tier and replace the condition: disclose teleop ratio, takeover latency with a P95, robots per operator and minimum risk state at deployment, and update the disclosure when the model or the operating domain changes.

If I were buying, I would put those four numbers in the contract with a test procedure and a remedy, and refuse "the operator is in the loop" as an answer to a technical question.

If I were deploying, I would instrument the human link before adding robots — link loss, takeover latency and minimum-risk-state entry as first-class telemetry — and cap concurrency at what is genuinely supervised, raising it only with data.

The wider lesson from ADAS is that the industries which measured got the better regulation. Robotics should start that measurement culture before an incident makes it mandatory, because boolean rules that fail do not get refined. They get replaced with something much blunter.

## 中文版

## 问题

Anthropic 的使用政策把「自主物理操作」划入高风险类别，要求 human-in-the-loop，同时单独禁止用其模型为无人机装载武器。我的结论是：风险分级是对的，布尔值写法是错的——那句话里该出现的是一组数字，而不是「有人在场」。禁武条款没有争议。「必须有人在回路」看起来像一条清晰的线，其实不是，因为它不含任何工程内容：人必须在多久内响应，一个人能看住几台机器，联系不上人时机器人做什么，这些怎么验证，它一个字都没说。

汽车行业已经走过这个阶段。「驾驶员在监督」既是 L2 辅助驾驶的监管答案，也是它的营销答案，三方都不满意：监管方要去定义「监督」，保险公司要给它定价，工程师要实现一句没有阈值的句子。真正把行业往前推的是量化——声明的 ODD、可测量的接管时间、公开的运行数据，以及大家同意在里面争论的分级体系。

时间点更有意思。同一周，Anthropic 自己发表研究：9B 模型在被测的 9 类硬件中有 8 类跑通推理，30B 及以下模型已完成自主复制的大多数单步；Epic Games 转向 AI 先行，由 Claude 编写全部代码、指导全部美术，项目关联迪士尼 15 亿美元合作；QGIS 在 2026-10-07 合并 QEP428，把 AI 辅助贡献定为 NOT ALLOWED。一周之内三种针对机器自主性的政策姿态，而其中只有一种所处的领域，错误是可以撤销的。

## 我的结论

把物理动作单列为独立风险面是对的，而且早该做；把它写成一个是/否开关才是错的。真正的分界不是「有没有人」，而是错误可不可回滚。软件 agent 在关掉终端后留下后台守护进程——Codex CLI 0.157.0 默认就是如此，`codex app-server --managed-daemon`——你 pkill 一下就结束了；而一条穿过共用通道的机械臂不可撤销，「操作员当时在看」不会让结果反过来。

该被要求写的不是布尔值，而是一份量化的 human-in-the-loop 规格，至少带四个数字：teleoperation ratio（运行时或任务尝试中真正需要远程人介入的比例）、以 P95 表述的接管延迟分布、单操作员并发机器人数，以及人在回路失效时机器人进入的最小风险状态。若还能加一条，就是权限模型：联系不上人、而停下来反而是更差选择时，机器人被允许做什么。

这把问题换了问法。「必须有人类在回路」问的是有没有人在看；「公开你的 teleop ratio 和接管 P95」问的是这套部署有没有被真正监督。只有后者可审计。

## 约束条件

四条约束决定这件事能走多远。

一是词汇。ADAS 继承了一套粗糙但共享的分级体系，也共享 ODD 这个概念。移动操作臂和人形机器人没有对应物：任务没有公认定义，混合通行的仓库没有 ODD 语言，也没有规则说明模型版本一变算不算同一套部署。没有这些，teleop ratio 在厂商之间不可比。

二是测量文化。汽车行业有 disengagement 和里程的披露习惯，机器人行业有的是 demo。机械操作领域没有公开的接管延迟基准，也没有跨厂商的操作员并发披露，所以那几个数字只能由每家部署方自己建，没有一个行业均值可以对照。

三是成本与风险不在同一处。合规义务落在模型提供方，部署成本落在集成方，物理风险落在现场，这条链上没有谁天然有动力出测量的钱。

四是真正决定结果的一条：可审计性不对称。布尔值检查起来便宜，对没有机器人编制的监管者格外有吸引力；P95 接管延迟需要埋点、定义和一个有能力的审计方。政策只会写到能被验证的粒度上——这正是布尔值会存活下来的原因，除非部署方先把数字摆出来。

## 取舍选项

| 选项 | 换来什么 | 代价 | 为什么选/不选 |
|---|---|---|---|
| 当作合规勾选项：指定一名远程监督员，其余不做定义 | 最快签下部署，人力成本最低 | 风险零下降，失效模式没变、只是被写进文档；第一起事故或第一次认真的采购审计就会击穿 | 不选。它把安全问题变成文档问题，也是 ADAS 里「监督」这个词变脏的原因 |
| 一台机器人配一名全职遥操作员 | 真实的后备权限；目前最难的任务上这是唯一真正可行的做法 | 车队规模变成编制的函数，单位经济算不过来；操作员疲劳会让它在一个班次内退化成勾选项 | 不作为目标，保留为数据采集阶段的模式 |
| 公开量化的 human-in-the-loop 规格：teleop ratio、接管 P95、单操作员并发数、最小风险状态 | 一个可验收的产品声明；在提高并发前逼出接管延迟埋点和最小风险状态定义；给采购一条能写进合同的条款 | 埋点贵；早期一定打不到自己定的数字；拖慢路线图 | 选它。只有数字是可以被争论的 |
| 等标准，或者游说删掉这条 | 不必在公开场合打不到自己的数字 | 期间什么都发不出去；买方默认退回勾选项，勾选项于是变成事实标准 | 不选 |

我的做法是：数据采集阶段有意用第二种，合同上承诺走到第三种。我放弃的，是「勾选项已经够了」的那份安心。

## 证据

可回滚性的不对称是直接可观察的。Codex CLI 0.157.0 打开交互式会话时默认启动常驻的 managed daemon，这是一次真实的 agent 失控，但几秒钟就能恢复。QGIS 敢对 AI 辅助贡献写 NOT ALLOWED（QEP428，2026-10-07 合并），因为最坏结果只是评审队列里一个坏补丁。Epic 由 Claude 驱动的流水线可以反复迭代，因为产物是代码和美术。这些领域负担得起粗暴政策，因为错误动作可回滚。物理动作恰恰是这条假设失效的类别——所以风险分级我接受，布尔值我不接受。

能力一侧同样有记录：Anthropic 自己的研究显示 9B 模型能在被测的 9 类硬件中的 8 类上跑推理，30B 及以下已完成自主复制的大多数单步。请把它读成对「廉价自主性尝试」的预测，而不是机器人在干有用活的证据。

我不当作证据的：汽车行业的历史记录——我熟悉到可以结构性描述，但不会在这里当基准引用。另外，机械操作领域没有公开的接管延迟数据集，所以那四个数字是提案，不是标准。

## 这个结论的边界

在有围栏、有硬联锁、只做一个重复任务的工位里，「运行时人不得进入围栏」就是正确且足够的规则，因为控制手段是围栏而不是人，布尔值在这里成立。

规模上：车队在个位数到十台上下这个量级以下时，操作员本身就是集成层，测接管 P95 是表演。这个论证真正生效，是从操作员与机器人比例进入招聘计划那一刻开始。

场景上：我谈的是与人共享空间、非结构化、多任务的机械操作。封闭通道里的固定路径 AGV 是另一个问题，可以有一条说得通的固定规则。数字本身也会移动：如果下一代 VLA 把介入压缩成异常处理，这四个指标仍是正确的规格，但可接受取值会移动很远——今天的数字不是目标。

公司规模与地区：小团队负担不起测量栈，部署方要提交什么也因司法辖区而异，规格应与车队规模和风险归属相匹配。最后，如果标准机构把数字写进规则，这个论证会以好的方式失效：那时布尔值的自我声明是可以的，因为工程内容由标准承载。

## 我会怎么改

如果由我起草条款，我会保留风险分级，换掉条件本身：在部署时披露 teleop ratio、以 P95 表述的接管延迟分布、单操作员并发机器人数和最小风险状态，并在模型版本或运行域变化时更新披露。

如果我是买方，我会把这四个数字写进合同，配测试程序和补救条款，并拒绝把「操作员在回路里」当作技术问题的技术答案。

如果我是部署方，我会在增加机器人之前先给「人」这条链路埋点——链路丢失、接管延迟、最小风险状态的进入都是一等遥测——把并发数压到真正被监督的水平，再只凭数据往上加。

从 ADAS 来的更普遍的教训是：愿意测量的行业，最后拿到的监管更好。机器人行业应该在强制之前就开始自己的测量文化，因为失败的布尔规则不会被修正，它们会被更粗暴的东西替换掉。