CH-01 · AMD Hackathon ACT III · Intelligent Industry

Nobody on the channel noticed. The radio
log did.

SayAgain checks each read-back of a radioed safety order and sends the wrong ones, and any it cannot check, to the shift supervisor. A model on the AMD Token Factory points at the words; plain code decides.

MISMATCH

H-01 · CH-01 · English

tag: XV-203 ≠ XV-208 XV-208 is on line A (registry)

OPERATORH-01-0

Close valve two-oh-three, line Bravo.

FIELD TECHH-01-1

Copy, closing two-oh-eight, line Bravo.

speaker text·scripted
extraction Gemma-4-31B-IT · AMD-hosted API
referee local · 1.07 ms

Scripted exchange H-01, redrawn from the queue: the written words, the stored Gemma-4-31B-IT extraction and the referee’s verdict. “Say again” is radio for “repeat that”. A wrong read-back deserves one.

SayAgain checks each read-back of a radioed safety order against the order and the plant registry, and sends the wrong ones, and any it cannot check, to the shift supervisor. A model on the AMD Token Factory points at the words; plain code decides.

GitHub Pitch deck Releases Open the replay

Receipts · 89 held-out exchanges · written scripts, no audio yet

2 versus 0wrong read-backs passed
LLM judge 2, referee 0
41/41seeded errors caught, Gemma-4-31B-IT
2/48false alarms on correct read-backs, same rung
24× OKresult rows re-derived offline in 2.9 s

bench/results/text/llm-judge-r1-reaudit.json · bench/results/text/ladder-r1-reaudit.json · evidence/terminal-2026-10-14.log

XV-203 → XV-208 · where the read-back points

One digit moves the crew to another line.

In the plant registry the demo runs on, XV-203 is an on/off valve on line B at the press station. XV-208 is on line A, in clarification. The read-back exists so the operator hears that difference. Sometimes nobody does.

data/registry.csv · fictional plant, 120 tags; the valves drawn are the registry’s own on lines A and B: XV-203 · on/off valve · line B · press station · XV-208 · on/off valve · line A · clarification

63 % tower · 50 % approach
of pilots’ read-back errors are caught by air-traffic controllers, in the literature. The rest go through.
Helmke et al., ATM Seminar 2021 · aviation, not plant radio · evidence/sources.json

1 killed, 6 injured
What a wrong valve state can cost: Goodyear, Houston, 2008. An isolation valve was left closed after maintenance and the remaining relief path was then blocked.
US Chemical Safety Board case study · evidence/sources.json

No rate
of read-back errors on industrial plant radio has been published that we could find. Measuring it is the pilot study’s first job.
Our literature search, 2026-10-09 · evidence/sources.json · the check is called three-way communication

Replay · run ladder-r1-reaudit · Gemma-4-31B-IT

A whole shift, replayed and judged again.

The supervisor queue replays 145 exchanges from a stored run. The referee judges each one again from the model output the run recorded, and every card lists the receipt of each model call behind it. The terminal is the real session, recorded on 2026-10-14.

SayAgain supervisor queue replaying the stored shift: the list of exchanges on the left, card H-01 on the right with two-oh-eight blocked red and the buttons Confirm error and Dismiss — Real capture of `docs/replay/`: the same page as the live queue, verdicts precomputed. Open it.

zsh · build/

# recorded 2026-10-14T18:48:05+07:00 in build/ at 53e9114 (macOS arm64); stdout kept verbatimjq -r '.rungs | to_entries[] | "\(.key)  false-VERIFIED \(.value.rows.held_out.false_verified.k)  false alarms \(.value.rows.held_out.false_alarm_load.k)/\(.value.rows.held_out.false_alarm_load.n)"' bench/results/text/ladder-r1-reaudit.jsonDeepSeek-V4-Flash  false-VERIFIED 0  false alarms 4/48Gemma-4-31B-IT  false-VERIFIED 0  false alarms 2/48MiniCPM5-2B  false-VERIFIED 0  false alarms 48/48Qwen3.8-27B  false-VERIFIED 0  false alarms 4/48regex  false-VERIFIED 0  false alarms 0/48# exit 0 · wall 0.0 sjq -r '.results.llm_judge.rows.held_out | "llm_judge  false-VERIFIED \(.false_verified.k)  false alarms \(.false_alarm_load.k)/\(.false_alarm_load.n)"' bench/results/text/llm-judge-r1-reaudit.jsonllm_judge  false-VERIFIED 2  false alarms 1/48# exit 0 · wall 0.0 sjq -r '.events[] | select(.type=="judged" and .card.exchange_id=="H-01") | .card.verdict.trace[]' docs/replay/data/ladder-r1-reaudit--Gemma-4-31B-IT.jsoninstruction: grounded + covered (1 command(s))read-back: grounded + covered (1 command(s))audio: no gate (no token probabilities, no unprompted pass)instruction: XV-203 line B (registry ok)compare: tag: XV-203 ≠ XV-208# exit 0 · wall 0.0 sjq -c '.events[] | select(.type=="judged" and .card.exchange_id=="H-01") | .card.turns[].receipt | {request_id, model_served, host, latency_s}' docs/replay/data/ladder-r1-reaudit--Gemma-4-31B-IT.json{"request_id":"980edf5c577d429b906b24cfae299218","model_served":"self-dploy/Gemma-4-31B-IT","host":"developer.amd.com.cn","latency_s":10.7585}{"request_id":"c9647a04cfcc4eeaadbfa965a4d74c6d","model_served":"self-dploy/Gemma-4-31B-IT","host":"developer.amd.com.cn","latency_s":35.4972}# exit 0 · wall 0.0 s

Transcript: evidence/terminal-2026-10-14.log (recorded at build 53e9114).

Re-derived offline

24 × OK

The stored model outputs behind 24 result rows re-scored by the referee code (the LLM-judge rows from the stored judge outputs), in 2.9 s with no network: uv run python scripts/eval.py --rescore

Pick a transmission · 10 of 145 · stored run, verdicts precomputed

Ten real cards from the stored shift.

Each card is the referee’s own output for the stored Gemma-4-31B-IT run: the verdict, the field diff, the trace, and the receipt of every model call (identical transcripts share one call, so two cards can show the same request id). Nothing on this page judges in your browser. Three of the ten fooled the LLM-as-judge baseline (two wrong read-backs and the XV-205 order); one is a false alarm of ours.

MISMATCH

A2-066 · CH-01 · English

action: missing (expected START)

OPERATORA2-066-0no recording

Start pump one-two-seven, line Alpha.

FIELD TECHA2-066-1no recording

Roger, pump one-two-seven, line Alpha. no action word

speaker text·scripted
radio none (no audio)
extraction Gemma-4-31B-IT · AMD-hosted API
referee local · 0.706 ms

Same exchange · LLM-as-judge baseline

VERIFIED · DeepSeek-V4.1-Flash, given the whole exchange and no plant registry · a wrong read-back passed

How the referee judged it 5 steps

instruction: grounded + covered (1 command(s))
read-back: grounded + covered (1 command(s))
audio: no gate (no token probabilities, no unprompted pass)
instruction: P-127 line A (registry ok)
compare: action: missing (expected START)

Receipts 2 model calls

A2-066-0 f9865ae58e204a05bb358fad58ee910f self-dploy/Gemma-4-31B-IT · developer.amd.com.cn · latency 1.5654 s · cost_usd 4.8587e-06
A2-066-1 efe40af5e62348b7b3a0a926ee2382cc self-dploy/Gemma-4-31B-IT · developer.amd.com.cn · latency 1.5373 s · cost_usd 4.8937e-06

Model ladder · 89 held-out exchanges · text·scripted

Climb only as high as the rule needs.

The rule was pre-registered before the first model call, for the audio headline row (bench/PROTOCOL.md logs its use on these written-word rows as a deviation): the smallest rung with 0 false-VERIFIED (wrong read-backs passed) and at most 5 % false-alarm load (correct read-backs sent to a human). Every model rung extracts spans on the AMD Token Factory; the referee makes each one fail toward a human.

RungFalse-VERIFIEDSeeded errors flaggedFalse alarms, 48 correct read-backsLatency, median / 95th pctMeets the rule

regexrung 0 · no model 0 41/41 0/48 no model call meets

MiniCPM5-2Bsmallest on the ladder 0 41/41 48/48 1.3975 s / 6.9094 s all 48 to a human

Gemma-4-31B-ITthinking off · re-audit 0 41/41 2/48 4.304 s / 35.8702 s meets

DeepSeek-V4-Flashthinking off · re-audit 0 41/41 4/48 4.0574 s / 11.1292 s over budget

Qwen3.8-27Bthinking on · re-audit 0 41/41 4/48 14.1714 s / 62.5185 s over budget

LLM-as-judgebaseline · DeepSeek-V4.1-Flash · no registry 2 35/414 not audited 1/48 n/a no · passed XV-205

Cells: the 48 correct read-backs, filled where a rung sent one to a human anyway (hatched: a failed call; MiniCPM5-2B sent the whole row). Dashed: the 5 % budget. Rows: the 89 held-out of the 145 cards (the others are the 40-card dev slice, 15 chatter cards and the refusal). Latency: each model rung’s first pass. Gemma-4-31B-IT, Qwen3.8-27B and DeepSeek-V4-Flash (1 call) rows are re-audits: as run, the first passes of Gemma and Qwen left 45 and 3 cards not audited (bench/results/text/ladder-r1.json).

With so few cards the rates are wide: Gemma-4-31B-IT’s 2/48 is 4.17 % with a 95 % Wilson interval of 1.15 % to 13.98 % (bench/results/text/RESULTS.md). The baseline is DeepSeek-V4.1-Flash, a different model from the DeepSeek-V4-Flash rung.

On written words, rung 0 already meets the rule. A frozen regex, no model: 41/41 errors flagged, 0/48 false alarms. Whether that survives real speech, noise and other people’s phrasing is what the audio rows measure.

Grounding is load-bearing. The same model returning values instead of copied words: false alarms 2/48 → 6/48 (one more correct card not audited), in 4 of the 6 turning a spoken pump or motor-valve tag into a valve tag the registry does not have.

Judging is not the model’s job. Given the whole exchange and no registry, the LLM-as-judge baseline passed 2 wrong read-backs as VERIFIED and passed the XV-205 order, a valve the plant does not have.

Referee · one model job, everything else is code

The model points. Code decides. A human closes the card.

01
The model points at words
For each turn, the model on the AMD Token Factory returns the words that carry the action, the tag and the line: copied spans, never values. JSON mode, schema checked in code; a failed call is NOT_AUDITED, never a pass.

0/194turns with a span not found in the transcript, Gemma-4-31B-IT, held-outcode: sayagain/extract.py
```
{
  "commands": [
    {
      "action_span": "closing",
      "line_span": "line Bravo",
      "state_span": null,
      "tag_span": "two-oh-eight"
    }
  ],
  "kind": "readback",
  "retracted_spans": []
}
```
02
The words must be there
Every span must appear in the transcript, word for word.

2/48 → 6/48false alarms when the same model returns values instead of copied words (one more card not audited)code: referee/grounding.py
03
Every safety word counted
A read-back that drops the action word, or keeps a line the operator retracted, cannot verify. A model that returns nothing usable sends the card to a human.

48/48correct read-backs sent to a human when MiniCPM5-2B returned spans the referee could not ground or cover; none passedcode: referee/coverage.py
04
The plant registry has the last word
Spoken digits and NATO letters become tags through alias tables, and the tag must sit on the line the order names.

120tags in the registry; XV-205 is not one of them, so its order is refuseddata: data/registry.csv
05
A human closes every card
Confirm, or dismiss with a reason. Nothing is ever sent to the plant.

10random VERIFIED exchanges per shift go to an end-of-shift spot-checkcode: referee/audit.py
06
The verdict is a pure function
The verdict module imports no model code, and a test fails the build if it ever does.

762tests collected by pytesttest: tests/test_verdict.py

Stage line · read from the receipts

Every stage says where it ran.

Span extraction, the model’s one job, runs on the AMD Token Factory: it copies out the words that carry the order. Each call leaves a receipt with the gateway host, the model served, the request id and the cost, and every card prints where each stage ran.

Speech-to-text
not measured yet
Local Whisper is planned and badged with where it ran. The Token Factory lists no transcription model.
no audio evaluation in this submission
Span extraction
AMD Token Factory · developer.amd.com.cn
Gemma-4-31B-IT, DeepSeek-V4-Flash, Qwen3.8-27B and MiniCPM5-2B served the ladder; every call has a receipt in bench/results/receipts/.
Referee
local · plain Python
1.07 ms for card H-01, from the stored run.
Supervisor queue
local · FastAPI
Server-sent events to the queue page; reviews in SQLite.

access probe · 2026-10-14 · bench/results/access-gate.json

12 models listed: 11 instruct, 0 audio

8 read the read-back “Copy, closing two-oh-eight, line Bravo.” as CLOSE · XV-208 · line B

response_format json_schema: rejected by every model probed, so the schema is checked in code (JSON mode)

token log-probs: returned by none of the models probed

/audio/transcriptions: “Transcription model not found: whisper-1”

bench/results/access-gate.json · Gemma-4-31B-IT’s probe answer was not valid JSON; it meets the rule in the ladder re-audit · friction log: docs/DX-REPORT.md

The ladder is also the sizing step for a plant: the smallest model that never waves a wrong read-back through is the one a site would run on its own servers. That benchmark is the next step, not a claim on this page.

Who pays · assumptions labelled

Sold per radio channel, to the people who own the audit.

1,318palm-oil mills operating in Indonesia, 2022 (Trase)

×30 %addressable: group-owned, radio-run mills (assumption)

×6radio channels per mill (assumption)

×USD 2,400per channel-year licence (assumption)

≈USD 5.7 Ma year from Indonesian mills alone: an estimate built from the labelled assumptions

Buyer. The HSE / K3 manager and the mill manager. Under PP 50/2012 a company with at least 100 workers or a high hazard potential must run SMK3, which is audited; a timestamped record of order, read-back, verdict and supervisor decision can serve as audit evidence.
Wedge. A free 30-day listen-only baseline at one mill: measure the plant’s own read-back error rate, the number nobody has published.
Privacy. Voice that identifies a person is personal data under Indonesia’s UU PDP 27/2022. Planned for production: run inside the company network, keep audio only for queue cards, score procedures, never people. Today the exchanges carry roles, not names, and no audio has been processed.
Compute. Gemma-4-31B-IT’s first pass over the shift: 300 calls, 178 answered (the other 122 failed at the gateway: server errors, time-outs and 4 rate-limited attempts), USD 0.000895 at the Token Factory’s price; the re-audit’s 118 calls for the cards they left cost USD 0.000512.

build/README.md · honest limits

What we abstain on.

SayAgain refuses to clear what it cannot check. So does this page: these are the claims it does not make yet.

ABSTAIN · honest limit 1
No recorded speech has been measured. Every result is text·scripted, each rung's result on clean written words.
build/README.md

ABSTAIN · honest limit 2
The held-out set is 89 cards, so the intervals are wide: the best model's 2/48 false alarms is a 4.17 % rate with a Wilson interval of 1.15 % to 13.98 % (bench/results/text/RESULTS.md).
build/README.md

ABSTAIN · honest limit 3
On scripted words the plain regex rung meets the rule with no model, so what a language model adds is unproven until the audio rows exist.
build/README.md

ABSTAIN · honest limit 4
The plant, its 120 tags and every exchange are fictional; the error mix is seeded, not sampled from a real site.
build/README.md

No audio evaluation in this submission

Audio rowerrors flagged, false-alarm load and false-VERIFIED on recorded speech at 0 dB
Failure case−6 dB look-alike pairs: XV-203 vs XV-208, P-114 vs P-140, MOV-31A vs MOV-31H
Kill testWhisper-large-v3 on 20 own-voice clips: forced English, forced Bahasa, auto
Blind auditorminutes per exchange and errors caught, by ear against the queue

Questions a judge would ask

FAQ

Where does the AMD Token Factory sit in the pipeline?

It does the one model job: span extraction. Each call returns the words for action, tag and line, and its receipt records the request id, the model served, the host and the cost.

Speech-to-text has not run on a recording yet; the Token Factory lists no transcription model. The referee and the queue are plain Python on the operator’s side.

Couldn’t an LLM just judge the read-back?

We measured that. DeepSeek-V4.1-Flash, given the whole exchange and no plant registry, passed 2 wrong read-backs as VERIFIED and passed the XV-205 order, a valve the plant does not have. In SayAgain the model only points at words; the verdict is code.

A regex already meets the rule. Why a model at all?

On clean written words it does: that is our own result, and the page says so. Real speech is not clean: recognition errors, code-switched Bahasa, other people’s phrasing. The audio rows decide whether a model earns its place; if they say it does not, the report will say that.

What happens when the radio is too noisy to hear?

On the written-word rows, a failed model call is NOT_AUDITED and an exchange the referee cannot clear reaches a human as ABSTAIN, in amber: listen yourself. How it behaves on noisy audio is not measured in this build; the −6 dB look-alike failure case needs the human recordings (no audio evaluation in this submission).

Does SayAgain control anything in the plant?

No. It has no write path to the plant. A card leaves the queue only when a supervisor confirms it or dismisses it with a reason.

Isn’t this a way to watch workers?

It is planned to check procedures, not people: the exchanges carry roles (operator, field tech), not names, and nothing scores a person. No audio has been processed yet. Voice that identifies a person is personal data under UU PDP 27/2022, so production is planned to run inside the company network, keeping audio only for cards in the queue.

Don’t radio loggers already do this?

They record and transcribe, and some spot keywords. Among the loggers we checked, none documents comparing an instruction with its read-back. Automatic read-back checking exists in air-traffic research (HAAWAII); we have not found it on plant radio.

Why is it called SayAgain?

“Say again” is the radio procedure word for “repeat that”. A wrong read-back deserves one; SayAgain makes sure somebody asks for it.

What on this page is real today?

Every verdict, trace, receipt and count comes from files in this repository: bench/results/, docs/replay/data/, evidence/. The exchanges are written scripts for a fictional plant (text·scripted); no recording has been measured. Dashed boxes are results that do not exist yet.

Nobody on the channel noticed. The radio log did.

One digit moves the crew to another line.

A whole shift, replayed and judged again.

Ten real cards from the stored shift.

Climb only as high as the rule needs.

The model points. Code decides. A human closes the card.

The model points at words

The words must be there

Every safety word counted

The plant registry has the last word

A human closes every card

The verdict is a pure function

Every stage says where it ran.

Sold per radio channel, to the people who own the audit.

What we abstain on.

FAQ

Nobody on the channel noticed. The radio
log did.