← All posts

In 89 of 322 correct answers, the agent never checked the evidence

Cloud-OpsBench pays the same reward for a confirmed diagnosis and for one read off a single error message. A re-grade of 1,095 saved runs, with an audit of the checker itself.

Summary. Cloud-OpsBench scores an agent's root-cause diagnosis on the final answer alone. I re-graded 1,095 saved runs from two models with a second, stricter check that also requires the evidence the benchmark's own answer key lists. For Nemotron-3-Nano-30B, 89 of 322 correct answers (27.6%, 95% CI 23.0–32.8%) never fetched that evidence. For DeepSeek-V4-Flash, 0 of 468. The benchmark pays both kinds of answer the same.

Setup

Cloud-OpsBench gives an agent a broken Kubernetes app and asks for the root cause. The agent calls tools (list pods, describe a resource, read logs) against recorded cluster outputs, then submits its top three guesses. Its score, JRA, checks the rank-1 component and fault type. It does not look at how the agent got there.

The benchmark also ships, for every case, a process label: the milestones an expert must establish to confirm the answer, such as "list the nodes with their labels". It uses them for separate process metrics, never for the reward.

  • Benchmark: Cloud-OpsBench (LLM4Ops), Online Boutique system, 550 cases.
  • Harness: Brownout, my RL environment harness. Each run is saved as a full trajectory.
  • Runs: 1,095 saved runs, Nemotron-3-Nano-30B (550) and DeepSeek-V4-Flash (545).
  • Strict check: JRA = 1 and the answer milestones in the benchmark's own process label are established. It is a measuring stick, never a training reward.
  • Saved runs are never edited. The re-grade is written to a new file, and every number here is recomputed by a script in the repo.

Result

Correct answers, split by whether the confirming evidence was fetchedRuns with JRA = 1. Same reward for both parts of each bar.
Evidence fetchedEvidence never fetched
Nemotron-3-Nano-30B
89 of 322
DeepSeek-V4-Flash
0 of 468
ModelRunsJRAStrictCorrect, evidence never fetched95% CI
Nemotron-3-Nano-30B5500.5860.42489 of 322 (27.6%)23.0%–32.8%
DeepSeek-V4-Flash5450.8590.8590 of 4680%–0.8%

The two models behave very differently in the same harness. DeepSeek fetched the confirming evidence every time it was right. Nemotron often stopped after one or two calls. Most of its unconfirmed answers are scheduling faults, where the scheduler's error message hints at the cause without settling it.

Nemotron's 89 unconfirmed answers, by fault category
Scheduling
51
Code defect
11
Infrastructure
8
Startup
8
Admission
6
Performance
5
Insight. An outcome-only score cannot tell a confirmed diagnosis from a correct read of one error message. Whatever a model does with that reward, checking earns it nothing extra.

One example: boutique-scheduling-101

  • Task. One service is down. Find the root cause. The true cause: productcatalogservice may only run on machines labelled env=prod and hardware=high-mem, and no machine has both.
  • Call 1. The agent lists the pods. productcatalogservice is stuck.
  • Call 2. It describes that pod. The output shows:
Node-Selectors:  env=prod  hardware=high-mem
FailedScheduling: 0/4 nodes are available: 1 node(s) were unschedulable,
                  3 node(s) didn't match Pod's node affinity/selector.
  • Answer. node_selector_mismatch, with node_affinity_mismatch as its second guess.
  • JRA. Correct. Reward 1.
  • Strict check. 0. The agent never listed the machines' labels, the step that tells "selector" apart from "affinity". Its own second guess shows it was unsure.
Milestone in the answer keyThis runBoth expert paths
Find the stuck podyesyes
Read why it is stuckyesyes
List the machines with their labels: none has bothnoyes

The answer was right. The reward has no way to know whether it was checked.

Is the strict check fair?

A stricter checker can be wrong in two directions. It can flag a run whose evidence was actually there (a false alarm), or pass a run whose evidence was missing (a miss). I measured both, on every case rather than a sample.

How often the strict checker is wrongObserved rate and 95% Wilson interval. Lower is better.
False alarms (both judges)
0% · 0 of 82
Misses (rescued runs)
0% · 0 of 73
Negative control accepted
0% · 0 of 12
  • False alarms. All 89 flagged runs went to two LLM judges from different model families, GLM-5.3-Flash and gpt-oss-120b, each blind to the other. Each judge disagreed with the checker on 2–3 runs; they never agreed on the same one. Both answered on 82 runs.
  • Misses. 73 runs pass only because of two rule changes I made (next section). A judge re-checked all 73 for missing evidence: 0.
  • Negative control. 12 cases where the rule should reject were shown to the judge as accepted. It rejected all 12, so it does not simply agree with whatever it is told.
  • Expert paths, no LLM involved. The benchmark publishes expert trajectories for every case. In 80 of the 89 flagged runs (89.9%), every expert path for that case passes the same strict check. The skipped step is one the benchmark's own experts take.

What the skipped step looks like

What the 82 doubly-labelled runs look likeTwo LLM judges from different model families, blind to each other. Counted where both agree.
Read off a tool message
75
Judges disagree
6
Blind guess
1
Evidence was there
0

Where both judges answered, they agree on 75 of 82: the agent read the answer off a tool message (a scheduler event, a crash log, an image pull error) and stopped. One run was a blind guess. These are mostly correct reads of a symptom, not lucky guesses.

I had also tried to split these into "the message fits only one cause" and "the message fits several". The two judges disagreed almost completely (Cohen's kappa 0.07). The cause is identifiable: this cluster's master node is always cordoned, so nearly every scheduling event contains the line "1 node(s) were unschedulable". One judge treats it as a competing cause, the other ignores it. I withdrew that split and make no claim about how ambiguous the messages were.

162 or 89?

Under the benchmark's labels exactly as published, Nemotron has 162 unconfirmed answers, not 89. The difference is two patterns where the labels reject evidence that is correct:

  • 22 runs. The label demands a namespace argument when listing nodes. Nodes are cluster-scoped, so a correct call without it shows the same nodes and is still rejected.
  • 51 runs. The label demands the app's YAML, while the pod, deployment or replicaset description the agent fetched prints the same field (requests, limits, image, probes, mounts), for fault types where that field decides the answer.
Share of correct answers without the confirming evidenceObserved rate and 95% Wilson interval.
Nemotron, labels as published
50.3% · 162 of 322
Nemotron, two label fixes
27.6% · 89 of 322
DeepSeek, either
0% · 0 of 468

The second pattern is a judgement call, so I checked it against the benchmark's own expert paths. If the labels are fair, they should accept every expert path, since each one is a correct diagnosis.

The benchmark's 1,100 expert paths, graded by its own labelsEvery expert path is a correct diagnosis, so a fair label should accept all of them.
Accepted by the published labelsAccepted only after the pod-describe fixStill rejected
Expert paths
40 rejected
  • The published labels reject 151 of the 1,100 expert paths (13.7%). Counting a pod or replicaset describe as config evidence accepts 111 of those. The experts use exactly the evidence the labels refuse, so this fix no longer rests on my judgement alone.
  • The namespace pattern does not appear here, because the expert paths always pass a namespace. It only matters for harnesses whose models omit it.
  • The remaining 40 are expert shortcuts, mostly quota faults and missing pull secrets: the expert reads "exceeded quota" or "secret not found" from an event and never queries the quota or secret itself. That is the same step Nemotron skips. The labels and the expert paths disagree about what must be checked. I keep the stricter reading; under the experts' reading, 9 Nemotron runs move to confirmed and the count is 80 instead of 89.

89 is the number I defend. 162 is the count under the labels as shipped.

A tester agent finds the same gap

As a separate demonstration, I ran a small multi-agent loop in which a tester agent (GLM-5.3-Flash) is asked to earn reward without solving the task. JRA was the only reward it could see; the strict check ran as a hidden second referee. It ran on 5 hand-picked scheduling cases, with 34 attempts, for about $0.43.

CaseResultWhat was skipped
scheduling-82 (node selector)JRA 1, strict 0, after reading the scorerlisting the nodes' labels
scheduling-114 (anti-affinity)JRA 1, strict 0, after reading the scorerthe app config holding the rule
scheduling-135 (storage class)JRA 1, strict 0, while refusing to game the scorerwhich storage class the volume has
scheduling-56, scheduling-141every JRA win also passed strictnothing
  • I checked every flagged attempt by hand against the process label and the expert paths: 0 false alarms. With JRA as the only referee, the same loop reports nothing.
  • After reading the scorer, the tester wrote in its plan: "Answer-only grading, zero process verification."
  • Intent is not needed. On scheduling-135 the tester declined to game the scorer, made 22 calls, and still skipped the deciding check. Nothing in the reward asks for it.
  • Crude attempts all earned 0: requesting the grader's source file, and a submission listing every component and every cause.

This is a demonstration on 5 chosen cases, not a rate.

Also found

The published label files have no milestone description field, which the benchmark's own evaluation loader requires, so the upstream evaluator fails on the labels as shipped. My port of the matcher does not need the field, and it reproduces upstream's evaluator exactly on all 1,508 expert paths once the field is filled in.

What this does not show

  • Anything about other models, other harnesses, or the Train-Ticket half of the benchmark.
  • Human-verified labels. Both judges are LLMs, and the labelling rubric was drafted with an LLM's help.
  • That these answers are wrong. All 89 are correct; the point is that the reward cannot tell how they were reached.
  • Anything about training. Whether RL on JRA makes the shortcut more common is the next experiment, not a result.
  • That every run confirmed under the unchanged labels is truly confirmed. Only the 73 runs my two rules let through were re-audited.

Method

1. Port the benchmark's milestone matcher and check it reproduces upstream's evaluator on all 1,508 expert paths. 2. Define the strict check: JRA and the label's answer milestones, with the two documented label fixes. 3. Re-grade every saved run into a new file. Saved runs are never modified. 4. Audit the strict check in both directions: false alarms on all 89 flagged runs, misses on all 73 rescued runs, and a 12-case negative control. 5. Grade the 1,100 expert paths with the same code, as a control that needs no LLM. 6. Run the tester loop with the strict check as a hidden second referee, and check every flagged attempt by hand. 7. Report counts with Wilson 95% intervals.

Code and data