Did the agent check, or just answer?
Brownout is an open harness for RL environments and evals. It grades how an agent reached its answer, not only whether the answer was right. On Cloud-OpsBench, one model never fetched the confirming evidence in 89 of its 322 correct answers. The benchmark's score paid for all of them.
call 1 list pods productcatalogservice Pending call 2 describe pod productcatalogservice Node-Selectors: env=prod hardware=high-mem FailedScheduling: 3 node(s) didn't match Pod's node affinity/selector answer node_selector_mismatch benchmark score 1 correct strict check 0 node labels never checked
A real saved run, shortened. The model was right, but never looked at the machines' labels.
Same reward, with or without the evidence.
I re-graded 1,095 saved runs with a second check next to the benchmark's score: the right answer and the confirming evidence the benchmark's own answer key lists.
| Model | Runs | Correct | Evidence never fetched | 95% CI |
|---|---|---|---|---|
| Nemotron-3-Nano-30B | 550 | 322 | 89 (27.6%) | 23.0–32.8% |
| DeepSeek-V4-Flash | 545 | 468 | 0 | 0–0.8% |
False alarms
Two LLM judges from different families, blind to each other, read all 89. Cases both called a false alarm: 0 of 82.
Misses
Runs my rule changes let through, checked the other way: 0 of 73 were missing the evidence.
Expert paths
In 80 of the 89, every expert trajectory the benchmark ships passes the same check. No LLM involved.
Not claimed: anything about other models, human-verified labels, or training. The answers are correct; the point is that the score cannot tell how they were reached. Full write-up →
Next: training on the same environments.
I'm turning the harness into an open post-training setup. None of this is released yet; v0.1 is planned for 30 Nov 2026.
The infra done before you start.
The aim: Brownout provisions the GPU box, the trainer and the environments, and connects the reward to training, so a first run works on day one.
- brownout up: drivers, Docker and a pinned image on Nebius or JarvisLabs
- Miles + SGLang + FSDP2 wired inside the image, set by one config
- Ready-made env packs, starting with LiveCodeBench via Harbor
- A versioned stack: we improve the base, you upgrade with one command
Every run repeatable and checkable.
With the stack pinned, a run becomes a record. Anyone could rerun it on their own GPUs and see what the reward was made of, including a second check like the one above.
- A run card per run: config, stack version, image digest, tasks, seed, cost
- One grader shared by eval and training
- Infra errors kept apart from model failures, never scored as 0
- Per-rollout answers and test logs, kept with the run
From task to trained checkpoint.
Four stages, one command each, with the eval and the trainer sharing the same reward code. Commands show the planned interface.
Pick tasks
Any Harbor dataset becomes an environment: instruction, sandbox image and tests.
brownout env add harbor:…Grade in a sandbox
Each answer runs against the task's own tests in a fresh container. Infra failures are kept apart from model failures.
brownout eval --env lcbTrain
GRPO with Miles, SGLang for rollouts and FSDP2 for the policy update.
brownout train --config …Evaluate again
Same tasks, same grader, before and after. Pass@1 with a bootstrap interval over tasks.
brownout report run-014Bring your own GPUs.
Planned: rent where it's cheapest, and Brownout sets up the box, runs the job and shuts it down.
GPU providers
Sandboxes for grading
Shipping in the open.
v0.1 · stack 0.1 planned 30 Nov
- brownout up on Nebius and JarvisLabs
- LiveCodeBench env pack via Harbor
- Eval with pass@1 and confidence intervals
- GRPO on Miles + SGLang, FSDP2
- A run card saved with every run
v0.2 · stack 0.2 next
- brownout reproduce <run-id>
- brownout doctor and cost estimates before launch
- Fully async rollouts
- SWE-Gym env pack
- Modal and Prime Intellect sandboxes
Questions and answers.
What exists today?
The harness: one episode loop, environments as folders, scoring, saved runs, and the strict checker behind the Cloud-OpsBench finding. LiveCodeBench tasks run through Harbor locally. Training is not wired in yet.
What does "a second check" mean?
The benchmark's score stays as it is. Next to it, a separate check asks whether the agent obtained the evidence that confirms its answer. It is a measuring stick, not a training reward.
What will v0.1 contain?
Planned for 30 Nov 2026: LiveCodeBench via Harbor, an eval with pass@1 and confidence intervals, and GRPO training on Miles and SGLang with FSDP2, on rented GPUs.
What is a Harbor task?
A folder with an instruction, a Dockerfile and a test script that writes a reward. Using Harbor's datasets as they are means the same tasks can serve eval and training.
Why FSDP2 instead of Megatron?
FSDP2 is native PyTorch and easier to read, change and debug for models in this size range. Miles supports both.
Is it open source?
Yes. The code is on GitHub now, including the finding's write-up, rubric and steps to reproduce it.
The code and the numbers are public.
Every number in the finding is recomputed by a script in the repo.