work in progress open source, built by one person

Did the agent check, or just answer?

Brownout is an open harness for RL environments and evals. It grades how an agent reached its answer, not only whether the answer was right. On Cloud-OpsBench, one model never fetched the confirming evidence in 89 of its 322 correct answers. The benchmark's score paid for all of them.

A real saved run, shortened. The model was right, but never looked at the machines' labels.

Working withHarborMilesSGLangPyTorch FSDP2Docker
First result · Cloud-OpsBench

Same reward, with or without the evidence.

I re-graded 1,095 saved runs with a second check next to the benchmark's score: the right answer and the confirming evidence the benchmark's own answer key lists.

ModelRunsCorrectEvidence never fetched95% CI
Nemotron-3-Nano-30B55032289 (27.6%)23.0–32.8%
DeepSeek-V4-Flash54546800–0.8%

False alarms

Two LLM judges from different families, blind to each other, read all 89. Cases both called a false alarm: 0 of 82.

Misses

Runs my rule changes let through, checked the other way: 0 of 73 were missing the evidence.

Expert paths

In 80 of the 89, every expert trajectory the benchmark ships passes the same check. No LLM involved.

Not claimed: anything about other models, human-verified labels, or training. The answers are correct; the point is that the score cannot tell how they were reached. Full write-up →

Where it's going · in progress

Next: training on the same environments.

I'm turning the harness into an open post-training setup. None of this is released yet; v0.1 is planned for 30 Nov 2026.

Set up for you

The infra done before you start.

The aim: Brownout provisions the GPU box, the trainer and the environments, and connects the reward to training, so a first run works on day one.

  • brownout up: drivers, Docker and a pinned image on Nebius or JarvisLabs
  • Miles + SGLang + FSDP2 wired inside the image, set by one config
  • Ready-made env packs, starting with LiveCodeBench via Harbor
  • A versioned stack: we improve the base, you upgrade with one command
Runs you can trust

Every run repeatable and checkable.

With the stack pinned, a run becomes a record. Anyone could rerun it on their own GPUs and see what the reward was made of, including a second check like the one above.

  • A run card per run: config, stack version, image digest, tasks, seed, cost
  • One grader shared by eval and training
  • Infra errors kept apart from model failures, never scored as 0
  • Per-rollout answers and test logs, kept with the run
Planned loop

From task to trained checkpoint.

Four stages, one command each, with the eval and the trainer sharing the same reward code. Commands show the planned interface.

01

Pick tasks

Any Harbor dataset becomes an environment: instruction, sandbox image and tests.

brownout env add harbor:…
02

Grade in a sandbox

Each answer runs against the task's own tests in a fresh container. Infra failures are kept apart from model failures.

brownout eval --env lcb
03

Train

GRPO with Miles, SGLang for rollouts and FSDP2 for the policy update.

brownout train --config …
04

Evaluate again

Same tasks, same grader, before and after. Pass@1 with a bootstrap interval over tasks.

brownout report run-014
Compute

Bring your own GPUs.

Planned: rent where it's cheapest, and Brownout sets up the box, runs the job and shuts it down.

GPU providers

NebiusSSH virtual machines
v0.1
JarvisLabsSSH virtual machines
v0.1
ModalServerless GPU containers
planned

Sandboxes for grading

Local DockerOn the GPU machine itself
v0.1
Prime Intellect sandboxesRemote containers for env creation
planned
Modal sandboxesRemote containers at high concurrency
planned
Roadmap

Shipping in the open.

v0.1 · stack 0.1 planned 30 Nov

  • brownout up on Nebius and JarvisLabs
  • LiveCodeBench env pack via Harbor
  • Eval with pass@1 and confidence intervals
  • GRPO on Miles + SGLang, FSDP2
  • A run card saved with every run

v0.2 · stack 0.2 next

  • brownout reproduce <run-id>
  • brownout doctor and cost estimates before launch
  • Fully async rollouts
  • SWE-Gym env pack
  • Modal and Prime Intellect sandboxes
FAQ

Questions and answers.

What exists today?

The harness: one episode loop, environments as folders, scoring, saved runs, and the strict checker behind the Cloud-OpsBench finding. LiveCodeBench tasks run through Harbor locally. Training is not wired in yet.

What does "a second check" mean?

The benchmark's score stays as it is. Next to it, a separate check asks whether the agent obtained the evidence that confirms its answer. It is a measuring stick, not a training reward.

What will v0.1 contain?

Planned for 30 Nov 2026: LiveCodeBench via Harbor, an eval with pass@1 and confidence intervals, and GRPO training on Miles and SGLang with FSDP2, on rented GPUs.

What is a Harbor task?

A folder with an instruction, a Dockerfile and a test script that writes a reward. Using Harbor's datasets as they are means the same tasks can serve eval and training.

Why FSDP2 instead of Megatron?

FSDP2 is native PyTorch and easier to read, change and debug for models in this size range. Miles supports both.

Is it open source?

Yes. The code is on GitHub now, including the finding's write-up, rubric and steps to reproduce it.

Read more

The code and the numbers are public.

Every number in the finding is recomputed by a script in the repo.