Terminal-agent benchmark built end to end, from task taxonomy to grading pipeline
Agentic evaluation is where crowd models break. Task authoring, Dockerised environment engineering, verified gold solutions and step-level trajectory grading, run as one accountable workstream.
A run that ends green can still be wrong in the middle. Scoring the trajectory is what separates an agent benchmark from a unit test.
A documented program shape with illustrative volumes, not a named client reference.
The problem
The team needed a benchmark that would survive external scrutiny. Reproducible environments, unambiguous success criteria, and grading that two independent engineers would agree on.
How it runs
- 1
Taxonomy first
Capability dimensions and difficulty bands defined before any task was authored, so coverage was a design decision rather than an artifact of what was easy to write.
- 2
Environment engineering
Each task ships as a pinned Docker image with a deterministic test harness. A task is not accepted into the suite until a gold solution passes the harness from a clean container.
- 3
Adversarial review
Every task is attacked by a second engineer looking for shortcut solutions, ambiguous success criteria and environment leakage before it enters the suite.
- 4
Trajectory grading
Step-level rubric scoring of agent trajectories with agreement tracked between graders, so the benchmark reports capability rather than grader variance.
Outcome
- Reproducibility
- 100%
- Gold solution passes from clean container
- Grader agreement
- κ ≥ 0.7
- Tracked per rubric dimension
- Shortcut audit
- Every task
- Adversarially reviewed before inclusion
- Handover
- Full
- Harness runs in your CI, not ours
What makes it stick
Benchmark programs are handed over complete. The harness runs in your infrastructure. We are not a dependency in your evaluation loop.
Other patterns
A 1,000-item coding evaluation, delivered a day early with the audit attached
The standard six-week play: pilot scoped against the client’s own acceptance bar, pod calibrated to an agreement gate before production, batch delivered with full QA evidence, converted to a dedicated pod inside two weeks.
ReadAI Data OperationsAuditing someone else’s output, the highest-trust way to start
A review and audit layer operated over a third party’s delivery: the client’s own crowd, or another vendor. We publish the metrics either way, including the ones that are inconvenient.
Read