Abstract
Coding agents have made rapid progress on repository-level software engineering tasks, but existing evaluations usually begin with a known issue and a predefined success signal. We introduce SWE-Prometheus, a benchmark in which an agent receives a fixed snapshot of a real repository and a general governance objective, but no defect list, base score, or oracle patch. The agent must inspect the repository, prioritize risks, implement a retrofit, and verify its claims under a limited budget. The benchmark evaluates six dimensions: Tests & CI, Code Quality Gates, Documentation & Collaboration, Structure & Maintainability, Reproducible Environment, and Dependency & Security Health, using executable base/treated evidence, hidden characterization tests, and multi-teacher scoring. The dataset60 release contains 60 repositories, including a 22-instance public shared subset and 38 private instances. Repository governance produces measurable differences between agents, but only where the evaluator executes.
Beyond fixing a known issue
Real maintenance includes discovering missing tests, fragile quality gates, undocumented setup, dependency drift, and collaboration gaps. SWE-Prometheus turns that open-ended work into a reproducible retrofit task: the agent receives a repository and a general governance objective, but must decide what matters, make coordinated changes, and preserve existing behavior.
Six governance dimensions
The benchmark evaluates Tests & CI, Code Quality Gates, Documentation & Collaboration, Structure & Maintainability, Reproducible Environment, and Dependency & Security Health. Each dimension is scored from 1 to 5 using evidence from the repository before and after treatment. Credit is tied to executed commands, collected tests, clean-environment checks, and other probes rather than the presence of configuration files alone.
Evidence and safety
For each rollout, the evaluator reconstructs the base snapshot, applies the agent patch to an independent treated snapshot, and reruns the same probes. Hidden characterization tests check whether existing behavior survives. Mutation testing labels verification gates as detected, blind, vacuous, or none, making it possible to distinguish a genuine safety signal from a test suite that merely exits successfully.
Dataset60 results
The release contains 60 real repositories: 22 public shared instances and 38 private holdout instances. On the public subset, one rollout for each of ten models produced mean Normalized Governance Improvement values from 0.0568 to 0.5760, with observed behavior-breakage rates from 0% to 23%. The results are descriptive rather than a definitive ranking: nine models used the same pi scaffold, while Claude-Opus-5 used Claude Code.
Interpretation
SWE-Prometheus measures repository stewardship under a specific protocol, not production readiness or the absence of vulnerabilities. The current release has one rollout per model-instance pair, uneven verification strength, teacher-model calibration limits, and scaffold differences. Repeated runs, stronger behavior gates, independent expert calibration, and a truly isolated private evaluator are needed for stable comparisons.