SWE Engineering Tasks
Software-engineering tasks drawn from real codebases. Each instance ships a runnable repo environment, the gold patch, the test patch, and an evaluation script — so a candidate fix is judged pass or fail against real FAIL_TO_PASS / PASS_TO_PASS tests, not toy problems.
Coverage
- GitHub PRs
- Issue fixes
- 8+ language ecosystems
- Test repair
- Real engineering constraints
Deliverables
- Task package (JSONL)
- Per-instance repo environment
- Gold & test patches
- Evaluation scripts
- Pass-rate records & acceptance report
From source to acceptance
We don't hand-label a pile and ship it. Every category moves through a closed, instrumented loop — generated to a brief, checked by machines, adjudicated by experts, and traceable end to end — but the path each data type takes is its own.
- 01
Task Sourcing
Tasks are drawn from real merged pull requests and their linked issues, inheriting genuine engineering constraints and acceptance criteria that were settled by working maintainers.
- 02
Environment Freeze
Each instance's dependencies and repository snapshot are frozen into a single-command reproducible environment, so a candidate fix is always evaluated against the exact same ground truth.
- 03
Oracle Construction
The gold patch and its accompanying test patch are reconstructed to form a runnable oracle, defining precisely which previously failing tests a correct solution must turn green.
- 04
Automatic Verification
An evaluation harness applies a candidate patch and runs the fail-to-pass and regression suites on hidden tests, settling pass or fail objectively rather than by human judgement.
- 05
Coverage Check
The set is audited for balance across languages and ecosystems, so measured ability reflects general engineering skill rather than fluency in a single stack.
- 06
Acceptance Report
A final stratified sample and acceptance report confirm that every instance builds, verifies, and traces back to its originating change before the package ships.
Every run emits a learning signal that feeds back into the source set — the pipeline tightens itself, batch over batch.
See the data itself
One real, trimmed sample from this category — the scenarios it serves, why it matters for training, and the shape of the data as delivered.
Where it’s used
- Evaluating patch-generation and issue-resolution ability
- Training on real multi-file bug fixes with test oracles
- Regression-safe RL from verifiable pass/fail rewards
Why it matters for training
Critical
The gold standard for measurable coding ability: every instance has a real test oracle, so reward is objective rather than judged.
Notable features
- catalog.json
- instances/
- pallets-flask-5917-3e16c3fc/
- evaluation.sh
- environment/
- Dockerfile
- materials/
- evidence/
- fastify-fastify-6716-01c9f40a/
- tokio-rs-tokio-pr8131-me-4faf89d2/
- symfony-symfony-64261-55a9b0ea/
- grpc-grpc-go-9102-15bfd03d/
- rubocop-rubocop-15139-19b52bd4/
# SWE Hard 50 50 high-difficulty software-engineering instances spanning Python, JavaScript, Java, Rust, Go, C#, PHP, and Ruby. For patch-generation, code-repair, and issue-resolution evaluation. Each instance provides a task description, gold code patch, test patch, environment build materials, and an evaluation script — for independent inspection, reproduction, and re-integration. - dataset.jsonl — all 50 instances, one JSON per line (primary index) - catalog.json — dataset catalogue and statistics - instances/<id>/task.json — full structured record - instances/<id>/evaluation.sh — evaluation script - instances/<id>/environment/Dockerfile — environment build
Need a sample or a custom build?
Tell us your spec and scale — we deliver to order.
← Back to Data