Terminal & System Operations
Terminal tasks built around real system operations, in a Terminal-Bench-aligned delivery format. Each task pins an initial environment, a precise instruction, criterion rubrics, and a deterministic offline verifier — so completion is judged against a target state, reproducibly, in a clean container.
Coverage
- Command chains
- System diagnosis
- Environment repair
- SQLite / data operations
- Deterministic verification
Deliverables
- Initial environment
- Instruction & task spec
- Rubrics
- Verification scripts
- Selection rationale & validation summary
From source to acceptance
We don't hand-label a pile and ship it. Every category moves through a closed, instrumented loop — generated to a brief, checked by machines, adjudicated by experts, and traceable end to end — but the path each data type takes is its own.
- 01
Environment Pinning
Each task begins from a pinned container image with explicit resource and timeout bounds, fixing a clean, identical starting state that any evaluation can reconstruct exactly.
- 02
Required-Behavior Authoring
A precise instruction specifies the required behaviour and target state in full, spanning diagnosis, repair, and data operations without leaving room for interpretation.
- 03
Deterministic Verifier
A deterministic offline verifier is constructed to judge completion strictly against the target state, so grading is reproducible, machine-settleable, and free of network dependence.
- 04
Clean-Room Reproduction
The task is re-run from scratch in a fresh container to confirm the verifier passes on the reference solution and fails on the flawed baseline, proving the oracle discriminates.
- 05
Rationale & Validation
A selection rationale and validation summary record why each task was kept and evidence it reproduces cleanly, giving the deliverable an auditable difficulty and reproducibility trail.
Every run emits a learning signal that feeds back into the source set — the pipeline tightens itself, batch over batch.
See the data itself
One real, trimmed sample from this category — the scenarios it serves, why it matters for training, and the shape of the data as delivered.
Where it’s used
- Evaluating system-operations and debugging under real constraints
- Training deterministic, reproducible tool-use behaviour
- Benchmarking against a Terminal-Bench-aligned format
Why it matters for training
Essential
Deterministic offline verifiers make these tasks cheaply and reproducibly gradable — the property RL loops need most.
Notable features
- README.md
- dataset_manifest.json
- selection_rationale.json
- reports/
- dataset_export/
- gh_git_history_audit_hoo_a0d76575/
- tests/
- solution/
- environment/
- gh_sqlite_data_repair_pa_e7ebb8b6/
- frontier_supervisor_stal_4ca7864e/
- premium_security_policy_2cc8f58e/
# Repair the Snapshot Permission Audit
This repository contains a small Python CLI that audits a filesystem
snapshot stored in SQLite and writes an unsafe-permission remediation
report. The current implementation is intentionally flawed: it loads too
much data into nested Python loops, merges policy layers incorrectly,
reports superseded snapshot rows, and emits chmod remediation rows for
symlinks.
Keep the public command surface unchanged:
./bin/snapshot-perm-audit --db fixtures/snapshots/prod.sqlite \
--policy fixtures/snapshots/policy.toml \
--json reports/snapshot.json --csv reports/remediation.csv
## Required Behavior
1. Policy precedence: default_policy.toml < policy.toml < CLI overrides.
Merge by stable rule id, independent of TOML table order.
2. Audit only the latest observation per normalized path (by observed_at;
ties broken by highest SQLite rowid).Need a sample or a custom build?
Tell us your spec and scale — we deliver to order.
← Back to Data