GUI Grounding & Interaction
Models can describe a screenshot; the open question is whether they can operate the interface in it. We build GUI data that pins natural-language intent to exact on-screen targets — every interactive element localized, every action grounded in the pixels and the accessibility tree it acted on, and every result checked against the interface state it produced. Captured across web, desktop, and mobile, so grounding transfers rather than overfitting to a single surface.
Coverage
- Web, desktop & mobile
- Element grounding
- Action trajectories
- Accessibility & DOM trees
- Screen-state verification
- Cross-resolution rendering
Deliverables
- Annotated screenshots
- Element boxes, roles & labels
- Action sequences
- Accessibility / DOM trees
- Step-level state checks
- Task & instruction specs
From source to acceptance
We don't hand-label a pile and ship it. Every category moves through a closed, instrumented loop — generated to a brief, checked by machines, adjudicated by experts, and traceable end to end — but the path each data type takes is its own.
- 01
Interface Capture
Screens are captured across web, desktop, and mobile at real resolutions, each frame paired with its accessibility tree and DOM so every visible element carries a machine-readable identity rather than only pixels.
- 02
Element Grounding
Each interactive target is localized with a tight bounding box, role, and label, so a referring instruction resolves to one unambiguous on-screen element instead of an approximate region.
- 03
Action-Trajectory Authoring
Tasks are executed step by step — click, type, scroll, drag — with every action recorded against the screen state it acted on, capturing the full instruction-to-interface loop rather than an isolated frame.
- 04
State Verification
Each consequential action is checked against the resulting interface state, turning a screen recording into a sequence of verified transitions with unambiguous success and failure signals.
- 05
Cross-Surface Balancing
Coverage is balanced by design across platforms, resolutions, themes, and layouts, so grounding learns the invariant structure of an interface instead of the quirks of a single application.
- 06
Consistency Review
A review pass reconciles element labels, action targets, and state checks against the raw capture, so the recorded trajectory matches exactly what happened on screen.
Every run emits a learning signal that feeds back into the source set — the pipeline tightens itself, batch over batch.
Where it’s used
- Training GUI grounding and referring-expression localization
- Teaching computer-use agents to plan and act on real interfaces
- Evaluating click / type accuracy against verified screen state
Why it matters for training
Critical
GUI control is the bridge between language and software: without pixel-accurate grounding and verifiable screen state, an agent can plan an action but not reliably land it.
Notable features
Need a sample or a custom build?
Tell us your spec and scale — we deliver to order.
← Back to Data