Abstract
Multimodal coding agents increasingly receive screenshots and other visual artifacts when repairing software, but existing benchmarks largely measure whether visual input improves patch success without testing whether agents connect evidence across multiple images to infer a shared cause. We introduce SWE-PolyVision, a benchmark for cross-image abductive reasoning in repository-level software engineering. Each task requires an agent to align states, differences, invariants, or transitions across visual observations, formulate a repository-level hypothesis, and produce a verified patch. SWE-PolyVision contains 91 instantiated tasks from 36 open-source organizations and 408 visual inputs. We evaluate ten coding models under Text-only, Native Vision, and Tool-mediated Vision settings. Results show no consistent advantage from visual access, while controlled interventions produce identical outcomes after shuffling images or reducing the input to one designated image. These findings suggest that current agents can receive multiple images without reliably using their joint structure.
Why cross-image reasoning?
A screenshot usually shows an effect rather than its cause. In real debugging, developers compare before/after states, viewports, themes, or successive interaction steps to determine which change is causal. SWE-PolyVision therefore treats multiple images as a structured set of observations rather than independent visual questions. The benchmark asks whether an agent can move from Observe → Align → Integrate → Abduce → Verify → Repair, connecting visual evidence to repository code and executable tests.
Benchmark design
Every task is grounded in a real repository snapshot, an issue description, ordered visual evidence, and a hidden evaluator. The 91-task main set spans 36 open-source organizations and includes at least two visual inputs per task. The same text and image set support three access conditions: Text-only, Native Vision, and Tool-mediated Vision. Environment failures and invalid visual-access runs are separated from hard failures, and primary comparisons use only matched tasks that are valid in both conditions.
Initial results
In the completed GPT-5.6-sol baseline, Text-only solved 32 of 72 matched tasks (44.4%), while Native Vision solved 27 (37.5%). The observed difference was −6.9 percentage points and was not statistically significant (two-sided exact McNemar test, p = 0.359). The paired outcomes still moved in both directions: 7 tasks succeeded only with Native Vision, while 12 succeeded only with Text-only. These flips show that images can change an agent’s repair behavior without establishing that it performed successful cross-image reasoning.
What remains open
End-to-end patch success cannot reveal whether an agent failed at visual observation, cross-image alignment, root-cause abduction, or patch implementation. The next analyses use single-image, shuffled-order, and leave-one-out interventions, together with trajectory labels, to separate information gain from context burden and genuine relational reasoning. Tool-mediated and multi-model runs are reported separately because a shared visual frontend measures a complete system, not the coding model in isolation.