Planning with the Views

1 · Single-turn tracking vs. multi-turn planning

Single-turn tracking ≫ multi-turn planning. The best VLMs reach ~70% on short-horizon P2V/V2P but collapse to at most 21.3% on Interactive View Planning. Most models score below 10%; on long-horizon samples most fall below 3%. Open-weight models stay below 5% on IVP.

Takeaway. Local view-action knowledge does not compose into multi-turn plans.

1b · When models succeed, are they reasoning or just matching?

A competent spatial reasoner need not see the target to localize it: after a few informative moves it could infer where the target lies and submit its pose without ever visiting it. We test whether frontier successes work this way. For every successful IVP rollout, we check whether the agent observed any view within the success threshold (0.5 m / 30°) of the target before answering. Across all five models, at least 90% of successes (up to 99.1% for Gemini 3.1 Pro) follow such a visual encounter. Genuine inference of a correct pose, without ever visiting a threshold-close view, accounts for at most ~10% of successes.

Takeaway. The planning gap is really a cognitive gap: models mostly succeed by view matching after they reach the target, not by localizing it in advance.

2 · Difficulty along rotation and translation axes

Decomposing view distance into rotation and translation axes flips the difficulty signal. P2V/V2P degrade primarily with rotation distance (cumulative rotations are hard to mentally simulate); IVP reverses this — success collapses with position distance, since 3D translation needs spatial-layout understanding and path planning.

P2V/V2P (left two): accuracy falls along the rotation axis. IVP (right): success collapses along the position axis (~7× drop for GPT-5.4 Pro).

Takeaway. The two task families are bottlenecked by different spatial reasoning skills.

2b · Which per-sample factors correlate with success?

Spearman correlation between 12 per-sample factors (geometric distance, visual overlap, directional geometry) and each model's binary success makes the difficulty signal even sharper. P2V / V2P success correlates most with orientation_agreement (ρ up to +0.30, same-facing camera pairs are easier). IVP success collapses with pos_dist (ρ down to −0.42 for GPT-5.4 Pro), with rotation barely registering. Visual-overlap factors help single-turn tasks but contribute almost nothing to IVP.

Rows: 12 sample-level factors. Columns: per-model binary success on P2V, V2P, IVP. Blue cells are negative correlations (factor makes the sample harder), red cells positive. The strong negative column under pos_dist in the IVP panel is the position-bottleneck signature.

Takeaway. Position distance is to IVP what orientation agreement is to P2V / V2P — the single dominant predictor of per-sample difficulty.

3 · Does the turn budget bottleneck IVP?

Doubling the turn budget from 10 → 20 helps every model (Claude Opus 4.6 nearly doubles), but 20 → 30 is essentially free. Models exhaust their effective strategies before the turn limit runs out.

Takeaway. IVP is bottlenecked by planning ability, not by horizon length.

4 · Does higher-fidelity rendering change the picture?

We re-render the test set with 3D Gaussian Splatting (GS), then re-evaluate at budget = 10. IVP improves only marginally. P2V/V2P show mixed and sometimes large swings — Gemini 3.1 Pro gains +6.4 on P2V, while GPT-5.4 and Grok 4.20 Beta lose 14.4 and 13.0 points on V2P.

Same scene, same target — three independent runs on scene0518_00, all succeed.

Each GIF cycles through target → initial → agent's per-turn views. The three runs vary on two axes (model and renderer); all reach the target within the unified-distance threshold, but turn counts differ. Higher fidelity occasionally accelerates planning (4 turns vs. 10), yet — as the table below shows — it does not unlock the broader IVP gap.

Takeaway. The bottleneck is composing view changes, not the visual fidelity of each observation.

5 · Comparing training recipes for IVP

Direct PPO plateaus at 3.2%; GRPO with reward-variance filtering reaches 5.2%; iterating PPO with SFT on only successful trajectories (Success-Only Bootstrapping) gets to 6.2%. The breakthrough is recognizing that even failed trajectories encode valid view transitions: A → B is supervision regardless of the original goal. Compressing all exploration into a graph and reformulating sampled paths into view-planning demos takes Qwen2.5-VL-7B from 2.5% → 47.8%.

Takeaway. Useful supervision comes from the geometry recorded by failed exploration, not from filtering for successes.

5b · How does the ranking shift under No-Snap and No-Submit?

Two evaluation knobs could in principle inflate our IVP numbers: rotation snapping to the discrete 30° grid, and the submit requirement. We re-evaluate under No-Snap (raw rotation magnitudes executed as-is, no on-grid rounding) and No-Submit (success the moment the pose enters the threshold). The ordering between models is unchanged under all three protocols — our trained models continue to dominate the proprietary baselines by a wide margin.

No-Snap lowers every model: without rounding, per-step rotation residuals accumulate over 10 turns and the agent drifts off the on-grid pose distribution from which targets are drawn. No-Submit raises every model because no commit to a final answer is required. Across both relaxations, our framework's gains transfer cleanly.

Takeaway. The 47.8% headline number isn't an artefact of rotation snapping or of the submit step — relax either one and the ranking is preserved.

6 · How does the trained agent's coverage evolve over turns?

Tracked 3D point-cloud coverage reveals a clean two-phase strategy: scene coverage grows rapidly in early turns as the agent looks around, then plateaus while the target-intersection ratio accelerates in the middle turns as the agent moves toward the target (peaking near 55%). Base and frontier models show flat or erratic target coverage instead.

Left: scene coverage ratio. Right: target intersection ratio. Our trained model is the only one with sustained monotonic growth on the target axis.

Full model comparison (all 15 models)

Takeaway. The trained policy follows a goal-directed two-phase trajectory; baselines do not.

7 · How does training reshape per-layer image attention?

Image-attention fraction (the share of response-token attention pointed at image tokens) reveals two patterns. Layer-wise: our trained model attends more to images in early layers (L0–L4) and less in deep layers (L8+) than the base — it grounds visually early, then operates in text space. Turn-wise: image attention decreases monotonically across turns, consistent with progressive information accumulation, while the base model stays flat.

Per-layer image-attention breakdown — full 28 layers (click to collapse)

Image-attention fraction per layer × turn for all 28 layers. Our trained model front-loads visual grounding then drops off; the base model is flatter across both axes. Click the figure to open at native resolution.

Takeaway. Training reshapes how the VLM uses its visual stream, not just what it outputs.

8 · How is turn-usage distributed across models?

The base Qwen and GPT-5.4 Pro terminate most rollouts in a single turn (no exploration). Our trained model and Gemini 3.1 Pro use the full 10-turn budget. Of the rollouts that do use all turns, our model maintains a much higher success rate on harder episodes.

(a) Total rollouts by turn count.  (b) Successful rollouts.  (c) Success rate by turns used.

Takeaway. Frontier VLMs that look like they're solving IVP often submit instantly without planning.

9 · Do the learned priors transfer to other view-related tasks?

Under identical GRPO post-training, our trained model beats the base on both internal and external view-dependent tasks. On the external MindCube benchmark (no shared scenes / actions / rendering pipeline) we gain ~10 points.

Takeaway. Interactive view planning is not a narrow skill — its priors strengthen view-dependent reasoning both within and beyond ViewSuite.