How vision-language models learn where things are

The earlier comparison between Phi-3.5 Vision and Florence-2 used output shape to distinguish broad vision-language work from region- and pixel-level tasks. After the Vision Transformer primer, this note follows the spatial thread: what lets a VLM answer not only what is present, but where it is?

Here, spatial grounding means responding to a language query with image coordinates for the relevant region. Bounding a “child in green shorts” is a stronger test than merely saying children are present.

Grounding is an output contract #

There are two contracts here. The semantic contract is “which object does this phrase refer to?” The geometric contract is “how do those four numbers map to the original image?” A system can satisfy one and fail the other. A perfect box around the wrong person is not a partially correct answer to a referring-expression query.

Qwen’s vision-language models and DeepSeek-VL2 use different implementations, but three decisions keep recurring: how much spatial detail the encoder retains, how positions reach attention, and what output format the decoder has been trained to produce. An OCR system adds another localization task, but reading text regions should not be confused with general object grounding.

Account for the preprocessing geometry #

Suppose a crop begins at $(x_c, y_c)$ in the original image. The preprocessing pipeline scales it by $s_x, s_y$ and adds left/top padding $p_x, p_y$. A predicted point $(u, v)$ in model-input pixels maps back as:

If coordinates are normalized, first convert them into the model-input pixel convention. Apply the inverse transform to both corners, validate their order, then clip to the original image bounds. Record whether the format is corner coordinates or center/width/height, whether the upper bound is inclusive, and which image in a multi-image request the coordinates describe. These are API details, not implementation trivia.

For example, a crop starting at (200, 100) is resized by 0.5 and padded by (16, 16). A returned box (41, 31, 116, 81) maps to (250, 130, 400, 230) in the original image. Skipping the padding correction moves every edge; forgetting the crop origin places the right-sized box in the wrong part of the scene.

Higher resolution can preserve small text and small objects, but it increases visual-token and prefill cost. With fixed-size patches and no token merging, doubling both image dimensions roughly quadruples the patch count. Dense self-attention over those tokens has a quadratic score-work term. Tiling, windowed attention, and token merging change the actual cost, so do not turn that scaling argument into a latency prediction without measuring the chosen architecture.

Encode more than sequence position #

Ordinary rotary position embeddings describe positions in a one-dimensional token sequence. Image patches live on two axes; video adds time. Flattening all of that into one position can blur relationships such as above, left, adjacent, and earlier.

Multimodal RoPE assigns parts of the positional representation to temporal, vertical, and horizontal coordinates. This gives attention a structured position signal rather than asking it to infer every relationship from a flattened ordering. It does not, by itself, provide depth, metric distance, or reliable left/right reasoning.

Preserve local visual tokens #

An adapter commonly projects vision features into the language decoder’s hidden width. A projection alone does not imply fewer spatial tokens: changing a vector’s width and reducing the number of vectors are different operations. Pooling, resampling, and patch merging can reduce token count; those are the places to look when reasoning about context savings and lost detail.

Local tokens let the decoder attend to different regions instead of relying on one global summary. Qwen3-VL’s DeepStack also supplies visual features from multiple encoder depths to several decoder layers. That changes how visual information enters the language model; it is not proof that every small object survives preprocessing or that a generated box is calibrated.

Train the decoder to emit coordinates #

Spatial features are not enough. A model must also learn that a request for a region should produce something like [x_min, y_min, x_max, y_max]. Fine-tuning on referring-expression datasets such as RefCOCO can teach the decoder that output contract. Schema validation can reject malformed coordinates, but it cannot determine whether a well-formed box contains the requested object.

Make the evaluation harder than the illustration #

Start with intersection over union: $\mathrm{IoU} = |B_p \cap B_g| / |B_p \cup B_g|$, where predicted and reference boxes are expressed in the same original-image coordinate frame. Report localization success at a declared threshold, not just a screenshot that looks plausible.

Then separate the failure slices. Small objects stress resolution. Crowded scenes stress reference resolution. Cropping can remove the context needed to interpret “the cup to the left of the laptop.” A query for an absent object tests whether the system can abstain rather than draw a convincing box around something else.

Thresholds create an operating point, not an improvement for free. Rejecting uncertain predictions may increase precision while reducing coverage. Measure both on a held-out set, and do not treat a model’s verbal confidence or token likelihood as a calibrated probability of localization success. For video, split by scene or recording rather than adjacent frames; otherwise near-duplicates can make the evaluation look much stronger than deployment.

A useful comparison records the model revision, prompt, resize and crop policy, decoding settings, hardware, and latency alongside per-slice errors. That gives an engineer something actionable: change the input resolution, fix the coordinate transform, collect missing examples, or choose a different model. “Spatial reasoning score went up” does not tell us which of those happened.

Spatial ability is not binary. CLIP- and SigLIP-style global encoders do not erase every local clue, and models not designed primarily for detection sometimes recover approximate boxes. A better question than “does this model support grounding?” is “how much spatial information survives, at what resolution, and can the decoder return it reliably for this task?”

Grounding is also only one axis of a VLM. Language understanding, world knowledge, reasoning quality, latency, and deployment size still matter. A model that draws a perfect box but misunderstands the instruction has located the wrong problem very precisely.

Sources #

• Qwen2-VL
• DeepSeek-VL2
• Qwen3-VL reference implementation and architecture overview