Zero-shot open-vocabulary segmentation labels pixels with class names alone. Image-level pre-training and the ambiguity of language leave a large gap to supervised models. RnS closes most of it with a visual support set: a handful of images with pixel masks.
The backbone stays frozen. Each support image contributes one compact feature per class it contains, and at test time k-nearest-neighbour retrieval picks only the support features that matter for the image at hand.
Class names and visual features are blended into fused class features, and a lightweight linear classifier is trained per test image on both, weighting each class by its relevance. No hand-crafted late fusion of two predictions.
Support can be partial: classes without images fall back to zero-shot pseudo-labels, classes without a name get a neutral text prior. The memory grows at any time, and a few examples of one particular object are enough to segment that object.
Open-vocabulary segmentation (OVS) extends the zero-shot recognition capabilities of vision–language models (VLMs) to pixel-level prediction, enabling segmentation of arbitrary categories specified by text prompts. Despite recent progress, OVS lags behind fully supervised approaches due to two challenges: the coarse image-level supervision used to train VLMs and the semantic ambiguity of natural language. We address these limitations by introducing a few-shot setting that augments textual prompts with a support set of pixel-annotated images. Building on this, we propose a retrieval-augmented test-time adapter that learns a lightweight, per-image classifier by fusing textual and visual support features. Unlike prior methods relying on late, hand-crafted fusion, our approach performs learned, per-query fusion, achieving stronger synergy between modalities. The method supports continually expanding support sets, and adapts seamlessly to fine-grained tasks such as personalized segmentation. Experiments show that we significantly narrow the gap between zero-shot and supervised segmentation while preserving open-vocabulary ability.
Two stages: build a compact support memory once, then adapt to every test image with a few gradient steps on frozen VLM features.
Left: every support image is pooled under its masks into per-image visual class features; their union is the visual support set V. Summing per class and blending with the text feature of the class name, fcλ = λ tc + (1 − λ) vc, gives the fused class features F. Right: for a test image, the nearest support features of its patch or region features are retrieved, and a linear classifier gθ is trained on them and on the fused features of the retrieved classes with cross-entropy, each sample weighted by a class relevance weight wc. The classifier then labels the same features. With SAM 2.1 proposals, region-pooled features are labelled instead of patches.
Pool patch features of each support image under each class mask into vci. Sum per class into vc and blend with the text feature tc at several mixing ratios λ. Adding an image only appends to the memory.
For every patch (or SAM region) feature of the test image take its k nearest neighbours in V. Their union Vr, and the fused features Fr of the retrieved classes, are the training set for this image.
Fit a linear classifier gθ with cross-entropy on Vr ∪ Fr. Each class is weighted by wc, the softmax similarity between the image-level feature and the class names, which suppresses retrieved classes that are irrelevant here.
Apply gθ to the same features and upsample, or paint the labels of SAM regions back at full resolution. Classes without visual support are handled by a pseudo-label loss on zero-shot predictions, so nothing outside the memory is lost.
Average mIoU over six OVS benchmarks (VOC, Pascal Context, COCO Object, COCO-Stuff, Cityscapes, ADE20K). Support images are sampled from each training split; four random seeds.
With a single image per class, RnS improves zero-shot by +8.4 (OpenCLIP) and +19.1 (DINOv3.txt) mIoU, and keeps improving as the support grows. kNN-CLIP and FreeDA fuse text by hand and saturate early; the gap widens on the stronger backbone. Text matters most when support is scarce (compare with the w/o-text variants). Hover a legend entry to isolate a curve.
Left: as more classes lose their visual support, RnS degrades gracefully thanks to the pseudo-label loss on zero-shot predictions, while kNN-CLIP and FreeDA soon drop below zero-shot. Right: when class names are missing, RnS stays best over the whole range and never falls below its own text-free variant. OpenCLIP ViT-B/16 with SAM 2.1.
| Method | Training set | Annot. images | VOC | City | ADE | C-59 | Food | CUB | Avg. |
|---|---|---|---|---|---|---|---|---|---|
| SAN † | COCO-Stuff | 118k | – | – | 32.1 | 57.7 | 24.5 | 19.3 | – |
| CAT-Seg | COCO-Stuff | 118k | 82.5 | 47.0* | 37.9 | 63.3 | 33.3 | 22.9 | 47.8 |
| LPOSS+ | — | — | 62.4 | 37.9 | 22.3 | 38.6 | 26.1 | 12.0 | 33.2 |
| CorrCLIP | — | — | 76.4 | 49.9 | 28.8 | 47.9 | – | – | – |
| DINOv3.txt + SAM (zero-shot) | — | — | 31.3 | 39.3 | 27.7 | 36.3 | 27.2 | 5.8 | 27.9 |
| + RnS, 1 image / class | domain | 66 | 75.0 | 61.0 | 37.4 | 52.3 | 43.6 | 33.8 | 50.5 |
| + RnS, 20 images / class | domain | 964 | 81.3 | 64.4 | 48.2 | 62.3 | 52.8 | 64.3 | 62.2 |
| Fully supervised (best per dataset) | domain | 20k | 90.4 | 87.0 | 63.0 | 70.3 | 45.1 | 84.6 | 73.4 |
mIoU (%). Bold: best per column, purple: second best. RnS uses DINOv3.txt ViT-L/16 with SAM 2.1 regions. domain: support sampled from each dataset's own training split; the annotated-image count is for ADE20K, the largest. † numbers from CAT-Seg, * self-evaluated. Fully supervised: best published model per dataset.
Pick a test image and see the zero-shot prediction. Then click support images in and out of the visual support set and watch the segmentation and its mIoU change instantly. Every one of the 28 support combinations per image was precomputed with the real model (DINOv3.txt ViT-L/16 + SAM 2.1).
@InProceedings{Aravanis_2026_CVPR,
author = {Aravanis, Tilemachos and Stojni\'c, Vladan and Psomas, Bill and Komodakis, Nikos and Tolias, Giorgos},
title = {Retrieve and Segment: Are a Few Examples Enough to Bridge the Supervision Gap in Open-Vocabulary Segmentation?},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2026},
pages = {27621-27632}
}