Instance-level classes. All visually indistinguishable depictions of the same particular object form one class — a concrete physical entity such as the Temple of Poseidon, or a fictional yet visually distinctive character such as Batman. In practice: if a human observer can confidently say that several images show the same object, they belong to the same instance.
The composed query. A query pairs a visual query — a photograph of the instance — with a text query describing a modification. The goal is to rank a database so that images showing the same instance under the requested modification come first; those are the composed positives, or simply positives. The text never names the instance: the photograph carries what, the text only how. For the Temple of Poseidon, the paper’s running example, the modifications are “in an old archival photo”, “during sunset”, “from an aerial viewpoint”, “at night, with full moon” and “as a scale model”.
Three kinds of hard negative, defined for every composed query and named exactly as the rows below:
How it is scored. All negatives, hard or not, count equally: the metric is mean Average Precision per composed query, averaged first within each instance and then across the 202 instances (macro-mAP). Each instance has its own database, shared by all of its composed queries (nv photographs × nt modifications) and by no other instance — which is what makes exhaustive, error-free labelling possible. Every pair has on average 7.1 positives (median 4, up to 127).
All numbers are computed from the released annotation files. Per instance: 1–25 image queries (median 3) and 1–5 modification texts (median 2); per composed query: 1–127 positives (median 5); per instance database: 951–10,045 curated hard negatives (median 3,420).
Every text query is labelled with one of seven categories according to the kind of transformation it asks for. The wording is deliberately brief — 4.4 words on average — and never mentions the instance.
| category | what changes | examples | pairs | composed queries |
|---|---|---|---|---|
| addition | One or more external elements are introduced into the scene alongside the instance — people, objects, animals. | “with a man proposing”, “next to coffee beans” | 140 | 356 |
| domain | The instance rendered in a different medium — sketch, painting, 3D render, magazine ad, comic. | “as a painting”, “in a manga panel” | 109 | 429 |
| context | A change of the surrounding scene, lighting or time of day; the setting changes, the object does not. | “at night”, “during sunset”, “outdoors on grass” | 83 | 265 |
| appearance | A full transformation of the instance’s physical form; the object stays the focus but appears in a completely different embodiment. | “as a figurine”, “as a sculpture”, “as a scale model” | 69 | 406 |
| attribute | A partial modification of the instance itself — colour, a minor structural difference, a subtle material change; identity unchanged. | “in purple color”, “with yellow shoelaces” | 55 | 159 |
| projection | The instance placed onto another object or surface, as decoration or branding — clothing, packaging, household items. | “on a t-shirt”, “printed on a pillow” | 40 | 187 |
| viewpoint | A change of camera perspective — aerial, top-down; instance and environment stay the same. | “from an aerial viewpoint”, “from a top-down viewpoint” | 22 | 81 |
Sources. Images come from three places: the LAION collection, Google Image Search restricted to Creative-Commons-licensed results, and photographs taken by the annotation team under written guidelines.
Per instance. Curators first fix the object and choose plausible modifications (“at sunset” for a temple; not “with snow”). They then create seed images — two to five good photographs of the instance, or of the instance already modified — and seed sentences of five kinds: the instance itself (“Temple of Poseidon”), another object of the same category (“Ancient Greek Temple”), a rephrasing of the modification (“a photo of dusk”), the instance under the modification, and a same-category object under the modification. Seeds retrieve their nearest neighbours by image-to-image and text-to-image search, and the kind of seed that found an image decides which kind of hard negative it is a candidate for: instance seeds yield visual hard negatives, modification rephrasings yield textual ones, and the two composed seed kinds yield composed ones.
Filtering and labelling. Low-resolution, watermarked and duplicate images are removed automatically (resolution checks, perceptual hashing). Annotators then inspect every remaining image and mark the positives of each composed query; everything unmarked is a negative. Visual hard negatives belong to all queries of an instance; textual and composed ones only to the query they were derived from. Finally the annotators choose the image queries from among the visual hard negatives. Everything not filtered out and not chosen as a query forms the instance’s database. Positives of one query are negatives for every other query of the instance.
Against model bias. Seed images are discarded and never appear in the dataset, and seed sentences never use the exact wording of a text query, so retrieval-based curation does not favour or disfavour CLIP-based methods (checked in the paper: a weaker CLIP ViT-B/32 scores lower, not higher, than ViT-L/14).
Compact, but hard. The result is 752,092 images, of which 748,417 are curated hard negatives — a median of 3,420 per instance (from 951 to 10,045). Replacing them with random LAION images, the paper shows that more than 40 million random distractors are needed to make retrieval as difficult as this curated set does.
People and privacy. Annotators are salaried members of the institution, not crowd-workers, trained for a week on the task, the tools, copyright and privacy, with weekly inter-annotator audits. Visible faces were pixelated automatically (face detection confirmed by a person detector, boxes conservatively enlarged), licence plates and other identifiers removed when encountered; stratified audits estimate residual unredacted PII below 2%. Reports of misuse or PII are reviewed and acted on — see the project page for contact.
Instances are organised in a three-level hierarchy of visual categories (hover a sub-category for its leaves). Roughly half are nameable — objects with a canonical, widely recognised name such as the Eiffel Tower — and half are everyday objects best described compositionally, such as a white-and-pink dolphin plushie. The two animals are the team’s dogs. Pick any instance from the menu above to open it.
The top band is the query: the image queries on the left — click one to select it, or its 🔍 to see it at full resolution (every tile has one) — and the modification text on the right, chosen from the menu or by clicking a tile in the last row. Below the line: the positives, then three rows of hard negatives showing what goes wrong when a system looks only at the photo, only at the text, or at both, and finally the same instance under its other modifications.
The three negative rows are computed rankings over the instance’s database, so that the browser shows the hardest cases rather than a random sample: visual — the hard-negative images most similar to the selected photo under SigLIP2 ViT-SO400M-16-384 (webli); textual — the wrong images most similar to the modification text under the same model’s text–image similarity; composed — the wrong images ranked highest by a training-free composed-retrieval method (BASIC (CLIP ViT-L/14, full preset)) given the selected photo and the text together. Textual and composed rows rank the hard negatives together with the instance’s images under other modifications; the visual row ranks the hard negatives only. “n above the weakest positive” counts how many candidates outscore the lowest-scoring positive — the higher, the harder the pair for that kind of system. Similarities are cosines; composed scores are BASIC’s own scale.