The benchmark outgrew the paper. Every frozen encoder is read three ways on ImageNet-1k under one protocol: k-NN (nothing trained), LP (a linear layer on the encoder's own global token) and EP (the same linear layer plus EP32 over the patch tokens). The gap between them separates how good the features are from how much learned aggregation matters.
| # | Family | Model | Pre-training data | Res. | k-NN | LP | EP | EP − LP | EP head |
|---|---|---|---|---|---|---|---|---|---|
| Loading results.csv… | |||||||||
arXiv[CLS] token alongside the patch tokens, because the encoder's pre-training objective is defined on it (self-distillation family). ‡ no [CLS] token exists, so LP and k-NN use global average pooling. § the encoder's pooled output comes from its own learned attention pooling (SigLIP, SigLIP2, PE-Core), so LP and k-NN use GAP to measure the features rather than the pretrained pooler. Rows at a resolution other than 224 are not exactly like-for-like; RADIO's LP reads three concatenated [CLS] tokens, 3× wider than what EP pools. Full notes in the repository README.Each point is one frozen encoder. Points above the dotted line gain from EP over linear probing; the further from the line, the more of the encoder's knowledge lives in the patch tokens rather than its global token. Hover a point for its numbers.
EP − LP in top-1 points, sorted. Masked-image-modelling and generative encoders gain the most; encoders trained to produce a strong [CLS] token (DINO, iBOT, CLIP ViT-B) gain little, and RADIO's LP arm is not a like-for-like comparison.
The same table and plots as a Gradio app on Hugging Face, with an About · submit · cite tab. It takes a few seconds to boot. Open it in a new tab if it does not load here.
The table is meant to grow. Run LP and EP with the fixed protocol, add one line to results.csv and open a pull request with your logs. Merged rows appear here and on the Hugging Face Space automatically.
--blr 0.1, effective batch size 4096; report the best epoch.--cls_features cls, or pos (GAP) with a ‡/§ marker if the encoder has no [CLS] or pools with its own attention head. Decide by inspecting the architecture, never by keeping the higher number.--ep_queries 32 to stay comparable, --cls_features ep, or ep_all when the pre-training objective is defined on [CLS].python tools/gen_leaderboard.py to regenerate the README tables, and open the PR with links to the logs.family,method,arch,pretrain,eval,image_size,lp,lp_gap,ep
Hybrid,MyModel,ViT-L/14,LVD-142M,IN-1K,224,85.0,no,86.2Fine-tuning is becoming impractical at scale, so frozen encoders are judged by probes. But a linear probe on a single global token understates encoders whose pre-training optimizes local patch representations. Attention can read those patches, if it is done without the usual baggage.
Joint-embedding and vision-language models push information into a [CLS] or pooled token. Masked image modelling, autoregressive and diffusion models do not: what they know is spread over patch tokens, and a [CLS]-only probe never sees it.
AIM, CAE, V-JEPA, CLIP, SigLIP, CoCa, CaiT, SimPool, AbMILP, DELF and more all aggregate patches with attention. We write them as instances of one framework, benchmark them under one protocol, and find that most spend parameters on projections that do not buy accuracy.
EP learns M query vectors directly in the encoder's feature space. The query and key projections disappear by mathematical equivalence, a single value projection and a linear classifier remain. Same accuracy as multi-head cross-attention, far fewer parameters, and queries that specialize in object parts.
Click through the two explorers. frozen trained
As fine-tuning becomes impractical at scale, probing is emerging as the preferred evaluation protocol. However, standard linear probing can understate the capability of models whose pre-training optimizes local representations rather than an explicit global representation. This motivates attentive probing, an alternative that uses attention to selectively aggregate patch-level features. Despite growing adoption, attentive probing is still underexplored: existing approaches are often over-parameterized and computationally inefficient. In this work, we revisit attentive probing through the lens of the accuracy vs. parameter-efficiency trade-off. We present the first comprehensive study of existing methods, analyzing their design choices and benchmarking their performance. Building on these insights, we propose efficient probing (EP), a lightweight yet effective multi-query cross-attention mechanism that eliminates redundant projections and reduces the number of trainable parameters. Across multiple benchmarks and pre-training paradigms, EP consistently outperforms linear probing and previous attentive probing methods, and remains effective when combined with parameter-efficient fine-tuning. Beyond evaluation, our analysis uncovers emerging properties of EP, including complementary attention maps, which open new directions for leveraging probing beyond protocol design.
Start from vanilla multi-head cross-attention over the frozen patch tokens X ∈ ℝDi×N, then remove what the maths says is redundant.
Take the N patch tokens X of a pre-trained ViT-style backbone. Nothing in the backbone is ever updated.
Each learnable query uj produces a softmax attention map aj over the patches with a single matrix–vector product.
A value projection WV splits into M slices; query j pools its slice, yj = Vjaj. Keeping WV matters (75.1 → 72.1 without it).
Concatenate the yj into y ∈ ℝDo and train a linear classifier. M and Do are the two knobs that trade accuracy for parameters.
Existing methods as aligned steps of the same framework (Table 1 of the paper). Blue: learnable. σm: softmax, σp: softplus, φ(x) = W2 GeLU(W1x).
| Method | Query source | Key transform | Value transform | Attention | Pooling |
|---|---|---|---|---|---|
| MHCA | qj ∈ ℝda | Kj = WKjX | Vj = WVjX | aj = σm(Kj⊤qj) | yj = Vjaj |
| AbMILP | q ∈ ℝDi | K = X | V = X | a = σm(K⊤q) | y = Va |
| AIM | qj ∈ ℝda | Kj = WKj BN(X) | Vj = WVj BN(X) | aj = σm(Kj⊤qj) | yj = Vjaj |
| DELF | q ∈ ℝDi | K = ReLU(WX) | V = WX | a = σp(K⊤q) | y = Va |
| SimPool | q = WQu, u = ⅟N X⊤1 | K = WK LN(X) | V = LN(X) | a = σm(K⊤q) | y = Va |
| V-JEPA | qj = WQju, u ∈ ℝDi | Kj = WKj LN(X) | Vj = WVj LN(X) | aj = σm(Kj⊤qj) | y = φ(WPVjaj) |
| EP (ours) | qj ∈ ℝDi | K = X | Vj = WVjX | aj = σm(K⊤qj) | yj = Vjaj |
CLIP, CaiT, SigLIP, CAE and CoCa are further MHCA variants that keep WQ and add output projections, MLP blocks or self-attention; they are described in Appendix B of the paper and included in every benchmark below.
Trainable parameters on top of a frozen backbone, including the C-way classifier. Biases and normalization layers are omitted, so counts are approximate.
Attention predictors: MHCA with a learnable input vector costs Da(2Di+1), MHCA with a learnable query (AIM) Da(Di+1), EP DiM, all with Da = Di. Every attentive method also trains WV (DoDi) and the classifier (DoC); linear probing trains only the classifier on the [CLS] token. On MAE ViT-B the paper reports 1.36M for EP12 and 1.95M for AIM12 at identical 75.1% accuracy.
ImageNet-1k top-1 accuracy, 90 epochs of probing on frozen backbones. The gains are largest exactly where linear probing is weakest: masked image modelling and generative pre-training.
[CLS] gives 67.7%[CLS] attention over seven backbonesEvery attentive probe on the same frozen tokens, drawn from the paper's plot data. EP variants are colored by output width Do; the dashed line is the Pareto frontier of the existing methods and the shaded area the region they dominate. Hover a point for the exact numbers.
| Method | Arch. | Pre-training | k-NN | LP | # par. | EP | # par. | Gain | FT† | # par. |
|---|---|---|---|---|---|---|---|---|---|---|
| MIM · masked image modelling | ||||||||||
| MAE | ViT-S/16 | IN-1K | 26.7 | 47.4 | 0.4M | 64.6 | 0.5M | +17.2 | 80.6 | 22M |
| MAE | ViT-B/16 | IN-1K | 46.1 | 67.7 | 0.8M | 75.6 | 1.4M | +7.9 | 83.6 | 87M |
| MAE | ViT-L/16 | IN-1K | 58.2 | 76.0 | 1.0M | 79.3 | 2.1M | +3.3 | 85.9 | 304M |
| BEiTv2 | ViT-B/16 | IN-1K | 74.8 | 79.0 | 0.8M | 81.7 | 1.4M | +2.7 | 85.0 | 87M |
| SimMIM | ViT-B/16 | IN-1K | 15.1 | 51.5 | 0.8M | 65.1 | 1.4M | +13.6 | 83.8 | 87M |
| CAPI | ViT-L/14 | IN-1K | 76.7 | 81.5 | 1.0M | 83.6 | 2.1M | +2.1 | – | 304M |
| JEA · joint-embedding | ||||||||||
| BYOL | RN-50 | IN-1K | 64.8 | 74.3 | 2.0M | 75.1 | 6.3M | +0.8 | 77.7 | 26M |
| DINO | ViT-B/16 | IN-1K | 76.1 | 77.3 | 0.8M | 77.8 | 1.4M | +0.5 | 82.8 | 87M |
| Hybrid · MIM + JEA | ||||||||||
| iBOT | ViT-B/16 | IN-1K | 77.0 | 78.7 | 0.8M | 79.2 | 1.4M | +0.5 | 84.0 | 87M |
| DINOv2 | ViT-B/14 | LVD-142M | 81.8 | 83.2 | 0.8M | 84.0 | 1.4M | +0.8 | – | 87M |
| DINOv2 | ViT-L/14 | LVD-142M | 83.5 | 85.2 | 1.0M | 85.6 | 2.1M | +0.4 | – | 304M |
| Franca | ViT-L/14 | IN-21K | 82.2 | 83.8 | 1.0M | 84.3 | 2.1M | +0.5 | – | 304M |
| DINOv3 | ViT-B/16 | LVD-1689M | 83.0 | 84.0 | 0.8M | 84.4 | 1.4M | +0.4 | – | 87M |
| DINOv3 | ViT-L/16 | LVD-1689M | 85.3 | 86.6 | 1.0M | 87.1 | 2.1M | +0.5 | – | 304M |
| VLM · vision-language | ||||||||||
| CLIP | ViT-L/14 | WIT | 77.2 | 82.3 | 0.8M | 83.4 | 2.1M | +1.1 | – | 305M |
| SigLIP | ViT-L/16 | WebLI | 83.7 | 84.1‡ | 1.0M | 86.1 | 2.1M | +2.0 | – | 305M |
| SigLIP2 | ViT-L/16 | WebLI | 84.4 | 85.2‡ | 1.0M | 87.0 | 2.1M | +1.8 | – | 305M |
| GEN · generative | ||||||||||
| DiT | DiT-XL/2 | IN-1K | 8.3 | 32.7‡ | 1.2M | 57.0 | 2.5M | +24.3 | – | 676M |
| AIMv2 | ViT-L/14 | DFN-2B + COYO + HQITP | 80.8 | 84.8‡ | 1.0M | 85.9 | 2.1M | +1.1 | – | 304M |
Table 2 of the paper, ImageNet-1k top-1 (%). # par.: trainable parameters including the classifier; EP is EP32. † Fine-tuning numbers from the official papers, under a different setup (e.g. augmentations); ‡ linear probe on global average pooling, the encoder has no usable [CLS]. EP changes the ranking of methods relative to LP and k-NN: MAE overtakes BYOL and CAPI overtakes CLIP. The standing benchmark above re-runs every row under one fixed protocol and keeps growing.
Drawn from the paper's plot data (Figure 4). Crosses are fine-tuning methods, dots are probes; hover any point for its configuration and exact numbers.
[CLS], AbMILP, DELF, AIM and EP at several Do.[CLS] linear probing (67.66%) with over 3× fewer parametersLoRA adapts the representation; EP selectively aggregates it. The hybrid configurations form a new dominant frontier in the accuracy–parameter plane, so EP is complementary to PEFT rather than redundant with it.
MAE ViT-B, ImageNet-1k top-1 from the patch tokens of intermediate blocks. LP falls from 67.7 at block 12 to 45.8 at block 6; EP stays within 6 points of its best from block 9 on and gains +23.8 over LP at block 6.
| ImageNet-1k subset | LP | EP | FT | LP→FT gap closed |
|---|---|---|---|---|
| 5% of train | 49.6 | 60.9 | 64.7 | 74.8% |
| 10% of train | 55.9 | 65.2 | 68.9 | 71.5% |
| 100% of train | 67.7 | 75.6 | 83.6 | 49.7% |
| Convergence | EP @ 10 ep. | LP @ 90 ep. | EP @ 90 ep. |
|---|---|---|---|
| MAE ViT-B/16 | 60.0 | 67.7 | 75.6 |
| BEiTv2 ViT-B/16 | 80.0 | 79.0 | 81.7 |
| CAPI ViT-L/14 | 82.3 | 81.5 | 83.6 |
| iBOT ViT-B/16 | 78.6 | 78.7 | 79.2 |
Low-shot probing of MAE ViT-B (top): EP closes most of the gap from linear probing to full fine-tuning when labels are scarce. Convergence (bottom): after 10 epochs EP already matches 90-epoch linear probing on BEiTv2, CAPI and iBOT.
ImageNet-1k MaxBoxAccV2 (WSOL protocol). The averaged EP attention maps, taken as they are after ImageNet probing, localize objects better than the last-block [CLS]→patch attention of the same backbones: +9.8 on average, +14.8 for SimMIM.
| Backbone | CUB200 R@1 | +EP | Cars196 R@1 | +EP |
|---|---|---|---|---|
| MAE ViT-B | 16.3 | 57.1 | 12.7 | 34.2 |
| BEiTv2 ViT-B | 57.0 | 68.2 | 48.3 | 44.2 |
| iBOT ViT-B | 51.8 | 62.1 | 31.5 | 40.1 |
| CLIP ViT-L | 75.0 | 80.3 | 79.6 | 79.8 |
| SigLIP ViT-L | 60.8 | 76.7 | 85.8 | 90.4 |
Recall@1 with the frozen backbone's own global feature vs the EP-pooled feature of a probe trained once on ImageNet-1k, never on the target dataset. Every image is a query; both datasets are fully unseen.
EP's queries are not interchangeable heads. Each one settles on a different part of the object, consistently across images and across backbones, and the sharper and more object-focused a query's attention, the more the classifier depends on it.
Hover or tap a query to isolate its attention. Tails, beaks, feet and backgrounds are picked up by the same query in every image.
Recomputed from the published EP heads of the leaderboard (Q = 32) on the same eight ImageNet validation images. The 32 query maps are clustered into eight groups by what they attend to, once for all images, so a colour means the same group of queries in every panel. Hover a group to isolate it. Backbones from four pre-training families, ViT-S to ViT-L.
One ImageNet validation image, MAE ViT-B, each query's attention drawn separately. With two queries one takes the head and one the body; with sixteen the head alone is split into beak, eye and crown. Pick Q.
Attention of each probing method on one ImageNet validation image, MAE ViT-B; multi-head methods show the mean over heads. The [CLS] column is the backbone's own last-block attention.
One minus the average pairwise cosine similarity between attention distributions, matched numbers of predictors. The internal last-block [CLS]→patch heads of every backbone are far more redundant than the external EP queries. On MAE ViT-B, EP (0.65) also beats V-JEPA heads (0.55) and AIM heads (0.59); the best internal block scores 0.37.
Replace one predictor's attention with a uniform distribution and measure the accuracy drop. Across AIM, V-JEPA and EP, the predictors that hurt most when removed are the ones whose attention mass sits inside the ground-truth box and has the lowest entropy. Shrinking Do lowers accuracy not only through capacity but through blurrier attention.
The practical consequences are in the results above: EP's maps localize objects without any training for it (+9.8 MaxBoxAccV2), and its pooled features retrieve fine-grained classes they were never trained on (+40.8 R@1 on CUB200 for MAE). An auxiliary loss that pushes the maps apart speeds up early training but ends at the same accuracy: the complementarity emerges on its own.
A drop-in attentive pooling for any ViT-style encoder: a small bank of queries, one key-free attention, identity values, one output descriptor plus interpretable attention maps.
from poolings.ep import EfficientProbing
# in your model.__init__ (EP_32):
self.ep = EfficientProbing(dim=embed_dim, num_queries=32)
# in your model.forward, tokens from a FROZEN backbone,
# shape (B, 1+N, D) with [CLS] or (B, N, D) without:
patch_tokens = tokens[:, 1:, :] # or `tokens` if no [CLS]
pooled = self.ep(patch_tokens) # (B, D)
logits = self.head(pooled) # your linear classifierFreeze the backbone and train only EfficientProbing and the head. num_queries trades speed for accuracy (8, 16, 32…); the output stays (B, D). Pass all tokens instead to include [CLS] among the values. The attention (B, Q, N) is available for visualization.
git clone https://github.com/billpsomas/efficient-probing
pip install -r requirements.txt # timm pinned at 0.9.16
torchrun --nproc_per_node=4 main_linprobe.py \
--amp bfloat16 --epochs 90 --optimizer lars --batch_size 1024 \
--model vit_base_patch16 --finetune vit_base_patch16_224.mae \
--dataset_name imagenet1k --nb_classes 1000 --data_path /path/to/imagenet \
--cls_features ep --ep_queries 32 \
--output_dir ./outputs/linprobe_mae_vitb_ep_imagenet1k--cls_features picks what the probe reads (cls, pos, ep, ep_all); --knn_eval swaps training for the k-NN protocol; --finetuning does full fine-tuning. The exact command behind every leaderboard row is generated in the README's Commands block, and every trained head is on 🤗 efficient-probing-heads.
@inproceedings{psomas2026attention,
title = {Attention, Please! Revisiting Attentive Probing Through the Lens of Efficiency},
author = {Bill Psomas and Dionysis Christopoulos and Eirini Baltzi and Ioannis Kakogeorgiou and Tilemachos Aravanis and Nikos Komodakis and Konstantinos Karantzalos and Yannis Avrithis and Giorgos Tolias},
booktitle = {The Fourteenth International Conference on Learning Representations (ICLR)},
year = {2026},
url = {https://openreview.net/forum?id=PXo0gtT7Al}
}
Questions: vasileios.psomas@fel.cvut.cz · dxristopoulos@mail.ntua.gr
