Query-time calibration overview (Figure 1), by Dominik Engel, Yousef Rajeh, Leon Sick, and Ivan Viola. Cropped from the author-uploaded paper to show the figure without its caption. Source: arXiv, CC BY 4.0.At a glance
- Improves segmentation of already-lifted CLIP features at query time, without retraining, source-image re-encoding, or feature re-lifting.
- Combines scene centering and reference-only text whitening with soft clustering for multiclass queries and a CSLS relevance margin for single queries.
- Evaluates seven lifting methods on four datasets, separating multiclass label competition from single-query foreground rejection.
Motivation
A reconstructed 3D scene contains a much narrower range of concepts than CLIP’s pretraining data. Its lifted visual features can concentrate around a shared scene direction, making raw cosine similarity a poor measure of which regions match a text query. Improving the comparison itself offers a way to recover useful semantics without reconstructing the scene or lifting its features again. A single object query also needs a foreground-versus-background decision, rather than an argmax over a complete class list.
Method
The method partially subtracts the normalized mean scene direction from both visual features and text prototypes, then renormalizes them. It fits a regularized whitening transform using only a fixed bank of 45 generic reference terms, such as “object” and “texture,” and applies that transform only to text features. For a fixed scene, the whitening statistics therefore do not depend on the requested class vocabulary.
For multiclass segmentation, calibrated similarities initialize text-guided soft clustering of the scene primitives, followed by an argmax over their refined class assignments. For a single query, reduced cross-domain similarity local scaling (CSLS) corrects the query and reference scores; the query’s margin over the strongest generic reference competitor is thresholded to reject background. Geometry and the stored lifted feature field remain unchanged.
Evaluation
The paper evaluates SFS, NormLift, LUDVIG, Occam, VALA, OpenGaussian, and LangSplat on LERF-OVS, 3D-OVS, ScanNet, and ScanNet++. It reports multiclass and single-query mIoU separately, with single-query thresholds selected on held-out scenes. For SFS, multiclass mIoU increases from 49.09 to 65.67 on LERF-OVS, 70.79 to 89.80 on 3D-OVS, 26.87 to 39.37 on ScanNet’s 19 classes, and 19.92 to 23.74 on ScanNet++’s 100 classes.
Single-query mIoU improves for all evaluated method–dataset combinations; multiclass gains are broad, but VALA on LERF-OVS is essentially unchanged (59.83 versus 59.68). Diagnostics show improved class-relative separation in every evaluated diagnostic scene. Comparisons against adapted TransCLIP, PTC, UMFC, and a Frolic-inspired baseline, together with component ablations, examine the contributions of feature calibration, score calibration, and the reference bank. The authors note that they could not reproduce reasonable LUDVIG results on 3D-OVS.
Limitations
Evaluation is limited to indoor scenes, and shared centering and whitening parameters were selected using held-out SFS indoor results; transfer to outdoor environments and other embedding models remains untested. Single-query evaluation asks for classes known to be present and uses held-out-selected thresholds, so it does not establish reliable rejection of entirely absent targets or arbitrary natural-language queries. Predictions depend on the requested vocabulary and the fixed reference bank. Calibration cannot reconstruct missing surfaces or recover semantic information absent from the lifted field.
