Leveraging Self-Supervised Vision Transformers for Segmentation-based Transfer Function Design
At a glance
- Turns sparse slice annotations into a transfer-function design workflow for volume rendering.
- Uses self-supervised vision-transformer features to find visually similar structures throughout a volume.
- Provides immediate visual feedback after each annotation, without training a volume-specific model.
Motivation
Transfer functions determine which structures are visible in a volume rendering, yet designing them from intensity and gradient values is often tedious and unintuitive. The goal is to let users specify structures directly rather than manually engineer a mapping in feature space.
Method
The workflow exposes a slice viewer in which a user marks a structure of interest. A self-supervised, pre-trained vision transformer supplies high-level features; similar features are selected across the volume to form a segmentation that drives the transfer function. Because the method relies on pre-trained features rather than per-volume training, edits can be reflected immediately.
Evaluation
The paper demonstrates annotation-guided transfer-function design on volume data and evaluates whether feature-based selections isolate target structures from only a small amount of user input. It also contrasts the interactive workflow with previous learning-based approaches that require costly optimization for each volume.
Limitations
The result depends on the discriminative power of the pre-trained features and on the ambiguity of the selected structure. Sparse annotations are powerful, but may need refinement when visually similar structures should receive different optical properties.
