AnyUp: Universal Feature Upsampling

The evolution of vision transformers and self-supervised models like DINO, CLIP, and MAE has led to powerful feature extractors that encode rich semantic information. However, their output resolution is limited by token count, hindering pixel-level tasks. Traditional upsampling methods such as bilinear interpolation are fast but often produce blurry results, especially in complex scenes. Recent learned upsamplers like FeatUp, LoftUp, and JAFAR leverage attention mechanisms to improve quality but are encoder-dependent, requiring retraining for each new feature source, which is computationally expensive and limits scalability. These limitations motivate the development of a universal, inference-time capable upsampling method that can handle diverse features without retraining.

The core challenge is creating a feature upsampling method that is both highly generalizable and efficient, capable of working with any feature extractor and resolution at inference time. Existing methods require retraining for each encoder, which is impractical given the proliferation of large vision models. Furthermore, maintaining feature semantics and local structure during upsampling remains difficult, especially when processing features from different sources with varying dimensions. Addressing these issues is crucial for enabling high-resolution dense predictions across multiple applications, from semantic segmentation to 3D reconstruction, in a cost-effective manner.

AnyUp introduces several key innovations: 1) a feature-agnostic convolution layer that processes features regardless of their source or dimension, 2) a local window attention mechanism that focuses on local structures, reducing computational load while capturing relevant details, 3) a crop-based training strategy that simplifies supervision and enhances local detail recovery, and 4) a unified architecture trained once and applicable across multiple feature types and resolutions. These innovations collectively enable a universal, efficient, and high-performance feature upsampling framework, addressing the limitations of prior encoder-dependent methods.

• �� Input features are processed through a feature-agnostic convolution layer, which convolves each channel with learned kernels and averages contributions, capturing local structure invariant to feature dimension;
• �� During training, local image crops are sampled, downsampled, and used to supervise the upsampling process, ensuring local detail preservation;
• �� Attention computation is restricted to local windows around each pixel, reducing complexity and focusing on relevant spatial regions;
• �� Consistency regularization enforces feature space preservation, aiding transferability and robustness;
• �� The entire model is trained end-to-end on ImageNet, optimizing feature reconstruction loss, regularization, and stability across tasks.

Experiments utilize ImageNet pretraining, comparing against FeatUp, LoftUp, JAFAR, and bilinear interpolation across semantic segmentation, depth, and normal estimation tasks. Metrics include mIoU, RMSE, and angular errors, evaluated on datasets like COCO, ADE20k, and NYUv2. Ablation studies analyze the impact of window size, cropping strategy, and feature regularization. The model's generalization is tested on features from different sources, including DINO and CLIP, with cross-model evaluation confirming robustness. Efficiency metrics demonstrate over 50% runtime and memory savings compared to prior attention-based methods.

AnyUp achieves state-of-the-art performance, with 3-5% higher mIoU and 10% lower RMSE across tasks. It maintains feature space integrity better than LoftUp, enabling effective transfer with linear probes. The ablation shows that window attention and crop supervision are critical for performance. The model generalizes well to unseen features, including those from different vision transformers, with minimal performance drop. Efficiency gains make it suitable for real-time applications, and qualitative results show sharper, more detailed feature maps than prior methods.

This method can be directly applied to high-resolution semantic segmentation, dense depth estimation, and 3D reconstruction tasks. It is particularly useful in multi-modal systems where features come from diverse sources, reducing the need for multiple retrained models. Its efficiency and generalization capabilities make it ideal for deployment in autonomous vehicles, robotics, and large-scale visual analytics, enabling high-fidelity dense predictions without extensive retraining.

Despite its advantages, AnyUp may struggle with extremely noisy or low-contrast images, where feature structures are ambiguous. Computational costs, while reduced, remain significant for very high resolutions or real-time scenarios. The current architecture primarily focuses on local structure, which might limit global context understanding. Future work should explore multi-scale fusion, robustness in dynamic environments, and further optimization for edge devices.