DecQ: Detail-Condensing Queries for Enhanced Reconstruction and Generation in Representation Autoencoders

Visual generation has witnessed remarkable progress with the advent of diffusion models, which generate high-quality images by iteratively denoising latent representations. Traditionally, these models rely on Variational Autoencoders (VAEs) to learn compressed latent spaces; however, VAEs often produce latent representations lacking strong semantic structure due to their reconstruction-centric objectives. To address this, Representation Autoencoders (RAEs) leverage frozen pretrained Vision Foundation Models (VFMs) such as DINOv2 and SigLIP2 as tokenizers, capitalizing on their rich semantic features learned via self-supervised or multimodal training. This approach accelerates diffusion model convergence and enhances generation quality by operating in a semantically meaningful high-dimensional latent space. Despite these advantages, VFMs are typically trained with objectives emphasizing semantic invariance rather than pixel-level fidelity, resulting in latent representations that inadequately preserve low-level visual details like color and texture. Consequently, RAEs built on frozen VFMs exhibit limited spatial reconstruction capacity, hindering fine-grained image generation and editing. Prior attempts to fine-tune VFMs or augment latent spaces with reconstruction-oriented features often disrupt the semantic latent space, leading to degraded generative performance and slower convergence. Thus, balancing semantic consistency with reconstruction fidelity remains a critical challenge in RAE design.

The core challenge addressed in this work is the inherent trade-off in RAEs between preserving the semantic latent space of frozen VFMs and achieving high-fidelity image reconstruction. Frozen VFMs provide stable, semantically rich representations that facilitate fast and high-quality generation but lack sensitivity to low-level details, causing reconstruction artifacts such as texture loss and color shifts. Fine-tuning VFMs to enhance reconstruction introduces conflicting objectives, perturbing the pretrained semantic space and impairing generative fidelity. Alternative methods that concatenate reconstruction features with semantic tokens risk misaligning the latent space, hindering downstream diffusion training. This trade-off limits the practical utility of RAEs in applications requiring both detailed reconstruction and high-quality generation. Therefore, a mechanism is needed to supplement frozen VFM representations with complementary low-level information without modifying the encoder parameters or disrupting semantic consistency, enabling simultaneous improvements in reconstruction and generation.

The principal innovation of DecQ lies in its introduction of detail-condensing queries—learnable tokens that attend to intermediate features of a frozen VFM via cross-attention condenser modules. This design achieves several key advances: (1) It preserves the frozen VFM's semantic latent space by ensuring unidirectional information flow from patch tokens to queries, preventing any modification of pretrained representations. (2) It aggregates multi-level features from shallow to deep VFM layers, capturing both low-level details and high-level semantics, thus addressing the reconstruction-generation trade-off. (3) The joint denoising of query and patch tokens during diffusion training integrates fine-grained detail recovery directly into the generative process, enhancing convergence speed and output quality. Unlike prior approaches that rely on fine-tuning or feature concatenation, DecQ maintains semantic consistency while enriching detail representation, enabling simultaneous improvements in reconstruction fidelity and generative performance with minimal computational overhead.

• �� Utilize a frozen Vision Foundation Model (e.g., DINOv2-B) as the encoder, producing semantic patch tokens representing image regions.

• �� Attach condenser modules at selected intermediate VFM layers (default: layers 0, 3, 6, 9). Each condenser comprises a cross-attention block and a feed-forward network (FFN).

• �� Initialize a small set of learnable detail-condensing query tokens (typically 8) that serve as queries in the cross-attention, attending to the intermediate patch tokens (keys and values) to extract complementary low-level features.

• �� Ensure unidirectional information flow from patch tokens to queries to preserve the frozen VFM's latent space integrity.

• �� Project query and patch tokens separately with positional embeddings (2D sinusoidal for patches, learnable for queries), then concatenate and input into a ViT-based decoder for image reconstruction.

• �� During training, add noise to both query and patch tokens and jointly optimize a flow matching diffusion objective to denoise and reconstruct images.

• �� During generation, jointly sample and denoise query and patch tokens in the latent diffusion model, then decode to produce high-fidelity images.

• �� Evaluate on ImageNet 256×256 with metrics including PSNR, SSIM, reconstruction FID (rFID), and generation FID, IS, Precision, Recall.

• �� Conduct ablation studies on query number, condenser layer selection, and training paradigms to optimize the reconstruction-generation trade-off.

Experiments are conducted on the ImageNet dataset at 256×256 resolution, using DINOv2-B as the default frozen VFM encoder and a ViT-XL decoder with approximately 500 million parameters. Baselines include the original RAE with frozen VFM, fine-tuned VFM variants with and without distillation losses, and feature concatenation methods. Reconstruction quality is assessed via PSNR, SSIM, and rFID, while generative performance is measured using FID, Inception Score (IS), Precision, and Recall. The diffusion model employs 50 sampling steps, with query-token loss weight set to 1. Ablation studies explore the impact of varying the number of detail-condensing queries, condenser module placement across VFM layers, and the effect of discarding query tokens at inference. Additional experiments validate DecQ's generalization on alternative VFMs such as SigLIP2-B. Results demonstrate DecQ's superior reconstruction and generation performance with minimal computational overhead and faster convergence compared to baselines.

DecQ significantly improves reconstruction PSNR from 19.13dB to 22.76dB and reduces reconstruction FID compared to the frozen VFM RAE baseline, indicating enhanced recovery of low-level details. In generative tasks, DecQ achieves an FID of 1.80 at 80 epochs without guidance, outperforming RAE's 2.16 and converging 3.3 times faster. With guidance at 800 epochs, FID further improves to 1.05, setting new state-of-the-art results for high-dimensional VFM latent space generation. Ablation studies confirm that 8 detail-condensing queries and condenser modules at layers 0, 3, 6, and 9 provide the best balance between reconstruction fidelity and generation quality. Notably, even when query tokens are discarded at inference, predicting them during training benefits the generation of patch tokens, highlighting the queries' auxiliary role. DecQ also generalizes effectively across different VFMs such as SigLIP2-B.

DecQ is well-suited for applications demanding high-fidelity image reconstruction and generation, such as medical imaging, satellite image analysis, and artistic content creation. Its ability to preserve semantic consistency while enhancing fine-grained details benefits image editing and virtual reality scenarios requiring precise texture and color reproduction. The framework's modular design facilitates integration with existing pretrained VFMs, reducing training costs and enabling rapid deployment. Furthermore, DecQ's approach can inform multi-task vision systems that require simultaneous semantic understanding and detailed reconstruction, advancing both academic research and practical deployments in computer vision and graphics.

DecQ's reliance on the representational capacity of pretrained VFMs may limit its effectiveness in extremely complex or detail-rich image domains where fixed query capacity constrains detail extraction. The number of detail-condensing queries and condenser layer selection require careful tuning; excessive queries can introduce redundant low-level information that hampers generative modeling. Additionally, while validated on DINOv2 and SigLIP2, the framework's adaptability to other VFM architectures and modalities remains to be fully explored. Computational overhead, though modest, may increase with higher-resolution or larger-scale applications, necessitating further efficiency improvements.