MEPA: Multi-Scale Representation Alignment for Visual Autoregressive Modeling with Mixture of Experts

The evolution of image generation has seen significant progress from GANs (e.g., StyleGAN series) to diffusion models (e.g., ADM, LDM), and recently to autoregressive (AR) models like VQGAN and DiT. VAR models, combining multi-scale residual prediction with causal autoregression, offer a promising balance of quality and efficiency. Despite these advances, multi-scale representation learning remains challenging due to conflicting optimization objectives across scales and the propagation of semantic errors. As models scale up, training becomes more resource-intensive, and the fidelity of generated images can degrade, especially in complex scenes. Addressing these issues is crucial for practical deployment of high-resolution, high-fidelity image synthesis.

The core challenge lies in the conflicting objectives of learning global semantics at lower scales versus fine details at higher scales within a shared architecture. This leads to optimization conflicts and hampers the model's capacity to learn effective multi-scale representations. Additionally, the causal autoregressive process causes early-stage semantic inaccuracies to propagate, adversely affecting the final output. These issues limit the scalability, efficiency, and quality of VAR-based image generation, especially when aiming for high-resolution outputs. Overcoming these bottlenecks requires innovative architectural designs that can decouple scale-specific learning and improve semantic stability.

The paper introduces three key innovations: 1) a scale-aware Token routing MoE (SMoE/STMoE) that dynamically allocates experts based on scale or token features, reducing optimization conflicts; 2) residual feature aggregation combined with external self-supervised features to enhance early-scale semantic modeling; 3) a comprehensive framework integrating expert load balancing, semantic guidance, and residual alignment, enabling efficient multi-scale learning with improved generation fidelity. These innovations collectively address the limitations of shared architectures and semantic error propagation, setting a new standard for scalable, high-quality VAR models.

• �� Extract global features using pretrained visual encoders (e.g., DINO).• Decompose images into multi-scale residual features via quantization and subtraction.• Implement scale-aware expert routing (SMoE) with scale embeddings to assign tokens to specialized experts.• Enhance to token-based routing (STMoE) by incorporating token features for finer expert assignment.• Aggregate residual features progressively, upsampling and combining with pretrained features for semantic enrichment.• Introduce semantic guidance loss to align VAR residual features with pretrained representations, reducing semantic errors.• Use load balancing loss to ensure experts are evenly activated, improving training efficiency.• Optimize jointly with autoregressive loss, balancing generation quality and training speed.

The models are trained on ImageNet 256×256 with 12 and 16 layers, using AdamW optimizer, batch size 96, and learning rate decay from 1e-4 to 1e-5. Evaluation metrics include FID, IS, precision, and recall. Ablation studies compare variants with and without STMoE, semantic guidance, and different expert routing schemes. Experiments demonstrate that MEPA achieves faster convergence—training 100 epochs yields better results than baseline models trained for 200 epochs—and maintains high image fidelity. The robustness is validated across multiple training settings, confirming the effectiveness of expert routing and residual alignment strategies.

MEPA achieves an FID of 2.32 after 200 epochs, outperforming the baseline VAR at 4.10, with a 50% reduction in training epochs. Ablation results show that adding STMoE and semantic guidance reduces FID by approximately 0.7, enhances detail, and stabilizes semantic consistency. Load balancing ensures expert utilization, preventing bottlenecks. The model maintains high performance across different training durations, confirming its efficiency and robustness. These results demonstrate that the proposed framework significantly advances the state-of-the-art in autoregressive image generation.

The framework can be applied to high-resolution image synthesis, content creation, and virtual reality applications, where efficiency and quality are critical. Its ability to accelerate training and improve fidelity makes it suitable for industry deployment in digital art, gaming, and AI-assisted design. The integration of semantic guidance also opens avenues for controllable generation and multi-modal extensions, broadening its impact.

Despite improvements, the model still faces challenges in generating highly complex or detailed scenes, where semantic errors may persist. Dependence on pretrained encoders means performance hinges on their quality; poor pretraining could limit results. Computational costs remain significant for ultra-high-resolution outputs, necessitating further optimization. Future work should focus on enhancing robustness, reducing inference costs, and exploring multi-modal data integration.