Fixed-Point Masked Generative Modeling
Introducing Fixed-Point Masked Generative Models (FP-MGMs), which use shared attention layers with a fixed-point solver for adaptive depth, reducing parameters by 38.8% and training time by 11.5%.
Key Findings
Methodology
This work proposes FP-MGMs by replacing parts of the denoiser with a shared fixed-point solver over attention layers, combined with cross-step consistency loss and three-state reuse (3SR). The training-to-inference framework CoFRe enables models like MDLM and MaskGIT to adapt their effective depth dynamically, reducing parameters and training costs while maintaining or improving quality. The training employs stochastic Jacobian-free backpropagation (SJFB), iteratively approximating fixed points without full backpropagation, and short distillation from pretrained models. Evaluations on OpenWebText and ImageNette demonstrate significant efficiency gains and quality improvements at low sampling budgets.
Key Results
- On OpenWebText, CoFRe reduces model parameters by 38.8%, training time by 11.5%, and VRAM by 16.9%, with perplexity dropping from 830.8 to 101.8 at 96 forward passes, outperforming MDLM.
- In ImageNette, training time is cut by 48.6%, VRAM halved, and FID scores improved across all budgets, confirming multi-modal effectiveness.
- The combination of fixed-point denoisers, cross-step consistency, and 3SR significantly enhances low-budget generation quality, validating the approach’s efficiency and robustness.
Significance
This research addresses the longstanding challenge of high computational and parameter costs in masked generative models, especially under limited sampling budgets. By enabling adaptive depth through shared fixed-point layers, it offers a practical solution for deploying high-quality generative models in resource-constrained environments. The framework’s flexibility and efficiency open new avenues for large-scale multimodal content creation, real-time applications, and edge deployment, marking a significant step forward in scalable AI generation.
Technical Contribution
The core innovation lies in integrating fixed-point solvers into discrete masked denoising architectures, allowing parameter sharing and dynamic depth control. The introduction of cross-step consistency loss and token-aware three-state reuse enhances stability and reusability across denoising steps. The training employs SJFB, avoiding full backpropagation through the solver trajectory, which drastically reduces training costs. The method also enables rapid transfer from pretrained models via short distillation, broadening practical applicability. These advances collectively push the boundaries of parameter efficiency and low-budget performance in masked generative modeling.
Novelty
This work is the first to embed fixed-point solvers within discrete masked generative models, bridging the gap between continuous diffusion techniques and discrete token generation. Unlike prior fixed-depth or looped transformer architectures, FP-MGMs leverage shared layers with iterative equilibrium solutions, providing controllable effective depth without increasing parameters. The combined use of cross-step regularization and token-dependent reuse strategies further distinguishes this approach, offering a novel paradigm for efficient, adaptive, and high-quality masked generation.
Limitations
- Despite improvements, the fixed-point solver's convergence may degrade in highly complex or noisy scenarios, especially with extremely limited steps. The approach has been validated primarily on text and image tasks, with less exploration in audio or video domains. Additionally, multiple iterations of the solver, although parameter-efficient, still incur computational overhead, which could be problematic in real-time applications. Future work should focus on optimizing solver algorithms, extending to broader modalities, and further reducing inference latency.
Future Work
Future directions include integrating more advanced and faster fixed-point algorithms, extending the framework to audio and video modalities, and exploring multi-modal joint training. Additionally, developing adaptive iteration schedules and hybrid models combining fixed-point and traditional layers could further improve efficiency and robustness. Investigating the theoretical convergence properties and stability guarantees of the fixed-point solvers in discrete settings also remains an important avenue for research.
AI Executive Summary
Masked generative models (MGMs) have revolutionized sequence synthesis across modalities, enabling parallel decoding and high-quality outputs. However, their reliance on full bidirectional transformers at each step results in high computational costs and limited efficiency, especially under low sampling budgets. Existing efforts to improve efficiency—such as better samplers or fixed-depth architectures—still fix the amount of computation per step, limiting flexibility and scalability.
This paper introduces Fixed-Point Masked Generative Models (FP-MGMs), a novel approach that replaces part of the denoising network with a shared attention layer coupled with a fixed-point solver. This design allows the model to adapt its effective depth dynamically by controlling the number of solver iterations, significantly reducing parameters and training time. To further enhance low-budget performance, the authors propose two key mechanisms: cross-step consistency loss, which aligns hidden representations across denoising steps, and three-state reuse (3SR), which intelligently warm-starts the solver based on token stability—fully reusing visible tokens, partially reusing masked tokens, and updating newly revealed tokens.
The entire framework, called CoFRe, is trained end-to-end and can be applied to pretrained MGMs with minimal fine-tuning. Extensive experiments on language and image datasets demonstrate remarkable gains: on OpenWebText, parameters are reduced by nearly 40%, training time by over 10%, and perplexity at low budgets improves from over 800 to below 102. Similarly, on ImageNette, training time is halved, and FID scores are consistently better across budgets. These results showcase the potential of fixed-point approaches to make masked generative models more practical, scalable, and efficient.
Despite these advances, challenges remain in ensuring solver convergence in highly complex scenarios and extending the framework to other modalities like audio and video. Nonetheless, this work paves the way for more resource-efficient, high-quality generative AI, with broad implications for industry and research alike.
Deep Analysis
Background
近年来，掩码生成模型（MGMs）在文本、图像等多模态任务中取得显著突破，代表性方法包括MDLM和MaskGIT。它们通过逐步掩码和解码实现高质量内容生成，但训练成本高、参数庞大，低预算采样效果不佳。为解决这一问题，学界不断探索参数共享、深度控制和高效采样策略，但仍存在模型固定深度、参数冗余等瓶颈。深度平衡和参数效率成为研究热点，Fixed-Point Diffusion Models（FPDMs）引入固定点求解器，改善连续任务中的效率，但在离散掩码场景中应用有限。
Core Problem
核心问题在于，如何在保证生成质量的同时，显著降低掩码生成模型的参数规模和训练成本，尤其在有限采样预算（少次前向）条件下依然保持优异性能。传统模型参数固定，深度不可调节，导致在低预算下性能下降。此外，输入状态在每个步骤变化剧烈，难以实现参数重用和效率提升。这限制了MGMs在实际工业中的应用，亟需一种既能控制模型深度，又能保证或提升生成效果的方法。
Innovation
本研究提出FP-MGMs，通过引入共享注意力层上的固定点求解器，实现模型深度的动态调节。结合交叉步一致性损失和三态重用（3SR）机制，有效应对输入变化带来的不稳定性，提升低预算采样质量。创新点包括：1）参数共享的固定点层，减少参数和训练时间；2）多状态重用策略，优化不同位置的重用效果；3）交叉步一致性损失，增强模型在噪声扰动下的稳定性。这些创新共同推动掩码模型在参数效率和生成质量上的突破。
Methodology
- �� 构建由预处理、输入调节、固定点求解和后处理四部分组成的解噪器架构。• 采用多次无梯度迭代逼近固定点，减少参数和计算量。• 引入交叉步一致性损失（LCONS），对齐噪声较大与较小状态的隐藏表示，提升低预算生成质量。• 设计三态重用（3SR）策略，根据不同位置的变化情况，调整固定点求解器的初始化和重用策略。• 训练过程中利用SJFB，避免反向传播完整求解轨迹，提升训练效率。• 通过短期蒸馏，将预训练模型转化为FP架构，减少从零训练的成本。• 在文本和图像数据集上验证模型效果，比较参数、训练时间和生成质量指标。
Experiments
在OpenWebText和ImageNette上，分别评估模型的参数量、训练时间、VRAM消耗和生成困惑度/FID指标。采用不同采样预算（如96、192、384）进行对比，验证FP-MGMs在低预算条件下的性能提升。通过消融实验分析交叉步一致性和三态重用的贡献，验证模型的参数节省和质量改善效果。模型还通过迁移学习，将预训练MDLM转化为FP-MDLM，验证快速适应能力。所有实验均在标准硬件环境下进行，确保结果的可复现性。
Results
模型在OpenWebText上，参数减少38.8%，训练时间缩短11.5%，困惑度从830.8降至101.8，表现优于MDLM。图像任务中，训练时间减半，FID指标提升明显。引入交叉步一致性和三态重用后，低预算生成质量显著改善，验证了方法的有效性。迁移学习实验显示，短期蒸馏即可将预训练模型转化为参数更少、效率更高的FP模型，保持或提升性能。这些结果证明了FP-MGMs在多模态任务中的优越性和实用性。
Applications
该技术适用于需要低成本高效生成的场景，如大规模文本生成、实时图像合成、边缘设备上的多模态交互等。模型的参数和训练成本降低，使得中小企业也能部署高质量生成模型。未来可结合多模态联合训练，推动智能内容创作、虚拟助手等行业的发展。
Limitations & Outlook
模型在极端低预算（少于50次前向）时仍可能出现性能下降，主要因固定点求解器在复杂场景中的收敛性不足。当前主要验证于文本和图像，尚未充分扩展到音频、视频等多模态场景。训练过程中多次无梯度迭代增加了计算复杂度，硬件资源仍有限。未来需优化求解算法和多模态适应性，提升泛化能力。
Plain Language Accessible to non-experts
想象你在做一道复杂的菜肴，通常需要按照固定的步骤逐步添加材料和调料，每一步都要花费时间和精力。现在，如果你有一个聪明的助手，他可以在每次添加材料后，快速判断是否还需要调整，甚至可以在不重新做全部步骤的情况下，直接给出最终的味道。这个助手就像论文中的固定点求解器，它能在不重复所有步骤的情况下，找到最佳的解决方案。这样一来，你做菜的效率大大提高，花费也更少，但菜的味道依然很好。这篇论文就是用类似的思路，让生成模型在保持高质量的同时，减少参数和计算量，变得更快、更省钱。这种方法就像你用一个聪明的机器人帮你做饭，它可以不断调整，直到味道刚刚好，又快又省力。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的游戏，每次你都要花很长时间去完成一个任务，但如果你有个聪明的朋友，他可以在你每次尝试后，告诉你下一步该怎么做，而且还能帮你省掉很多重复的工作。这篇论文就像那个聪明的朋友，它用一种特别的方法，让电脑在生成内容时不用每次都重新计算所有细节，而是用之前的结果作为起点，快速找到答案。这样一来，电脑可以用更少的“努力”做出同样甚至更棒的内容，比如写文章或画画。它还能根据不同的情况，决定用多少“努力”来完成任务，既省钱又快，还能保证内容的质量。是不是很酷？未来，这种方法可以让我们的电脑变得更聪明、更省电，也能帮我们做出更棒的作品！
Abstract
Masked Generative Models (MGMs) enable parallel decoding and achieve strong performance across modalities, but require full-sequence bidirectional transformers at every step, making training costly and degrading quality under low sampling budgets. Existing work improves efficiency via better samplers or cheaper fixed-depth denoisers, but they still allocate a fixed amount of denoiser computation to each refinement step. We introduce Fixed-Point Masked Generative Models (FP-MGMs), which replace part of the denoiser with a fixed-point solver over shared attention layers to enable adaptive depth with fewer parameters. To make it more effective for masked generation, we first introduce a cross-step consistency loss, which aligns hidden representations at neighboring denoising steps and, second, three-state reuse (3SR) which warm-starts the solver using the previous solution by treating differently unchanged, still-masked, and newly revealed tokens respectively. Together, these components define our complete training-to-inference framework for fixed-point masked generation, \emph{CoFRe}. We also show that pre-trained MGMs can be converted into FP-MGMs with short fine-tuning, avoiding full retraining. Across modalities, CoFRe improves the quality and cost trade-off. On OpenWebText, CoFRe reduces parameters by 38.8\%, training time by 11.5\%, and VRAM by 16.9\%, while improving generative perplexity from 830.8 to 101.8 at a budget of $96$ transformer-block forward passes, compared to MDLM. In ImageNette, CoFRe reduces training time by 48.6\% and VRAM by 50.7\%, while improving FID in all sample budgets tested. Overall, CoFRe offers a practical framework for cheaper training and stronger low-budget masked generation.