Equilibrium Matching: Generative Modeling with Implicit Energy-Based Models
EqM learns a time-invariant energy gradient landscape, enabling optimization-based sampling with FID 1.90, outperforming diffusion/flow models.
Key Findings
Methodology
EqM constructs a stable, time-invariant energy landscape by learning its gradient, replacing the non-equilibrium dynamics of diffusion and flow models. The training minimizes the difference between the model output and a target gradient derived from a corruption scheme, supporting both explicit and implicit energy functions. During inference, samples are generated via gradient descent methods like Nesterov's accelerated gradient, with support for adaptive step sizes and compute. Experiments on ImageNet 256×256 show a state-of-the-art FID of 1.90, surpassing diffusion (2.12) and flow (2.06) baselines.
Key Results
- On ImageNet 256×256, EqM achieves FID 1.90, outperforming diffusion (2.12) and flow (2.06) models by significant margins.
- The framework supports flexible sampling strategies, including gradient descent and Nesterov acceleration, reducing inference cost by up to 60%.
- It demonstrates robustness in tasks like partial denoising, out-of-distribution detection, and image composition, validating multi-task capabilities.
Significance
This work introduces a unified, theoretically grounded framework that overcomes the limitations of non-equilibrium dynamics in generative models. By focusing on equilibrium energy landscapes, EqM enhances sample quality, inference efficiency, and task flexibility, paving the way for practical deployment in real-world applications such as image synthesis, editing, and anomaly detection.
Technical Contribution
The paper develops a novel equilibrium framework that learns a stable energy gradient, supported by theoretical guarantees of data manifold learning and convergence. It integrates explicit and implicit energy modeling, supports adaptive optimization-based sampling, and bridges the gap between energy-based and flow models, offering new avenues for scalable, high-quality generation.
Novelty
This is the first comprehensive framework to replace time-dependent non-equilibrium dynamics with a single, time-invariant energy landscape for generative modeling. It combines the strengths of energy-based models and flow models, supported by rigorous theoretical analysis and flexible inference strategies, marking a significant departure from prior approaches.
Limitations
- Training requires extensive data and computational resources, with potential instability in high-dimensional settings.
- High-resolution and complex scene generation still pose challenges due to the difficulty in accurately modeling the energy landscape.
- The current approach may need further optimization for real-time applications and large-scale deployment.
Future Work
Future research will focus on extending EqM to multi-modal data, improving energy landscape learning efficiency, and developing more scalable, adaptive sampling algorithms. Exploring unsupervised and self-supervised strategies could further enhance generalization and robustness, broadening the framework’s applicability.
AI Executive Summary
Generative modeling has seen rapid progress through diffusion and flow-based methods, which map simple noise distributions to complex data via dynamic processes. These models, while achieving high sample quality, rely heavily on non-equilibrium, time-dependent dynamics, which impose constraints on sampling flexibility and efficiency. Recognizing these limitations, this work introduces Equilibrium Matching (EqM), a fundamentally different approach rooted in energy landscape theory.
EqM abandons the traditional time-dependent velocity fields, instead learning a time-invariant energy gradient landscape that vanishes on the data manifold. During training, the model minimizes the discrepancy between its predicted gradient and a target gradient derived from a corruption scheme, ensuring the learned landscape supports the data distribution. In inference, samples are generated through gradient descent, supported by advanced optimization techniques like Nesterov acceleration, enabling adaptive step sizes and compute, significantly reducing inference time.
Empirical results on ImageNet 256×256 demonstrate that EqM achieves an FID of 1.90, surpassing existing diffusion and flow models. The framework exhibits excellent scalability across training epochs, model sizes, and patch resolutions. It also supports tasks such as partial denoising, out-of-distribution detection, and image composition, highlighting its versatility. Theoretically, the authors prove that EqM learns the data manifold as a set of local minima, with convergence guarantees for the sampling process.
Overall, EqM offers a new paradigm for generative modeling, unifying energy-based and flow perspectives, and enabling more flexible, efficient, and theoretically sound inference. Future directions include multi-modal extensions, improved sampling algorithms, and broader real-world applications, promising a significant impact on both academia and industry.
Deep Analysis
Background
生成模型的发展经历了从变分自编码器（VAE）到生成对抗网络（GAN），再到扩散模型（Diffusion）和流模型（Flow）的演变。Diffusion模型如Ho等（2020）提出的反向扩散过程，通过逐步去噪实现高质量生成，但训练和采样过程复杂，受限于非平衡动态。Lipman等（2021）提出的流模型通过学习可逆变换实现高效采样，但依赖时间条件和固定步长。能量基础模型（LeCun等，2006）提供了直接学习能量景观的途径，但训练不稳定，样本质量有限。近年来，能量匹配（Balcerak等，2025）尝试结合能量模型与流模型，但性能仍未突破瓶颈。
Core Problem
现有模型普遍依赖非平衡的时间条件动态，限制了采样的灵活性和效率。能量模型虽理论优越，但训练不稳定且难以达到高质量生成。如何在保证生成质量的同时，简化模型结构、提升采样速度，成为亟待解决的问题。此外，模型在多任务适应性和推理效率方面仍有较大提升空间。
Innovation
提出平衡匹配（EqM）框架，核心创新在于：1）学习时间不变的能量梯度场，摒弃非平衡动态；2）支持显式和隐式能量学习，增强模型表达能力；3）采用优化驱动采样策略，提升采样速度和质量；4）理论上证明模型能学习数据流形，保证样本的真实性。这些创新有效解决了传统模型的动态限制和训练不稳定问题，为生成模型提供了新思路。
Methodology
- �� 训练阶段：定义扰动样本xγ，γ在[0,1]间均匀采样；• 目标：匹配中间扰动样本的目标梯度（(ε−x)·c(γ)），支持显式或隐式能量；• 损失函数：最小化f(xγ)与目标梯度的差异；• 采样阶段：利用梯度下降或Nesterov加速，支持自适应步长和计算资源调度；• 训练模型：采用Transformer架构，设置t=0，确保无时间条件依赖。
Experiments
在ImageNet 256×256上，采用不同模型规模和采样策略，比较FID指标，验证EqM优越性。训练80轮，调优超参数如c(γ)、步长和梯度乘数。还进行消融实验，验证不同目标梯度和能量形式的影响。支持多样化采样策略，显著缩短采样时间，提升样本质量。
Results
EqM在ImageNet 256×256任务中实现FID 1.90，优于Diffusion（2.12）和Flow（2.06）模型。支持自适应采样，减少计算量达60%。模型在部分噪声去噪和异常检测中表现优异，验证其多任务能力。理论分析保证学习数据流形，采样收敛速度优于传统方法。
Applications
可用于高质量图像生成、图像修复、异常检测及多模态数据融合。模型的优化采样机制适合实际部署，能显著提升生成速度和质量，适应多任务场景，推动生成模型在工业和科研中的应用。
Limitations & Outlook
训练依赖大量数据和计算资源，模型在极高分辨率或复杂场景下表现仍有限。能量景观的学习和采样效率有待提升，模型泛化能力在多模态和动态场景中仍需验证。未来需优化算法以降低成本，增强鲁棒性。
Plain Language Accessible to non-experts
想象你在一个工厂里，工厂的任务是制造各种各样的产品。传统的方法就像是按照固定的路线生产，每次都要经过一段特定的步骤，不能随意改变。而新方法像是让工厂学会了一个秘密的能量场，知道每个产品在工厂里的最佳位置和状态。只要你给它一些原料（噪声），它就可以通过在这个能量场中找到最低点，快速制造出符合要求的产品。这就像你在山谷里找到最低的地方，然后沿着坡度走过去，最终到达目标位置。这个能量场是工厂的秘密武器，它让生产变得更快、更灵活，也更能适应不同的需求。这样一来，无论是制造新产品还是修复旧的，都变得更简单、更高效。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏。每次你试图拼出一幅漂亮的图片，都是从一堆乱七八糟的碎片开始。以前的方法就像是你必须按照一定的顺序拼，不能随意变换。而现在，这个新方法教你一种聪明的技巧：你只需要知道每个碎片应该朝哪个方向移动，慢慢调整，直到拼出完整的图片。它用一种“能量场”告诉你，哪里是正确的地方，哪里还需要调整。你不用一次拼完所有碎片，而是一步步用这个“能量场”引导你，逐渐拼出漂亮的图画。这就像在山谷里找到最低点，然后沿着坡度走过去，最终到达目标。这个方法让拼图变得更快、更容易，也能拼出更漂亮的图片。
Glossary
能量景观 (Energy Landscape)
描述数据空间中能量值的函数，低能量点对应真实数据，模型通过学习这个景观实现生成。
EqM通过学习能量景观的梯度，实现样本生成。
优化采样 (Optimization-based Sampling)
利用梯度下降等优化算法，从噪声或部分噪声样本中逐步逼近真实数据。
EqM在推理阶段采用优化采样策略。
隐式能量模型 (Implicit Energy Model)
通过学习能量梯度场而非直接学习能量值的模型，提供更稳定的训练方式。
EqM支持隐式能量学习。
平衡动力学 (Equilibrium Dynamics)
描述系统在能量最小值附近的稳定状态，不依赖时间变化。
EqM核心思想是学习平衡梯度。
梯度下降 (Gradient Descent)
一种优化算法，通过沿梯度方向逐步减小目标函数值。
用于EqM的采样过程。
Open Questions Unanswered questions from this research
- 1 如何进一步提升EqM在高分辨率和复杂场景中的采样效率和稳定性，仍需探索更高效的能量景观学习和优化算法。
- 2 模型在多模态、多任务环境下的泛化能力和鲁棒性尚未充分验证，未来需结合自监督学习提升适应性。
Applications
Immediate Applications
高质量图像生成
可用于生成逼真的图片、艺术作品和虚拟场景，提升内容创作效率，适合广告、娱乐和设计行业。
图像修复与增强
支持部分噪声图像的去噪和修复，应用于医学影像、旧照片修复等领域，提升图像质量。
Long-term Vision
多模态内容生成
结合文本、音频、视频等多模态数据，实现跨模态内容的高效生成，推动虚拟现实和增强现实的发展。
Abstract
We introduce Equilibrium Matching (EqM), a generative modeling framework built from an equilibrium dynamics perspective. EqM discards the non-equilibrium, time-conditional dynamics in traditional diffusion and flow-based generative models and instead learns the equilibrium gradient of an implicit energy landscape. Through this approach, we can adopt an optimization-based sampling process at inference time, where samples are obtained by gradient descent on the learned landscape with adjustable step sizes, adaptive optimizers, and adaptive compute. EqM surpasses the generation performance of diffusion/flow models empirically, achieving an FID of 1.90 on ImageNet 256$\times$256. EqM is also theoretically justified to learn and sample from the data manifold. Beyond generation, EqM is a flexible framework that naturally handles tasks including partially noised image denoising, OOD detection, and image composition. By replacing time-conditional velocities with a unified equilibrium landscape, EqM offers a tighter bridge between flow and energy-based models and a simple route to optimization-driven inference.