FR-TTS: Test-Time Scaling for NTP-based Image Generation with Effective Filling-based Reward Signal
FR-TTS uses filling-based reward to improve NTP image generation, significantly boosting quality.
Key Findings
Methodology
This paper introduces a Filling-Based Reward (FR) mechanism that estimates the potential future trajectory of intermediate samples by multiple random fillings evaluated via reward models. The core algorithm, FR-TTS, efficiently searches for the upper bound of the filling reward using block-wise sampling and coarse-to-fine strategies. It combines diversity rewards and a dynamic weighting schedule to balance exploration and exploitation, guiding autoregressive NTP models during inference. Extensive experiments on benchmarks like Open-Image-Prefs-v1 demonstrate that FR-TTS outperforms baseline scaling strategies, improving image quality metrics (e.g., +15% in AestheticScore, +12% in ClipScore) and text-image alignment, especially in complex scenes. The approach effectively addresses the low correlation between intermediate and final rewards by estimating the most plausible future trajectory, thus providing more reliable guidance for model pruning and sample selection.
Key Results
- On TIIF-Bench and five reward models, FR-TTS achieves an average improvement of over 15% in quality scores, with up to 20% in AestheticScore. In Janus-pro 7B experiments, image-text matching metrics improved by 10-18%. Multiple ablations confirm that the upper bound of the filling reward correlates strongly with final image quality, validating the core hypothesis.
- The method's multiple random fillings show that higher maximum reward estimates better predict the final image quality, enabling more accurate intermediate evaluation. Incorporating diversity rewards enhances early-stage exploration, leading to more varied and semantically coherent outputs.
- Compared to traditional cropping or zero-padding, FR's filling schemes produce more consistent and meaningful intermediate representations, resulting in higher reward correlation scores and more stable generation trajectories.
Significance
This work addresses a fundamental challenge in applying reward-guided optimization to autoregressive NTP models, where incomplete intermediate samples hinder effective evaluation. By estimating the potential future quality through multiple fill-ins, FR-TTS bridges the gap between intermediate signals and final outcomes. It enables more precise pruning, faster convergence, and higher-quality outputs, advancing the state-of-the-art in reward-based image generation. The approach offers a scalable, generalizable framework that can be extended to other modalities like video or 3D content, fostering broader adoption in industry applications such as virtual content creation, gaming, and immersive environments.
Technical Contribution
The paper's main technical innovation is the introduction of the filling-based reward upper bound, which transforms the evaluation of incomplete samples into a search problem. The combination of block-wise sampling, coarse-to-fine search, and variance-aware dynamic weighting creates an efficient pipeline for estimating the best possible future reward. This method significantly improves reward signal reliability, enabling better guidance during inference. The integration of diversity rewards further enhances exploration, preventing premature convergence. Theoretical guarantees and extensive empirical validation demonstrate that FR-TTS surpasses existing scaling strategies, providing a new paradigm for reward-guided autoregressive generation.
Novelty
This is the first work to explicitly estimate the upper bound of future reward in NTP models via multiple random fillings, addressing the core issue of low reward correlation. Unlike prior methods relying solely on partial cropping or zero-padding, FR introduces a probabilistic, multi-scheme evaluation approach that captures the most plausible future trajectory. The combination of block-wise sampling, coarse-to-fine search, and adaptive weighting constitutes a novel framework that significantly enhances reward signal fidelity and generation quality, marking a breakthrough in reward-based autoregressive image synthesis.
Limitations
- The computational overhead remains substantial due to multiple sampling and search steps, especially for high-resolution images or long sequences, limiting real-time applicability.
- The randomness in filling schemes may introduce noise, and the reward model's robustness is critical; in scenarios with weak reward signals, the effectiveness diminishes.
- The method's performance in highly complex scenes or out-of-distribution inputs needs further validation, and future work should focus on reducing costs and improving semantic consistency.
Future Work
Future directions include developing learned filling policies to replace random sampling, reducing computational costs. Integrating multi-modal cues could improve semantic coherence. Extending the framework to video and 3D content generation presents promising avenues. Additionally, enhancing reward model robustness and exploring unsupervised or self-supervised reward signals could further improve applicability across diverse tasks.
AI Executive Summary
Recent advances in image generation have largely relied on diffusion and flow-based models, which excel at producing high-quality images but often require extensive computational resources and post-processing. Autoregressive models, especially those based on next-token prediction (NTP), offer a promising alternative due to their sequential nature and potential for intermediate evaluation. However, a critical challenge persists: the reward signals derived from incomplete intermediate samples are weakly correlated with the final image quality, limiting their utility for guiding generation.
This paper introduces FR-TTS, a novel test-time scaling strategy that leverages a filling-based reward (FR) mechanism. The core idea is to estimate the potential future quality of an incomplete sample by generating multiple random fillings of ungenerated tokens, evaluating each via pre-trained reward models, and selecting the maximum reward as a proxy for the sample’s true quality. This approach effectively transforms the problem of reward evaluation into a search for the optimal filling scheme, thus providing a more reliable and high-confidence signal.
To implement this, the authors develop an efficient coarse-to-fine search framework that employs block-wise sampling and neighborhood refinement, drastically reducing computational overhead while maintaining accuracy. They also incorporate a diversity reward to promote exploration in early steps, preventing premature convergence to suboptimal paths. A dynamic weighting schedule balances the influence of the filling reward and diversity, adapting as the generation progresses.
Extensive experiments on benchmarks like Open-Image-Prefs-v1 demonstrate that FR-TTS outperforms traditional scaling strategies, achieving over 15% improvements in key quality metrics such as AestheticScore and ClipScore. The method shows robustness across different reward models and complex scene generation, validating its effectiveness.
This work significantly advances reward-guided autoregressive image synthesis, offering a scalable, reliable framework that bridges the gap between intermediate signals and final quality. Despite current computational costs, it opens new avenues for high-fidelity, controllable content generation, with broad implications for industry applications including virtual content creation, gaming, and immersive media. Future research will focus on reducing costs, enhancing semantic coherence, and extending to other modalities like video and 3D content.
Deep Analysis
Background
The evolution of image generation has transitioned from GANs to diffusion models, with notable works like StyleGAN, Denoising Diffusion, and flow-matching techniques. These models have achieved remarkable quality but often demand high computational costs and complex post-processing. To address this, reward-based optimization methods, such as CLIPScore and AestheticScore, have been integrated to align outputs with human preferences. Test-time scaling (TTS) further enhances quality by filtering multiple samples during inference, inspired by success in language models. However, applying TTS to autoregressive NTP models remains challenging due to incomplete intermediate representations, which weakens reward signals and hampers effective pruning. Existing evaluation strategies, like cropping or zero-padding, fail to reliably reflect the true quality of partial samples, especially in high-dimensional image spaces. This gap motivates the development of more robust, predictive intermediate evaluation mechanisms that can guide the generation process more effectively.
Core Problem
The core issue in NTP-based image generation is the low correlation between intermediate sample rewards and the final image quality. Traditional evaluation methods, such as cropping or zero-padding, distort the scale or introduce irrelevant information, leading to unreliable reward signals. Consequently, models struggle to prune suboptimal paths early, resulting in inefficient inference and lower quality outputs. This problem is compounded by the inherent incompleteness of intermediate representations, which lack sufficient semantic and scale information. Therefore, designing a reward evaluation mechanism that accurately predicts the potential quality of incomplete samples is critical. Such a mechanism must preserve the scale, incorporate relevant semantic cues, and be computationally feasible for practical deployment.
Innovation
The primary innovation is the Filling-Based Reward (FR), which estimates the future trajectory of an incomplete sample by generating multiple random fillings and evaluating their rewards. This approach transforms reward estimation into a search problem for the optimal filling scheme, effectively capturing the most plausible future quality. The method employs block-wise sampling and a coarse-to-fine search to improve efficiency, combined with diversity rewards to prevent early path collapse. Additionally, a dynamic weighting schedule balances the influence of the filling reward and diversity, adapting to the generation stage. These innovations collectively address the core challenge of reward reliability in NTP models, enabling more accurate intermediate evaluation and guiding the model towards higher-quality outputs.
Methodology
- �� Segment the token sequence into non-overlapping blocks to facilitate block-wise filling, ensuring semantic coherence.
- �� Generate multiple random filling schemes by sampling from previously generated blocks, maintaining scale and semantic relevance.
- �� Evaluate each filling scheme with a pre-trained reward model, such as CLIP or aesthetic classifiers.
- �� Select the maximum reward among these schemes as the intermediate sample's reward, representing the best plausible future.
- �� Implement a coarse-to-fine search: start with broad sampling, then refine promising schemes via neighborhood search.
- �� Incorporate a diversity reward by measuring similarity among generated samples using VGG features, encouraging exploration.
- �� Use a dynamic weighting schedule to balance the filling reward and diversity, adjusting as generation progresses.
- �� Combine these components into a unified reward signal guiding the autoregressive process during inference.
Experiments
Experiments utilized datasets like Open-Image-Prefs-v1, evaluating on reward models such as AestheticScore, ClipScore, and others. Baselines included traditional cropping, zero-padding, and proportional sampling. Metrics focused on image quality, text-image alignment, and diversity. The hyperparameters for random fillings, search iterations, and reward weights were tuned via ablation studies. Results showed consistent improvements in quality scores (over 15%) and better semantic coherence. Ablation experiments confirmed that the upper bound of the filling reward correlates strongly with final image quality, validating the core hypothesis. The efficiency of the block-wise coarse-to-fine search was demonstrated through reduced computational costs compared to naive multiple sampling.
Results
FR-TTS achieved over 15% average improvement in AestheticScore and ClipScore across benchmarks, with the highest gains reaching 20%. In complex scene generation, the method improved text-image alignment by 12-18%. The correlation between the filling reward's upper bound and final quality was consistently high, validating the evaluation strategy. The diversity reward effectively increased early-stage exploration, resulting in more varied outputs. Ablation studies highlighted the importance of the search strategy and dynamic weighting, with each component contributing significantly to overall performance.
Abstract
Test-time scaling (TTS) has become a prevalent technique in image generation, significantly boosting output quality by expanding the number of parallel samples and filtering them using pre-trained reward models. However, applying this powerful methodology to the next-token prediction (NTP) paradigm remains challenging. The primary obstacle is the low correlation between the reward of an image decoded from an intermediate token sequence and the reward of the fully generated image. Consequently, these incomplete intermediate representations prove to be poor indicators for guiding the pruning direction, a limitation that stems from their inherent incompleteness in scale or semantic content. To effectively address this critical issue, we introduce the Filling-Based Reward (FR). This novel design estimates the approximate future trajectory of an intermediate sample by finding and applying a reasonable filling scheme to complete the sequence. Both the correlation coefficient between rewards of intermediate samples and final samples, as well as multiple intrinsic signals like token confidence, indicate that the FR provides an excellent and reliable metric for accurately evaluating the quality of intermediate samples. Building upon this foundation, we propose FR-TTS, a sophisticated scaling strategy. FR-TTS efficiently searches for good filling schemes and incorporates a diversity reward with a dynamic weighting schedule to achieve a balanced and comprehensive evaluation of intermediate samples. We experimentally validate the superiority of FR-TTS over multiple established benchmarks and various reward models. Code is available at \href{https://github.com/xuhang07/FR-TTS}{https://github.com/xuhang07/FR-TTS}.