Group Diffusion: Enhancing Image Generation by Unlocking Cross-Sample Collaboration

The evolution of diffusion models, exemplified by DDPM and Score-based models, has revolutionized high-fidelity image synthesis. These models, trained on vast datasets like ImageNet, excel at learning complex data distributions through iterative denoising. However, during inference, images are generated independently, neglecting potential inter-sample relationships that could enhance quality and diversity. Recent efforts in semantic correspondence and multi-modal fusion have addressed some issues but remain limited to single-sample conditioning. As demands for more coherent, diverse, and high-resolution outputs grow, leveraging sample interdependence during inference emerges as a promising direction. This aligns with broader trends in AI towards collaborative, context-aware systems, yet remains underexplored in diffusion frameworks.

Despite advances, current diffusion models lack mechanisms to utilize relationships among multiple samples during inference, restricting their capacity to produce semantically consistent and diverse outputs. This limitation hampers applications requiring multiple outputs under the same condition or context, such as multi-view generation or style transfer. The core challenge is designing a model that can jointly denoise a set of related images, learning their inter-sample semantic and visual correlations without incurring prohibitive computational costs. Addressing this problem involves developing scalable attention mechanisms, effective sample retrieval strategies, and training protocols that facilitate inter-sample learning, all while maintaining high fidelity and diversity.

The key innovation is the integration of cross-sample attention within the diffusion process, enabling multiple images to influence each other's denoising steps. This is achieved by reshaping tokens in the Transformer architecture to include samples from a constructed group, allowing patches across images to attend to each other. The method employs a pre-trained encoder (like CLIP) to retrieve semantically similar samples, ensuring meaningful interactions. During training, joint denoising loss encourages learning inter-sample correlations. During inference, the model generates multiple images simultaneously, leveraging learned relationships to improve quality. This approach differs fundamentally from prior methods by explicitly modeling sample relationships rather than relying solely on conditioning or post-processing.

• �� Sample group construction: Use a pre-trained encoder (e.g., CLIP) to retrieve semantically similar images, forming a group. • Patch concatenation: Flatten and concatenate patches from all images in the group, forming a unified token sequence. • Cross-sample attention: Implement multi-head attention across the concatenated tokens, enabling patches to attend to patches from other samples. • Training: Minimize joint denoising loss over all samples, optimizing the model to learn inter-sample relationships. • Inference: Generate multiple images conditioned on the same input, with optional cross-sample attention enabled or disabled, balancing quality and efficiency. • Flexibility: Adjust group size and attention layers to control computational cost and performance.

The experiments utilize ImageNet-256x256, training on 800K iterations with AdamW optimizer, batch size 256. Sample retrieval employs CLIP and DINO encoders, with group sizes of 1, 2, 4, 8. Evaluation metrics include FID, Inception Score, and detail fidelity. Ablation studies analyze the impact of group size, attention layer depth, and sample retrieval methods. Visualizations of attention maps confirm early-stage cross-sample interactions are most critical. The model’s ability to disable cross-sample attention at later stages demonstrates that early interactions are sufficient for high-quality generation. Results show that larger groups and similarity-based sample selection consistently outperform baseline models, with the best FID of 2.84 achieved at group size 4.

The primary result is a 32.2% FID reduction (from 4.21 to 2.84) on ImageNet-256x256, outperforming baseline models. Larger group sizes (up to 8) improve quality, with visual attention maps illustrating stronger cross-sample interactions correlating with better details and semantic coherence. Ablation studies reveal early-layer attention and similarity-based sample retrieval are crucial. The model’s ability to selectively disable cross-sample attention at later stages maintains high quality with reduced computational load. These findings validate the hypothesis that sample collaboration enhances generative performance, establishing a new paradigm for diffusion-based image synthesis.

This method benefits tasks requiring multiple related outputs, such as multi-view synthesis, style transfer, and semantic editing. It can be integrated into creative industries, virtual environment generation, and AI-assisted content creation, where sample relationships are vital. The approach also opens avenues for multi-modal and video generation, enabling more coherent and diverse outputs by leveraging inter-sample correlations. Its flexibility allows deployment in resource-constrained settings by adjusting group size and attention layers, making it suitable for real-world applications.

The increased computational cost with larger groups poses scalability challenges, especially at high resolutions. Dependence on pre-trained encoders for sample retrieval may introduce biases, affecting relevance and diversity. In heterogeneous sample scenarios, cross-sample attention could introduce noise, reducing detail fidelity. Future work should focus on optimizing attention efficiency, robust sample selection, and extending the framework to dynamic or multi-modal data, addressing these limitations for broader applicability.