PeRL: Permutation-Enhanced Reinforcement Learning for Interleaved Vision-Language Reasoning

The evolution of multimodal AI has seen rapid progress with models like VisualBERT, LXMERT, and VL-BERT, which primarily focus on single-image understanding. As applications demand more complex scene comprehension, multi-image reasoning has gained attention, exemplified by models such as Flamingo, LLaVA-Interleave, and Migician. These models attempt to encode inter-image relationships but face challenges like positional bias and limited generalization in complex spatial configurations. Reinforcement learning has been explored to enhance reasoning, yet most approaches remain constrained by static data and lack robust handling of multi-image spatial relations. The gap persists in effectively modeling diverse spatial arrangements and mitigating positional biases, which are critical for real-world scene understanding and visual question answering.

Multi-image reasoning involves understanding complex spatial and semantic relationships across images, which is hindered by positional bias—models overly sensitive to image order—and difficulty in generalizing to diverse spatial configurations. Existing methods struggle to adapt to varied real-world scenarios where image sequences can vary arbitrarily, leading to inconsistent reasoning results. Addressing these issues requires a strategy that not only enhances spatial understanding but also reduces positional sensitivity, ensuring models can reliably interpret multi-image inputs in dynamic environments. The core challenge lies in designing training mechanisms that promote invariance to image order while capturing intricate spatial relations.

The key innovations include: 1) Sequence permutation strategy that randomly swaps images during training, simulating diverse spatial arrangements to improve spatial relation learning. 2) Multi-stage data filtering pipeline that balances sample difficulty, incorporating rule-based filtering, text rephrasing, and semantic variation checks to ensure high-quality training data. 3) Rollout filtering mechanism that dynamically resamples trajectories based on informativeness, stabilizing training and accelerating convergence. 4) Integration of GRPO, a policy optimization algorithm that estimates advantages over multiple responses, effectively addressing positional bias. These innovations collectively enable the model to learn more robust spatial and semantic relationships, significantly improving multi-image reasoning performance.

• �� Construct a multi-stage data pipeline: apply rule-based filtering to remove low-quality samples, use GPT-4o for question rephrasing to maintain semantics, implement rollout filtering with Qwen2.5-VL to evaluate sample informativeness, and perform semantic variation checks to ensure invariance under permutation. • During training, randomly swap image sequences per sample, modifying textual references accordingly, to simulate varied spatial configurations. • Use GRPO to generate multiple responses per permutation, compute advantages via rule-based rewards, and perform dynamic resampling to focus on informative trajectories. • Incorporate a decaying probability of permutation application to ensure training stability. • Optimize policy parameters via gradient updates based on combined advantage estimates, balancing exploration and exploitation across diverse spatial scenarios. • Validate model performance on multiple benchmarks, analyzing ablation effects of permutation and filtering strategies.

The experimental setup includes training on 22K multi-image instruction samples from Mantis-Instruct and 36K single-image math samples, with the policy initialized from Qwen2.5-VL-7B. The evaluation covers five multi-image benchmarks (Mantis-Eval, BLINK, MMIU, Remi, MV-Math) and three single-image benchmarks (MathVista, MathVerse, MathVision). Hyperparameters include one permutation per sample, six responses per permutation, training for two epochs, learning rate 1e-6, batch size 256. Ablation studies compare permutation versus static training, analyze the impact of rollout filtering, and evaluate the effect of sample difficulty balancing. Results show that permutation significantly improves spatial relation understanding, and rollout filtering stabilizes training, leading to superior performance across benchmarks.

PeRL achieves 76.39% accuracy on Mantis-Eval, outperforming previous models by 5.59 points, and maintains high scores on single-image math benchmarks (73.00 on MathVista, 49.56 on MathVerse). Ablation results confirm that sequence permutation enhances spatial reasoning, while rollout filtering improves training stability and efficiency. Cross-scenario testing demonstrates strong generalization, especially in out-of-distribution datasets like Remi and MV-Math. The model’s robustness and accuracy gains highlight the effectiveness of the proposed strategies, setting new standards in multi-image reasoning performance.

The approach can be directly applied to visual question answering, scene understanding, autonomous navigation, and robotic perception, where understanding complex spatial relationships is critical. Its robustness makes it suitable for real-world applications requiring reliable multi-image interpretation under varying conditions. Long-term, integrating this framework with knowledge graphs and self-supervised learning could enable more intelligent, context-aware systems capable of reasoning in dynamic environments, significantly advancing AI’s perceptual and reasoning capabilities.

Despite its strengths, the method incurs high computational costs due to multiple permutations and responses. Handling extremely complex or large-scale multi-image scenes remains challenging, with potential semantic noise introduced during permutation. The reliance on pre-trained models and rule-based filtering limits end-to-end learning and adaptability. Future work should focus on reducing computational overhead, improving semantic robustness, and exploring more scalable, integrated learning paradigms to extend applicability.