HyperEyes: Dual-Grained Efficiency-Aware Reinforcement Learning for Parallel Multimodal Search Agents
HyperEyes enhances efficiency in multimodal search, improving accuracy by 9.9% with 5.3x fewer tool calls.
Key Findings
Methodology
HyperEyes employs a Dual-Grained Efficiency-Aware Reinforcement Learning framework, combining TRACE and On-Policy Distillation. TRACE suppresses redundant tool calls through dynamically tightening trajectory-level references, while On-Policy Distillation injects dense token-level corrective signals from an external teacher model.
Key Results
- HyperEyes-30B achieves a 9.9% accuracy improvement on the IMEB benchmark with 5.3x fewer tool-call rounds, significantly outperforming existing open-source agents.
- Across six benchmarks, HyperEyes demonstrates superior performance in both accuracy and efficiency, Pareto-dominating existing models.
- Ablation studies confirm the effectiveness of TRACE and OPD, showcasing their advantages in multi-entity scenarios.
Significance
This research introduces a parallel multimodal search agent, addressing the redundancy issues of traditional sequential tool calls and significantly enhancing search efficiency and accuracy. It holds substantial potential for applications in multi-entity visual search, advancing the development of multimodal search agents.
Technical Contribution
HyperEyes technically integrates visual grounding and retrieval into a unified action space for parallel search across text and visual modalities, introducing a Dual-Grained Efficiency-Aware RL framework that optimizes tool-use efficiency.
Novelty
HyperEyes is the first to introduce parallel search in multimodal agents, combining TRACE and OPD frameworks to overcome the limitations of traditional sequential search and offer new efficiency optimization pathways.
Limitations
- In complex scenarios, the model may still face redundancy issues, especially when initial localization is inaccurate.
- The training and inference costs are high, potentially limiting its application in resource-constrained environments.
Future Work
Future work could explore applications on larger datasets, optimize computational efficiency, and validate performance in more real-world scenarios.
AI Executive Summary
Existing multimodal search agents typically process target entities sequentially, leading to redundant interaction rounds. HyperEyes introduces a parallel multimodal search agent that fuses visual grounding and retrieval into a single atomic action, enabling concurrent search across multiple entities while prioritizing inference efficiency.
HyperEyes employs a two-stage training strategy, starting with a Parallel-Amenable Data Synthesis Pipeline for cold-start supervision, followed by a Dual-Grained Efficiency-Aware Reinforcement Learning framework to optimize search efficiency. At the macro level, TRACE suppresses redundant tool calls through dynamically tightening trajectory-level references, while at the micro level, On-Policy Distillation provides dense token-level corrective signals from an external teacher model.
Across six benchmarks, HyperEyes-30B surpasses the strongest open-source multimodal search agent in accuracy by 9.9% with 5.3x fewer tool-call rounds, demonstrating significant advantages in multi-entity visual search. The introduction of the IMEB benchmark enables joint evaluation of search capability and efficiency, advancing the development of multimodal search agents. Future work will explore applications on larger datasets and validate performance in more real-world scenarios.
Deep Analysis
Background
Multimodal search agents often rely on sequential tool calls to handle complex multi-hop reasoning tasks, leading to redundant interaction rounds. Existing benchmarks primarily evaluate accuracy, neglecting inference cost and parallel search capability. HyperEyes addresses this issue through a parallel multimodal search agent.
Core Problem
Traditional multimodal search agents face redundancy issues due to sequential tool calls when handling decomposable queries. Improving search efficiency while maintaining accuracy is a critical challenge.
Innovation
HyperEyes introduces a parallel multimodal search agent, integrating visual grounding and retrieval into a single atomic action for concurrent search across multiple entities. It employs a Dual-Grained Efficiency-Aware RL framework to optimize tool-use efficiency.
Methodology
- �� Employ a Parallel-Amenable Data Synthesis Pipeline for cold-start supervision.
- �� At the macro level, TRACE suppresses redundant tool calls through dynamically tightening trajectory-level references.
- �� At the micro level, On-Policy Distillation provides dense token-level corrective signals from an external teacher model.
Experiments
Experiments were conducted across six benchmarks, using the IMEB benchmark to evaluate search capability and efficiency. HyperEyes-30B demonstrates superior performance in both accuracy and tool-call rounds, showcasing its advantages in multi-entity scenarios.
Results
HyperEyes-30B achieves a 9.9% accuracy improvement on the IMEB benchmark with 5.3x fewer tool-call rounds. Ablation studies confirm the effectiveness of TRACE and OPD, showcasing their advantages in multi-entity scenarios.
Applications
HyperEyes holds substantial potential for applications in multi-entity visual search, particularly in scenarios requiring efficient parallel search, such as image recognition and information retrieval.
Limitations & Outlook
The model may still face redundancy issues in complex scenarios, especially when initial localization is inaccurate. The training and inference costs are high, potentially limiting its application in resource-constrained environments.
Plain Language Accessible to non-experts
Imagine a library where traditional multimodal search agents act like a librarian handling one request at a time, leading to inefficiency. HyperEyes is like a super librarian that can handle multiple requests simultaneously, greatly improving efficiency. It conducts parallel searches to quickly find the needed information, much like finding multiple books in a large room at once.
ELI14 Explained like you're 14
Hey there! Imagine you're playing a game where you need to find several treasures at once. The old way is to find them one by one, super slow! But HyperEyes is like a super helper that finds all the treasures at the same time, cool right? That's the power of parallel search!
Glossary
HyperEyes
A parallel multimodal search agent that integrates visual grounding and retrieval into a single atomic action.
Used to enhance multi-entity search efficiency.
TRACE
Tool-use Reference-Adaptive Cost Efficiency, a trajectory-level reward.
Used to suppress redundant tool calls.
On-Policy Distillation
A method providing dense token-level corrective signals from an external teacher model.
Used to correct failed reasoning steps.
IMEB
A human-curated benchmark for joint evaluation of search capability and efficiency.
Used to assess multimodal search agent performance.
Parallel Search
The ability to process multiple queries simultaneously.
Used to improve search efficiency.
Open Questions Unanswered questions from this research
- 1 How to further improve search efficiency in more complex scenarios? Current methods still face challenges when initial localization is inaccurate.
- 2 How to reduce the training and inference costs to enable application in resource-constrained environments?
Applications
Immediate Applications
Image Recognition
In scenarios requiring rapid identification of multiple targets, HyperEyes can significantly enhance efficiency.
Long-term Vision
Information Retrieval
Apply on large-scale datasets, optimize computational efficiency, and advance the development of multimodal search agents.
Abstract
Existing multimodal search agents process target entities sequentially, issuing one tool call per entity and accumulating redundant interaction rounds whenever a query decomposes into independent sub-retrievals. We argue that effective multimodal agents should search wider rather than longer: dispatching multiple grounded queries concurrently within a round. To this end, we present HyperEyes, a parallel multimodal search agent that fuses visual grounding and retrieval into a single atomic action, enabling concurrent search across multiple entities while treating inference efficiency as a first-class training objective. HyperEyes is trained in two stages. For cold-start supervision, we develop a Parallel-Amenable Data Synthesis Pipeline covering visual multi-entity and textual multi-constraint queries, curating efficiency-oriented trajectories via Progressive Rejection Sampling. Building on this, our central contribution, a Dual-Grained Efficiency-Aware Reinforcement Learning framework, operates at two levels. At the macro level, we propose TRACE (Tool-use Reference-Adaptive Cost Efficiency), a trajectory-level reward whose reference is monotonically tightened during training to suppress superfluous tool calls without restricting genuine multi-hop search. At the micro level, we adapt On-Policy Distillation to inject dense token-level corrective signals from an external teacher on failed rollouts, mitigating the credit-assignment deficiency of sparse outcome rewards. Since existing benchmarks evaluate accuracy as the sole metric, omitting inference cost, we introduce IMEB, a human-curated benchmark of 300 instances that jointly evaluates search capability and efficiency. Across six benchmarks, HyperEyes-30B surpasses the strongest comparable open-source agent by 9.9% in accuracy with 5.3x fewer tool-call rounds on average.