Test-Time Scaling for Small VLMs on Multilingual Visual MCQ
Test-Time Scaling (TTS) significantly improves reasoning in small VLMs on multilingual visual MCQ, achieving 84.1% accuracy in the best configuration.
Key Findings
Methodology
The study compares self-consistency, describe-then-reason (DTR) with PRM-guided search, and two post-hoc selectors across Qwen2.5-VL-7B-Instruct and Qwen3.5-4B models on the EXAMS-V dataset.
Key Results
- Result 1: Qwen3.5-4B achieves 84.1% accuracy in the best configuration, outperforming other methods.
- Result 2: Increasing token budget per chain from 1k to 2k improves accuracy by 3.7 pp, while doubling chain count (8 to 16) adds only 0.15 pp.
- Result 3: PRM-guided search costs 8.7x more than self-consistency but underperforms by 0.39 pp.
Significance
The research highlights that TTS effectiveness in small VLMs depends on decoding budget and parseability rather than complex search or verification mechanisms, offering valuable insights for resource-constrained reasoning tasks.
Technical Contribution
Introduces an efficient TTS framework optimizing parse format and token budget, demonstrating the superiority of self-consistency in multilingual visual MCQ tasks.
Novelty
First systematic study of TTS applicability in small VLMs, proposing parse repair steps and validating their effectiveness.
Limitations
- Limitation 1: Results are limited to Qwen models, potentially not generalizable to other architectures.
- Limitation 2: PRM calibration for multilingual non-mathematical content remains weak, affecting search efficacy.
Future Work
Future work could explore dedicated reasoning models to decouple perception and reasoning and improve PRM calibration for diverse multilingual content.
AI Executive Summary
Test-Time Scaling (TTS) has proven effective for large language models, but its applicability to small open vision-language models remains unclear. This study evaluates two models (Qwen2.5-VL-7B-Instruct and Qwen3.5-4B) under various TTS strategies on the EXAMS-V multilingual visual MCQ benchmark.
Findings reveal that TTS effectiveness hinges on decoding budget and parseability rather than complex search or verification mechanisms. Increasing token budget per chain from 1k to 2k improves accuracy by 3.7 pp, while doubling chain count adds only 0.15 pp. Parse repair steps significantly reduce parse failure rates, enhancing model performance further.
Ultimately, Qwen3.5-4B achieves 84.1% accuracy in its best configuration, ranking first on the Visual MCQ leaderboard. This study provides critical guidance for optimizing reasoning in resource-constrained environments while identifying future directions such as dedicated reasoning models and improved PRM calibration for multilingual tasks.
Deep Analysis
Background
Test-Time Scaling (TTS) optimizes inference strategies to improve model performance, showing success in large language models. However, its potential in small vision-language models for multilingual and multimodal reasoning tasks remains underexplored.
Core Problem
Multilingual visual MCQs require models to combine structured visual content with multi-step reasoning. Existing methods face bottlenecks in parse failures and insufficient reasoning depth.
Innovation
The study proposes an optimized TTS framework, including parse repair steps and token budget adjustments, significantly improving performance on the EXAMS-V benchmark.
Methodology
- �� Compare self-consistency and describe-then-reason (DTR) with PRM-guided search.
- �� Increase token budget per chain to reduce parse failures.
- �� Apply parse repair steps to ensure answer generation.
- �� Validate using Qwen models on EXAMS-V dataset.
Experiments
Experiments use the EXAMS-V dataset covering 13 languages and 20 subjects. Two models are evaluated under various TTS configurations, with ablation studies on parse repair and token budget adjustments.
Results
Qwen3.5-4B achieves 84.1% accuracy in its best configuration. Parse repair steps significantly reduce parse failure rates, and token budget adjustments enhance performance.
Applications
Applicable to multilingual educational assessments, cross-language visual Q&A systems, and reasoning tasks in resource-constrained environments.
Limitations & Outlook
The study focuses on Qwen models, and PRM calibration for multilingual content remains limited. Future work should explore broader models and tasks.
Plain Language Accessible to non-experts
Imagine you're solving a complex visual puzzle that requires combining clues from images and text. This study proposes a strategy to give the model more time to think and fix errors in its answers, much like giving a student extra time and a final review step to ensure accuracy.
ELI14 Explained like you're 14
Think of playing a game where you need to solve puzzles using pictures and clues. If you rush, you might miss important details. This study's method is like giving you extra time and a helper to check your answers, so you score higher and win more often!
Glossary
Test-Time Scaling (TTS)
A technique to optimize inference by adjusting chain count or token budget.
Used to improve reasoning in multilingual visual MCQs.
Self-Consistency
A method that generates multiple reasoning chains and selects answers by majority vote.
Proven more efficient than complex search methods in this study.
Process Reward Model (PRM)
A model that scores reasoning steps to guide search processes.
Combined with DTR framework for experiments.
Parse Repair
A technique to fix parse failures by forcing answer token generation.
Significantly reduces parse failure rates.
EXAMS-V
A multilingual visual MCQ benchmark covering 13 languages and 20 subjects.
Used to evaluate multimodal reasoning capabilities.
Open Questions Unanswered questions from this research
- 1 PRM calibration for multilingual content remains weak, requiring further optimization.
- 2 How to develop dedicated reasoning models to decouple perception and reasoning.
Applications
Immediate Applications
Educational Assessment
Improves accuracy in multilingual exam systems for automated grading.
Visual Q&A
Enhances user experience in cross-language question-answering systems.
Long-term Vision
Intelligent Education Assistant
Develop AI capable of understanding multilingual visual content for personalized learning.
Abstract
Test-time scaling (TTS) reliably improves reasoning in large language models, but whether it transfers to small open vision-language models remains unclear. We examine this on EXAMS-V, a multilingual visual multiple-choice benchmark, comparing self-consistency, describe-then-reason with PRM-guided beam search, and two post-hoc selectors across Qwen2.5-VL-7B-Instruct and Qwen3.5-4B. What matters is the conditions under which TTS runs, not the search or verification machinery. The largest factor is parseability: an early prompt format left many chains reasoning correctly yet never committing to an answer letter, which a standard answer cue and a guided repair step largely remove. A larger decoding budget removes the rest: raising the per-chain token limit from 1k to 2k recovers 3.7 pp, whereas sampling more chains (8 to 16) adds only 0.15 pp. Once chains have room to finish, elaborate methods contribute little: PRM-guided beam search trails plain self-consistency by 0.39 pp at over eight times the cost, and neither a training-free generative critic nor a trained multimodal PRM beats majority vote across both policies. The largest gain comes instead from the policy model itself (+11.4 pp). Our best configuration reaches 84.1% on the held-out ImageCLEF 2026 test split, ranking first on the Visual MCQ leaderboard.