Balanced Thinking: Improving Chain of Thought Training in Vision Language Models
SCALe method enhances reasoning and answer accuracy in vision-language models through dynamic weight adjustment.
Key Findings
Methodology
SCALe employs dynamic adjustment of weights between <think> and <answer> segments to optimize reasoning and answer generation in vision-language models. It uses a cosine scheduling policy to gradually shift focus from reasoning to answer segments, ensuring the model progressively emphasizes answer accuracy during training.
Key Results
- SCALe improved model accuracy by 3% on ScienceQA and 5% on IconQA datasets.
- Compared to standard SFT, SCALe reduces training time to one-seventh while achieving similar performance without GRPO.
- When combined with GRPO, SCALe achieves the best overall performance, further enhancing model capabilities.
Significance
SCALe significantly enhances vision-language models' performance in multimodal reasoning tasks, reducing verbose reasoning's impact on answer accuracy, providing a lightweight training alternative suitable for resource-constrained environments.
Technical Contribution
SCALe introduces a dynamic weight adjustment mechanism to address the imbalance between reasoning and answer segments in traditional SFT, offering a new training framework that reduces reliance on GRPO.
Novelty
SCALe is the first to introduce a dynamic weight adjustment strategy in training vision-language models, differing from traditional methods by achieving a balance between reasoning and answer segments through adaptive loss scheduling.
Limitations
- SCALe may still experience performance degradation when handling extremely long reasoning segments.
- The method may require additional tuning for specific tasks.
Future Work
Future work could explore SCALe's application in other multimodal tasks and further optimize its adaptability across different model architectures.
AI Executive Summary
Vision-language models typically rely on a two-stage process for multimodal reasoning: supervised fine-tuning and reinforcement learning. However, standard supervised fine-tuning often overlooks the inherent token imbalance in reasoning data, leading to verbose reasoning overshadowing critical answer segments. The SCALe method addresses this by dynamically adjusting weights between reasoning and answer segments, significantly improving model accuracy and training efficiency.
SCALe employs a cosine scheduling policy to gradually shift focus from reasoning to answer segments, ensuring the model progressively emphasizes answer accuracy during training. Experimental results demonstrate SCALe's superior performance across multiple benchmarks, not only improving model accuracy but also reducing training time.
The introduction of SCALe offers a lightweight alternative for training vision-language models, especially in resource-constrained environments. Future research could further explore SCALe's application in other multimodal tasks and its adaptability across different model architectures.
Deep Analysis
Background
Recent advances in vision-language models have significantly improved performance in multimodal reasoning tasks, typically adopting a two-stage training paradigm: supervised fine-tuning on annotated data followed by reinforcement learning for task-specific alignment. However, traditional supervised fine-tuning methods often overlook the imbalance between reasoning and answer segments, leading to verbose reasoning overshadowing answer accuracy.
Core Problem
The core problem in multimodal reasoning tasks for vision-language models is achieving a balance between reasoning and answer segments. Traditional supervised fine-tuning methods assign equal weight to all tokens, neglecting the functional asymmetry between reasoning and answer segments, resulting in verbose reasoning overshadowing answer accuracy.
Innovation
SCALe introduces a dynamic weight adjustment mechanism to address the imbalance between reasoning and answer segments in traditional SFT. Its innovation lies in using a cosine scheduling policy to gradually shift focus from reasoning to answer segments, ensuring the model progressively emphasizes answer accuracy during training.
Methodology
- �� SCALe employs dynamic adjustment of weights between <think> and <answer> segments to optimize reasoning and answer generation.
- �� Uses a cosine scheduling policy to gradually shift focus from reasoning to answer segments, ensuring the model progressively emphasizes answer accuracy during training.
- �� Initially assigns higher weight to reasoning segments to encourage the model to learn structured reasoning patterns, gradually reducing reasoning weight and increasing answer weight as training progresses.
Experiments
Experiments were conducted on ScienceQA and IconQA datasets using the Vision-R1 framework's standard two-phase pipeline. Compared to standard SFT, SCALe reduces training time to one-seventh while achieving similar performance without GRPO. Experimental results demonstrate SCALe's superior performance across multiple benchmarks, not only improving model accuracy but also reducing training time.
Results
SCALe improved model accuracy by 3% on ScienceQA and 5% on IconQA datasets. Compared to standard SFT, SCALe reduces training time to one-seventh while achieving similar performance without GRPO. When combined with GRPO, SCALe achieves the best overall performance, further enhancing model capabilities.
Applications
SCALe is suitable for multimodal reasoning tasks requiring efficient training, especially in resource-constrained environments. Its dynamic weight adjustment mechanism significantly enhances reasoning and answer generation capabilities without relying on reinforcement learning.
Limitations & Outlook
SCALe may still experience performance degradation when handling extremely long reasoning segments. The method may require additional tuning for specific tasks. Future work could explore SCALe's application in other multimodal tasks and further optimize its adaptability across different model architectures.
Plain Language Accessible to non-experts
Imagine you're cooking in a kitchen. Traditional methods are like using the same heat for all dishes, whether it's a slow-cooked soup or a quick stir-fry. SCALe is like a smart stove that automatically adjusts the heat based on each dish's needs. This way, the soup doesn't overcook, and the stir-fry isn't underdone. SCALe ensures each dish reaches its optimal cooking state.
ELI14 Explained like you're 14
Imagine you're playing a game where you have to manage multiple tasks. Traditional methods are like giving each task the same time, regardless of importance. SCALe is like a smart assistant that adjusts time allocation based on task priority. This way, you can complete all important tasks within limited time and score higher! That's what SCALe does in vision-language models, making them smarter in allocating attention for better results.
Glossary
SCALe (Scheduled Curriculum Adaptive Loss)
A method that optimizes model training by dynamically adjusting weights between reasoning and answer segments.
Used to address the imbalance between reasoning and answer segments.
SFT (Supervised Fine-Tuning)
A model fine-tuning method on annotated data.
Used to establish reasoning priors in models.
GRPO (Group Relative Policy Optimization)
A reinforcement learning method that optimizes models by rewarding answer accuracy and preferred response styles.
Used to refine model reasoning behavior.
Vision-R1
A framework for multimodal reasoning with 200K samples.
Used to evaluate the effectiveness of the SCALe method.
Cosine Scheduling Policy
A strategy for dynamically adjusting weights to ensure smooth transitions during training.
Used to gradually shift focus from reasoning to answer segments.
Open Questions Unanswered questions from this research
- 1 How to maintain SCALe's performance with extremely long reasoning segments?
- 2 How adaptable is SCALe to other multimodal tasks?
- 3 How to further optimize SCALe's performance across different model architectures?
Applications
Immediate Applications
Multimodal Reasoning Tasks
SCALe is suitable for multimodal reasoning tasks requiring efficient training, especially in resource-constrained environments.
Long-term Vision
Intelligent Vision-Language Models
SCALe is expected to advance intelligent vision-language models, enhancing their performance in practical applications.
Abstract
Multimodal reasoning in vision-language models (VLMs) typically relies on a two-stage process: supervised fine-tuning (SFT) and reinforcement learning (RL). In standard SFT, all tokens contribute equally to the loss, even though reasoning data are inherently token-imbalanced. Long <think> traces overshadow short but task-critical <answer> segments, leading to verbose reasoning and inaccurate answers. We propose SCALe (Scheduled Curriculum Adaptive Loss), which explicitly separates supervision over reasoning and answer segments using dynamic, length-independent weighting. Unlike vanilla SFT, which overweights the <think> segment, SCALe-SFT gradually shifts the focus from <think> to <answer> throughout training via a cosine scheduling policy, encouraging concise and well-grounded reasoning. We evaluate SCALe across diverse benchmarks and architectures. Results show that SCALe consistently improves accuracy over vanilla SFT and matches the performance of the full two-phase SFT + GRPO pipeline while requiring only about one-seventh of the training time, making it a lightweight yet effective alternative. When combined with GRPO, SCALe achieves the best overall performance, highlighting its value both as a standalone method and as a strong foundation for reinforcement refinement.