Divide-and-Conquer CoT: RL for Reducing Latency via Parallel Reasoning

Recent advances in large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and reasoning, especially with techniques like chain-of-thought (CoT) prompting. Long CoT enables models to generate detailed reasoning chains, significantly improving performance on complex tasks such as mathematical problem solving and logical inference. However, the sequential generation process inherently leads to high latency, which becomes problematic for real-time applications. Prior efforts mainly relied on heuristic prompt engineering or prompt optimization, which are limited in enabling models to autonomously identify opportunities for parallelization. Reinforcement learning (RL) has been explored to optimize reasoning strategies, but existing methods lack explicit mechanisms for subtask decomposition, especially in the context of long, complex reasoning chains. Consequently, there remains a critical need for architectures that can learn to efficiently partition reasoning processes into parallelizable components, reducing inference time without sacrificing accuracy.

The core challenge addressed in this paper is reducing the latency of long chain-of-thought reasoning in large language models. Traditional models generate responses sequentially, resulting in long response times that hinder deployment in real-time systems. While some tasks naturally decompose into independent subproblems, current models lack the ability to recognize and leverage this structure. The key bottleneck is the inability to balance the trade-off between maintaining high reasoning accuracy and minimizing inference latency, especially as the length of reasoning chains increases. Developing a trainable mechanism for models to autonomously identify and execute sub-tasks in parallel is essential to overcoming this bottleneck, enabling faster and more scalable reasoning systems.

This work introduces the DC-CoT architecture, which explicitly trains models to decompose reasoning tasks into parallel subtasks. The key innovations include:

• �� A director-worker paradigm where the model as a director identifies subtask opportunities and spawns worker modules to perform these tasks concurrently.
• �� A multi-stage RL training process that optimizes a reward function combining correctness and the longest path length, a proxy for latency.
• �� A rewriting strategy to convert sequential CoT responses into parallelizable formats, facilitating supervised training.
• �� Use of the longest path length as a differentiable metric to guide RL optimization.
• �� Empirical validation that the approach reduces latency by 35-40% while maintaining accuracy, outperforming baseline models.

This framework enables models to learn autonomous subtask decomposition, a significant departure from previous heuristic or prompt-based methods, providing a scalable solution for efficient large-scale reasoning.

• �� Start with a base long CoT model (DeepScaleR-1.5B-Preview), perform supervised fine-tuning (SFT) on a curated dataset to teach format following.
• �� Generate initial sequential CoT responses for training data, then use a rewriting model to convert responses into a parallel format, preserving reasoning content.
• �� Fine-tune the model on rewritten data to instill the ability to spawn workers and perform parallel reasoning.
• �� Design a multi-stage RL training process:
• �� Use correctness and path length as reward signals.
• �� Filter data dynamically to balance accuracy and speed.
• �� Incorporate penalties for responses exceeding a path length cutoff.
• �� Rerun RL with different filtering strategies to avoid plateauing.
• �� During inference, implement a two-phase process:
• �� Initial reasoning to identify subtasks.
• �� Parallel execution of subtasks with multiple worker modules.
• �� Aggregation of worker outputs and further reasoning if needed.
• �� Use attention masks and position IDs to simulate parallel and sequential stages during training, ensuring consistency with inference.

The evaluation involved benchmarks like AIME 2024 and HMMT 2025, focusing on accuracy and response latency. The models were trained with different RL reward weights and filtering strategies, comparing the baseline DeepScaleR-1.5B-Preview with DC-CoT variants. Metrics included pass@1 accuracy and the longest path length. Ablation studies examined the impact of data filtering, RL stages, and response format. The experiments demonstrated that DC-CoT reduces the longest path length by approximately 37% on AIME 2024, with negligible accuracy loss, and achieves similar improvements on HMMT 2025. The results validate the effectiveness of the multi-stage RL approach in balancing speed and accuracy, confirming the model’s ability to identify and execute parallel subtasks efficiently.

DC-CoT achieves a 37.4% reduction in longest path length on AIME 2024, with accuracy within 1% of the original long CoT model. In HMMT 2025, the latency decreases by about 40%, demonstrating practical speedups. The multi-stage RL training effectively balances accuracy and latency, outperforming baseline models trained without subtask decomposition. Additional experiments with majority voting further improve accuracy, confirming the robustness of the approach across different settings.

This architecture is suitable for real-time mathematical problem solving, logical reasoning, and decision-making systems where latency is critical. It can be integrated into AI assistants, automated tutoring, and intelligent decision support tools. The ability to identify and execute subtasks in parallel makes it highly scalable for industrial applications requiring high throughput and low response times. Future extensions could include multi-modal reasoning and broader domain adaptation, expanding its utility in complex AI systems.

Despite promising results, the approach depends heavily on the initial reasoning quality, which may limit performance on ambiguous or highly nested tasks. RL training incurs high computational costs, restricting accessibility. The subtask identification process may sometimes produce suboptimal partitions, affecting accuracy. Further research is needed to improve robustness, efficiency, and generalization across diverse problem types.