Compress-Distill: Reasoning Trace Compression for Efficient Knowledge Distillation
Proposes Compress-Distill, compresses reasoning traces to 8.6-21%, reduces training tokens by 12-30%, speeds up training 2-7.6×, retains 96% accuracy.
Key Findings
Methodology
This paper introduces a three-stage pipeline: first, large models generate correct reasoning traces; second, trained compression models (e.g., Ministral-14B, Llama-3.3-70B) perform post-hoc trace compression, reducing length to 8.6-21%; third, student models are trained on raw, compressed, or answer-only targets. Experiments involve two teacher models (Qwen3.5-397B, gpt-oss-120B), multiple student sizes, and various datasets, evaluating downstream accuracy, training cost, inference length, and efficiency. The approach systematically analyzes compression ratios and performance trade-offs across different settings.
Key Results
- Compressed traces reduce training tokens to 12-30% of original, accelerate training by 2.0-7.6×, and shorten inference outputs by 3-19×. Despite efficiency gains, raw traces consistently outperform compressed ones in downstream accuracy, reaching up to 96%. Compression ratios vary by dataset complexity, with longer, redundant reasoning traces being more compressible. In different training regimes, compressed traces improve per-token efficiency by up to 18×, especially benefiting smaller models (0.8B).
- Across both teachers and multiple model sizes, raw traces maintain the highest accuracy, but compression offers a practical trade-off: significant cost reduction with minimal accuracy loss. Length-matched truncation experiments show that compression's benefits are not solely due to token budget reduction; model-based rewriting preserves reasoning quality better than naive truncation.
- In various fine-tuning strategies, compressed traces outperform naive truncation in accuracy at equivalent token budgets, especially for smaller models. The efficiency gains are most pronounced in LoRA-based training, with improvements up to 18× per token, demonstrating the method's scalability and practical relevance.
Significance
This work highlights the potential of reasoning trace compression as a practical tool to balance accuracy and efficiency in large-scale language models. By systematically evaluating post-hoc trace compression, it offers a new paradigm for reducing training and inference costs without substantial performance degradation. The findings have broad implications for deploying powerful models in resource-constrained environments, advancing the state-of-the-art in model compression, and optimizing knowledge distillation workflows for real-world applications.
Technical Contribution
The paper introduces a novel post-processing compression framework that leverages instruction-tuned models to rewrite reasoning traces, significantly shortening them while preserving logical integrity. It combines multiple datasets, teacher models, and training strategies to empirically characterize the trade-offs involved. The approach demonstrates that model-based trace rewriting can outperform naive truncation, especially for smaller models, and provides detailed analysis of compression ratios, training efficiency, and accuracy metrics. This systematic evaluation advances understanding of how trace compression impacts downstream performance and efficiency.
Novelty
This is the first comprehensive study to apply model-based post-hoc trace compression in the context of knowledge distillation for reasoning tasks. Unlike prior work focusing on pruning or length control, this method employs instruction-tuned models to generate shorter, logically consistent traces, enabling significant efficiency gains. The work uniquely combines multiple teacher-student configurations, datasets, and training regimes to analyze the trade-offs, establishing new benchmarks for trace compression effectiveness and its impact on downstream accuracy.
Limitations
- Although compression reduces training tokens and speeds up training, it still results in a slight accuracy drop compared to raw traces, especially at larger scales. The method relies heavily on the quality of the compression model, which may struggle with highly complex or ambiguous reasoning chains.
- Current compression models may discard subtle reasoning steps, leading to potential information loss in certain tasks. The approach also introduces additional preprocessing complexity and computational overhead, which could limit scalability in some scenarios.
- Further research is needed to optimize compression algorithms, adaptively control trace length, and ensure robustness across diverse tasks and datasets. The method's effectiveness diminishes when reasoning traces are inherently dense or low-redundancy, such as in medical or scientific domains.
Future Work
Future directions include developing adaptive compression strategies that dynamically adjust trace length based on task complexity, integrating reinforcement learning to optimize trace rewriting, and extending the approach to multimodal reasoning tasks. Additionally, exploring unsupervised or semi-supervised compression models could further reduce preprocessing costs. Broader validation across more datasets and larger models will help establish the generality and robustness of the method, paving the way for real-world deployment in resource-constrained environments.
AI Executive Summary
The rapid growth of large language models (LLMs) has revolutionized natural language processing, yet their long reasoning chains—while beneficial for complex tasks—pose significant computational challenges. These chains, often comprising hundreds or thousands of tokens, inflate training costs and slow inference, limiting practical deployment. Existing solutions like pruning or length control methods offer partial relief but often sacrifice reasoning fidelity or require complex adjustments.
In response, this study introduces Compress-Distill, a novel pipeline that employs post-hoc model-based trace compression to address these issues. The core idea is to generate high-quality reasoning traces from large teacher models, then use instruction-tuned compression models—such as Ministral-14B and Llama-3.3-70B—to rewrite these traces into shorter, logically consistent versions. This process reduces the token length to 8.6-21% of the original, cutting training tokens to 12-30%, and accelerating training by up to 7.6×. Simultaneously, inference outputs are shortened by up to 19×, significantly improving efficiency.
Extensive experiments across multiple datasets, models, and training strategies demonstrate that compressed traces, while slightly less accurate than raw traces, retain up to 96% of the original performance. Notably, compressed traces outperform naive truncation, especially for smaller models, by better preserving reasoning quality. The results highlight a practical accuracy-efficiency trade-off: substantial cost savings with minimal accuracy loss.
This work advances the understanding of trace compression in knowledge distillation, providing a scalable, flexible approach to deploying large models efficiently. It opens avenues for adaptive compression, multimodal reasoning, and real-world applications where resource constraints are critical. Despite some limitations—such as potential information loss in dense reasoning—this method offers a promising direction for balancing performance and practicality in AI systems.
Deep Dive
Limitations & Outlook
What gaps remain?
Abstract
Reasoning models produce long chain-of-thought traces that are costly to distill and encourage verbose student outputs. We study post-hoc compression of such traces before knowledge distillation. Two teachers, Qwen3.5-397B-A17B and gpt-oss-120B, generate about 283k correct traces each; two instruction-tuned models then compress them to 8.6-21.0% of their original character length. Across a 48-run main grid plus seven Qwen-teacher truncation ablations, compressed traces reduce training tokens to 12-30% of raw, speed up training by 2.0-7.6x, and shorten inference outputs by 3-19x with smaller reductions under the shorter gpt-oss teacher. However, raw traces retain the highest downstream accuracy at every scale and for both teachers. A length-matched raw-trace truncation ablation shows that compression is not merely benefiting from a smaller token budget: model-compressed traces usually beat or match naive truncation, especially for smaller students, while maintaining shorter inference outputs. Overall, reasoning-trace compression offers an accuracy-efficiency trade-off rather than a free improvement: students retain up to 96% of raw-trace accuracy while gaining up to 18x higher per-token efficiency, and at the 0.8B scale under LoRA compressed traces narrow the raw-vs-compressed gap but do not exceed raw.