ACTD: Anchor-Based Cross-Tokenizer Distillation with Residual Regularization

Recent years have seen exponential growth in large language models (LLMs) such as GPT, BERT, and their derivatives, which excel in reasoning and understanding tasks. Early knowledge distillation techniques like Hinton’s soft-label approach (2015) enabled model compression within homogeneous architectures. However, as models diversified with different tokenizers and structures, cross-model distillation became essential. Works like UOT (Boizard et al., 2024) and ALM (Minixhofer et al., 2025) addressed vocabulary mismatch, but still faced challenges in sequence alignment and noise control. With the increasing deployment of heterogeneous models, effective knowledge transfer across different vocabularies and architectures remains a pressing issue. This paper builds on these foundations, proposing a comprehensive framework to improve alignment and noise suppression, thus facilitating robust multi-model distillation.

The core challenge lies in aligning models with different vocabularies and sequence segmentations. Vocabulary mismatch causes distribution distortion, introducing noise into the distillation process. Sequence misalignment further complicates the transfer, especially in reasoning tasks requiring precise token-to-token correspondence. Existing methods often rely on approximate mappings or full distribution matching, which are susceptible to noise and information loss. These issues hinder the effectiveness of knowledge transfer, especially in multi-teacher settings where accumulated errors degrade performance. Addressing these problems requires a systematic alignment mechanism and noise mitigation strategy to ensure high-fidelity knowledge transfer across heterogeneous models.

The paper’s key innovations include: 1) First-subtoken vocabulary alignment, which maps teacher tokens to student tokens based on the initial subtoken, reducing vocabulary mismatch. 2) Character boundary-based sequence alignment, ensuring positional correspondence despite different token segmentations. 3) Anchor-based loss focusing on high-probability tokens, capturing the most informative signals. 4) Residual regularization penalizing low-confidence tokens, suppressing noise and sharpening the student distribution. 5) Multi-teacher extension, aggregating supervision from diverse models to improve robustness. These innovations collectively form a novel framework that effectively addresses heterogeneity and noise issues in cross-model distillation, outperforming prior approaches.

• �� Vocabulary alignment: Define a mapping Γ from teacher vocabulary VT to student vocabulary VS, prioritizing shared tokens and re-tokenizing unshared tokens via character boundary matching. • Sequence alignment: Use character boundary detection to align teacher and student token positions, ensuring accurate correspondence. • Anchor selection: For each position, select top-k teacher tokens with highest probabilities, map them to student tokens as anchors. • Anchor loss: Compute the divergence between teacher’s top-k distribution and student probabilities on anchor tokens, focusing on high-confidence signals. • Residual regularization: For residual tokens outside the anchor set, penalize their total probability mass with L1 norm, suppressing noise. • Multi-teacher extension: Aggregate losses from multiple teachers by averaging, enhancing robustness. • Optimization: Minimize combined anchor loss and residual regularization loss to train the student model.

The evaluation employs OpenThoughts-114k math dataset, comparing ACTD with SFT, KD, ULD, ALM baselines across five reasoning benchmarks (AIME24/25, AMC23, Minerva, MATH500). Models include Qwen-3-1.7B, Llama-3.2-3B, and GLM-4-9B. Hyperparameters like top-k=128, residual weight λres, and sequence truncation are tuned via ablation studies. Performance metrics include AVG@8 and PASS@8, with statistical significance confirmed through multiple runs. Ablation experiments analyze the impact of prefix truncation, residual weight, and top-k size, validating design choices.

ACTD consistently outperforms baseline cross-tokenizer methods, with an average gain of 4.5 points in accuracy across five benchmarks. The multi-teacher version, Multi-ACTD, further improves scores by 4.9 points, demonstrating the benefit of heterogeneous supervision. Ablation results show that removing prefix truncation reduces performance by 0.7–1.6 points, confirming its importance. Residual regularization effectively reduces entropy and sharpens distributions, especially in noisy long-tail scenarios. The optimal top-k value is 128, balancing coverage and noise suppression. Overall, results validate the effectiveness and robustness of the proposed framework.

This framework is directly applicable in deploying lightweight models for complex reasoning tasks in AI-powered applications such as intelligent tutoring systems, automated reasoning engines, and resource-constrained environments. It enables efficient knowledge transfer from large, diverse models, reducing computational costs while maintaining high accuracy. Long-term, the approach can facilitate multi-modal and multilingual model fusion, supporting broader AI deployment in industry, education, and scientific research, especially where model heterogeneity and resource limitations are critical constraints.

The reliance on accurate token boundary detection and vocabulary mapping may limit performance with highly divergent vocabularies or noisy data. Residual regularization might cause information loss in rare or long-tail tokens, impacting domain-specific tasks. The caching of teacher logits and alignment computations increase training overhead, restricting scalability. Future work should focus on adaptive alignment strategies, reducing computational costs, and extending to multi-modal data to enhance robustness and applicability across diverse real-world scenarios.