Acquire, Repair, Preserve: A Diagnosis-Guided Post-Training Recipe for Small-Model Dialogue Game Agents
This study introduces a diagnosis-guided post-training approach—Acquire, Repair, Preserve—that boosts small dialogue agents' scores from 10.67 to 38.92, focusing on local decision failures.
Key Findings
Methodology
The approach involves three stages: first, broad supervised fine-tuning (SFT) to enable the model to participate in diverse dialogue games; second, diagnosing local failures—such as repeated guesses or malformed actions—in specific game families like Wordle, and repairing them using turn-local preference pairs with DPO (Direct Preference Optimization); third, applying delta scaling in weight space to balance performance and capability preservation. The method leverages LoRA for parameter-efficient fine-tuning, constructs preference pairs based on mechanically verifiable failures, and employs a model-agnostic approach to repair localized decision errors without degrading general abilities.
Key Results
- The final model achieved a public clemscore of 38.92, up from 10.67, and a closed in-domain score of 41.17, from 13.41, with static performance remaining nearly unchanged (44.14 vs. 44.24). Out-of-domain clemscore increased from 3.72 to 7.88, indicating effective transfer within related variants. The approach significantly improved interaction quality while maintaining broad capabilities.
- Local decision failures, such as repeated guesses and malformed actions in Wordle, were precisely diagnosed and corrected via turn-local preference pairs, leading to substantial performance gains in targeted game families. The use of DPO at decision points proved more effective than whole-dialogue preference tuning, which often disrupted protocol compliance.
- Weight scaling (delta scaling) enabled the model to retain static performance while enhancing interactive scores, demonstrating a practical method for capability preservation in resource-constrained models. The results highlight the importance of targeted supervision and local repair in small models for complex interactive tasks.
Significance
This work demonstrates that small models can achieve substantial improvements in interactive dialogue tasks through targeted diagnosis and repair, emphasizing the critical role of local decision correction over broad knowledge enhancement. It offers a scalable, low-cost framework for deploying reliable conversational agents, especially in resource-limited settings. The findings challenge the notion that large-scale models are necessary for high-quality interaction, showing instead that strategic supervision and repair can unlock significant performance gains. This approach has broad implications for developing robust, efficient AI assistants capable of nuanced, context-aware dialogues in real-world applications.
Technical Contribution
The core technical innovation lies in integrating fault diagnosis with preference-based local repair, employing DPO for decision-level optimization, and using weight-space delta scaling for capability preservation. Unlike traditional fine-tuning or reinforcement learning, this method focuses on mechanically verifiable failures, enabling precise correction without extensive retraining. The use of LoRA allows efficient adaptation with minimal parameter overhead, making the approach suitable for small models. The combination of diagnosis, preference learning, and weight scaling provides a novel, effective pipeline for improving interactive performance while maintaining general capabilities.
Novelty
This research is the first to systematically combine fault diagnosis, local preference learning, and weight-space adjustment in a unified framework for small models in dialogue tasks. Its emphasis on mechanically verifiable failures and targeted repair distinguishes it from prior works that rely on global fine-tuning or reinforcement learning, offering a practical, scalable solution for enhancing specific decision points without sacrificing overall performance. The approach shifts the paradigm from broad knowledge enhancement to precise, localized correction, representing a significant innovation in model fine-tuning strategies.
Limitations
- The repair mechanism depends on explicit, mechanically verifiable failure scenarios, limiting its applicability to tasks where such failures are well-defined, such as Wordle. Handling semantic or strategic errors remains challenging.
- Transferability beyond related variants within the same family (e.g., Wordle variants) is limited; out-of-domain performance gains are modest, indicating the need for more general diagnostic and repair strategies.
- Weight scaling, while effective, lacks formal theoretical guarantees and may risk degrading overall model capabilities if not carefully tuned. Further research is needed to develop multi-objective optimization methods for capability preservation.
Future Work
Future directions include extending fault diagnosis to encompass semantic and strategic failures, developing more generalizable repair mechanisms, and integrating multi-task learning to improve out-of-domain transfer. Additionally, automating the selection of optimal weight scaling parameters and exploring adaptive, dynamic repair strategies could further enhance robustness. Investigating the combination of local repair with global fine-tuning or reinforcement learning may yield models with both high specificity and broad generalization, advancing the deployment of reliable small-scale dialogue agents.
AI Executive Summary
This research addresses a critical challenge in deploying small-scale dialogue agents: achieving high-quality, reliable interaction without sacrificing general capabilities. Traditional approaches rely heavily on large models or extensive retraining, which are costly and often impractical. The authors propose a novel diagnosis-guided post-training framework—Acquire, Repair, Preserve—that leverages targeted supervision, local decision repair, and weight-space adjustment to substantially improve performance.
The first stage involves broad supervised fine-tuning (SFT), enabling the model to participate in diverse dialogue games. This creates a foundation for further refinement. Next, the focus shifts to diagnosing local failures—such as repeated guesses, malformed actions, and feedback violations—in specific game families like Wordle. These failures are mechanically verifiable, allowing precise construction of preference pairs. Using DPO, the model's decision-making at failure points is optimized, correcting errors without disrupting overall abilities.
Finally, the model undergoes a weight scaling process—delta scaling—that balances interactive gains with static performance preservation. This step ensures that improvements in dialogue quality do not come at the expense of general knowledge or static benchmarks. Experimental results demonstrate a dramatic score increase—from 10.67 to 38.92 in public clemscore—while static performance remains stable.
The approach's strength lies in its targeted, efficient repair strategy, which effectively transfers within related variants, especially in the Wordle family. Although out-of-domain transfer remains limited, the method offers a scalable, low-cost pathway for enhancing small models in interactive settings. Future work aims to extend diagnosis to more complex failure types and improve out-of-domain robustness, paving the way for practical, reliable AI assistants in resource-constrained environments.
Deep Analysis
Background
近年来，预训练语言模型（如GPT、BERT）在自然语言处理领域取得巨大突破，推动多任务、多场景应用。然而，面对需要持续状态维护和反馈解释的对话游戏，静态评测难以反映模型的动态交互能力。现有研究多关注模型的知识覆盖和生成质量，缺乏对局部决策错误的系统诊断与修复方法。Playpen和clembench平台推动了对话游戏的评估框架，强调模型在动态环境中的表现，但仍存在模型在特定任务中反复犯错、格式或策略性失误的问题。尤其在小模型（如2B参数）中，这些问题更为突出，严重影响其实用性。此前的工作多采用全局微调或强化学习优化，成本高且难以控制模型能力的保持。因此，如何在有限参数下实现高效、精准的局部修复，成为研究热点。
Core Problem
核心问题在于小模型在对话游戏中的局部决策频繁出错，如重复猜测、格式错误、反馈违反等机械可验证的错误。这些错误虽不影响模型的知识面，但严重阻碍其在特定任务中的表现。传统微调难以针对这些局部错误进行有效修复，且全局优化可能破坏模型的通用能力。如何在有限参数下实现高效、精准的局部修复，提升模型交互质量，成为亟待解决的问题。此外，模型迁移能力有限，难以在未见变体中保持性能，限制了实际应用推广。
Innovation
本研究的创新点包括：1）提出基于故障诊断的偏好对（Preference Pairs）构建机制，实现机械可验证的局部修复；2）结合LoRA微调技术，提升参数效率和训练速度；3）利用DPO（Rafailov et al., 2023）在决策层面进行偏好优化，确保修复目标的局部性和有效性；4）引入权重空间缩放（delta scaling）策略，在保持静态性能的同时，平衡交互能力，避免过度调整带来的负面影响。这一体系突破了传统全局微调的局限，为小模型的交互优化提供了低成本、可控的解决方案。
Methodology
- �� 通过监督微调（SFT）在多样化对话数据中训练模型，使其参与广泛的对话场景。• 利用诊断机制识别模型在Wordle等游戏中的局部决策错误，如重复猜测、格式不符。• 构建偏好对（preference pairs），在模型产生错误后，选取成功对话中的对应决策作为偏好目标。• 采用DPO（Rafailov et al., 2023）在局部决策点进行偏好优化，调整模型输出概率分布，修正错误行为。• 在修复后，通过权重缩放（delta scaling）在模型参数空间中选择最优平衡点，确保模型在交互性能与静态能力间取得折中。• 最终，将修正的模型参数进行融合，获得既具修复能力又保持能力的模型。
Experiments
- �� 使用Playpen平台中的14个对话游戏，包含Wordle、Codenames等，评估模型的交互能力。• 训练数据来自playpen-data的成功与失败对话，采用不同阶段的微调策略。• 采用clemscore和statscore作为主要指标，评估模型的交互表现和静态能力。• 设计偏好对构建机制，针对Wordle中的局部故障生成偏好对，进行两轮偏好优化。• 通过不同的训练阶段（广泛SFT、局部修复、能力保持）逐步提升模型性能。• 实验中还包括偏好对的消融分析，验证局部偏好学习的效果，及不同参数缩放比例的影响。
Results
- �� 经过多阶段训练，模型的公共clemscore从10.67提升至38.92，闭域得分从13.41提升至41.17，静态性能几乎不变（44.14对比44.24）。• 在未见变体中，OOD clemscore由3.72提升至7.88，显示局部修复具有良好的迁移能力。• 局部偏好对显著改善了模型在Wordle中的局部决策错误，特别是重复猜测和格式问题，验证了偏好学习的有效性。• delta scaling策略成功在保持静态性能的基础上，显著提升交互表现，验证了模型能力调节的可行性。
Applications
- �� 该方法适用于需要高效交互的智能助手、游戏AI等场景，尤其在资源有限的小模型中表现优异。• 未来可扩展至多任务、多场景的对话系统，通过局部修复提升模型的鲁棒性和用户体验。• 还可结合自动故障诊断与偏好学习，构建自我修复的智能系统，适应复杂动态环境。
Limitations & Outlook
- �� 依赖机械可验证的故障场景，难以应对语义或策略性错误，限制了应用范围。• 迁移能力主要局限于相关变体，超出相关家族的泛化仍不足。• 权重缩放策略虽有效，但缺乏理论保证，可能在极端调整下影响模型整体性能。未来需结合多目标优化与更丰富的故障检测机制。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂里有很多机器，每个机器都负责生产不同的产品。有时候，某台机器会出错，比如重复生产、装错零件或者操作不符合规定。这些错误虽然不影响整个工厂的基本运作，但会降低效率甚至导致产品不合格。工厂的工程师会用专门的工具检测出这些错误，然后告诉机器怎么改正。经过多次调整后，机器变得更聪明，能自己避免这些错误。这个过程就像我们让模型学习对话游戏一样：先让它广泛参与（类似工厂的培训），再针对具体问题（如Wordle中的重复猜测）进行修正（偏好对），最后用调节参数的方法让它在保持整体能力的同时，专注于改正错误。这种方法让模型变得更聪明、更可靠，就像工厂里的机器一样。
ELI14 Explained like you're 14
想象你在学校里参加一个游戏比赛，比赛规则很复杂，你要不断猜答案、听老师的反馈，然后做出正确的动作。有时候，你会反复猜错，比如猜了两次一样的答案，或者没有按照老师的提示行动。这些错误让你得分变低，但其实你已经知道了很多规则，只是偶尔会犯一些小错误。科学家们发现，要让模型变得更聪明，不是让它学会所有的知识，而是告诉它在哪些地方容易出错，然后帮它改正。就像你在游戏中遇到问题时，老师会告诉你哪里错了，然后你自己调整策略。通过不断检测错误、修正偏好，再用一些调节技巧，模型可以在特定任务中表现得更好。这样，模型就像一个聪明的学生，既懂得规则，又知道怎么避免犯错，变得更厉害了！
Abstract
Interactive dialogue games test a capability that static benchmarks largely leave implicit: a model must carry state across turns, interpret feedback, and choose valid actions under changing constraints. We study this setting in the LM Playschool Challenge with a 2B open-weight model, and find that many failures are not only broad knowledge failures but also local decision failures: repeated guesses, malformed actions, and violations of feedback that the model has just seen. These diagnostics motivate a training recipe organized around three steps: acquire broad game participation through supervised fine-tuning, repair mechanically verifiable failures within one targeted dialogue-game family using turn-local preference pairs, and preserve general capabilities beyond these dialogue games. In the official final evaluation, our submission improves public clemscore from 10.67 to 38.92 and closed in-domain score from 13.41 to 41.17, while approximately preserving aggregate static performance (44.14 vs. 44.24 for the baseline). Out-of-domain clemscore remains low at 7.88, with the largest gains concentrated in unseen variants of the targeted family. Our results suggest that broad SFT brings most of the model's capability improvement; turn-local supervision can be effective when failure detection is precise, with observed transfer concentrated primarily within-family.