Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks
RTT framework maps response-level scores to token rewards via a Token Relevance Discriminator, improving instruction-following accuracy by 2.5% over baselines.
Key Findings
Methodology
RTT introduces a Token-Level Relevance Discriminator to predict token responsibility for specific constraints, translating coarse response scores into fine-grained token rewards. It employs RTT-GRPO, combining response-level and token-level advantages, and addresses multi-dimensional reward normalization with Intra-sample Token Group Normalization. Experiments on multiple datasets validate its effectiveness, outperforming baselines in instruction and rubric accuracy. The framework integrates structured rubric scores with token-level credit assignment, enabling precise policy optimization in complex tasks.
Key Results
- Across multiple instruction-following benchmarks, RTT-CSR and RTT-AON achieved an average improvement of 2.5% in instruction accuracy and 1.6% in rubric-level satisfaction over RL-AON and RL-CSR baselines. In out-of-domain tests, RTT maintained or improved performance, demonstrating robustness. Ablation studies confirmed that Intra-sample normalization outperforms Inter-sample by approximately 1.2%, highlighting the importance of normalization strategy. The method's ability to assign rewards at the token level significantly enhances training efficiency and model performance.
- Model evaluations on Qwen3-4B and Llama3.2-3B show consistent gains, validating the approach's generality.
Significance
This work advances reinforcement learning for large language models by enabling fine-grained, multi-dimensional reward signals aligned with structured rubrics. It addresses longstanding issues of reward sparsity and ambiguity, facilitating more precise behavior control and better generalization across tasks. The approach opens new avenues for complex instruction adherence, multi-constraint satisfaction, and structured evaluation in AI systems, with broad implications for deploying safer, more reliable models in real-world applications.
Technical Contribution
The paper proposes a novel framework combining a Token-Level Relevance Discriminator with multi-dimensional reward normalization, integrated into RTT-GRPO for policy optimization. It introduces Intra-sample Token Group Normalization to handle multi-dimensional reward spaces, solving the group partitioning problem. These innovations enable effective, fine-grained credit assignment, improving training stability and performance. The method's design allows seamless integration with existing RL algorithms and demonstrates scalability across different models and tasks.
Novelty
This is the first work to incorporate multi-dimensional, structured rubric-based rewards into RL training through token-level credit assignment. It uniquely combines a Token Relevance Discriminator with a normalization strategy tailored for high-dimensional reward spaces, addressing reward sparsity and ambiguity issues that previous methods could not solve. The approach significantly enhances the granularity and effectiveness of reward signals, setting a new standard for structured reinforcement learning in language models.
Limitations
- The quality of token-level rewards heavily depends on the accuracy of the Token Relevance Discriminator, which may struggle in highly ambiguous or complex tasks, potentially leading to noisy signals. The computational overhead for training and inference increases due to additional discriminator and normalization steps, limiting scalability. In scenarios with very sparse or vague constraints, the discriminator's predictions may be unreliable, affecting reward quality and training stability.
Future Work
Future directions include developing more robust, generalizable token relevance models, exploring multi-modal multi-dimensional reward spaces, and optimizing normalization techniques for large-scale deployment. Additionally, integrating human-in-the-loop feedback and unsupervised discriminator training could further improve reward accuracy. Extending the framework to multi-turn dialogues and multi-modal tasks will broaden its applicability, aiming for more autonomous, adaptable AI systems capable of complex multi-constraint reasoning.
AI Executive Summary
In recent years, large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding and generation. However, guiding these models to follow complex, multi-constraint instructions remains a challenge. Traditional reinforcement learning approaches rely on response-level rewards, which are often sparse and coarse, limiting the models' ability to learn nuanced behaviors. This paper introduces the RTT (Rubrics to Tokens) framework, a novel approach that bridges the gap between coarse response scores and fine-grained token-level credit assignment.
RTT leverages a Token-Level Relevance Discriminator to predict which tokens in a response are responsible for satisfying or violating specific constraints. This enables the translation of structured rubric scores into detailed token rewards. The framework further employs RTT-GRPO, an adaptation of the Group Relative Policy Optimization algorithm, to jointly optimize response-level and token-level advantages. A key innovation is the Intra-sample Token Group Normalization method, which addresses the challenge of normalizing multi-dimensional reward spaces, ensuring stable and effective training.
Extensive experiments across multiple datasets and models demonstrate that RTT consistently outperforms existing baselines in instruction-following benchmarks. The results show an average improvement of over 2.5% in instruction accuracy and 1.6% in rubric satisfaction, validating the effectiveness of fine-grained reward signals. Notably, RTT maintains strong out-of-domain performance, indicating good generalization. The approach offers a significant step forward in structured reinforcement learning, enabling models to better understand and adhere to complex, multi-faceted instructions.
This work has broad implications for AI deployment in real-world scenarios, where nuanced behavior and multi-constraint satisfaction are critical. By providing a detailed reward signal at the token level, RTT enhances model interpretability, training efficiency, and robustness. Future work will explore multi-modal extensions, improved normalization strategies, and unsupervised discriminator training to further advance the field. Overall, RTT marks a substantial contribution to the development of more intelligent, controllable language models.
Deep Analysis
Background
近年来，随着GPT、LLaMA等大规模语言模型的快速发展，指令遵循能力成为衡量模型智能水平的重要指标。早期方法主要依赖监督微调（SFT），通过高质量数据引导模型学习行为。随后，RLHF（强化学习结合人类偏好）逐渐成为主流，结合人类反馈优化模型表现。近年来，结构化评价体系（rubrics）被引入，用于多维度评价模型输出，提升模型的行为细粒度控制能力。代表性工作包括硬约束的二值奖励、聚合式奖励以及多任务、多场景的评价体系。这些方法在一定程度上解决了模型泛化和复杂任务适应的问题，但仍存在奖励稀疏、模糊和归一化难题，限制了其进一步提升。
Core Problem
当前的Rubric-based RL多依赖响应级别的单一评分，导致奖励信号稀疏且模糊，难以提供细粒度的行为引导。尤其在多约束、多目标场景中，响应评分的模糊性使得模型难以明确哪些Token贡献了成功或失败，影响训练效率和效果。此外，奖励空间的多维结构未被充分利用，归一化策略难以适应不同尺度，导致训练不稳定。这些问题限制了模型在复杂任务中的表现和泛化能力，亟需一种更细粒度、更结构化的奖励机制。
Innovation
本研究提出RTT框架，核心创新在于：
- �� 引入Token-Level Relevance Discriminator，将响应级评分映射到Token层面，实现细粒度奖励分配。
- �� 设计Intra-sample Token Group Normalization，有效解决多维奖励空间中的归一化边界定义和长度偏差问题。
- �� 将多维奖励空间与RTT-GRPO结合，优化模型行为，提升指令遵循和规则满足能力。
- �� 在多个模型和任务上验证，显示其优越性。这些创新突破了奖励稀疏和模糊的限制，为大模型RL训练提供新工具。
Methodology
- �� 训练Token相关判别器：利用多源数据（如HiR-16K）和负样本策略（最小修改、约束遗漏）训练判别器，预测每个Token的约束相关性。
- �� 构建多维奖励空间：将响应评分映射到Token级别，结合判别器输出，生成Token奖励。
- �� 归一化策略设计：采用Intra-sample Token Group Normalization，按单个响应内的Token集合进行标准化，避免长度偏差。
- �� 联合优化：结合响应级和Token级优势，利用RTT-GRPO进行模型参数更新。
- �� 训练流程：采样多响应组，计算奖励，归一化优势，反复迭代，确保多维奖励的有效利用。
Experiments
采用HiR-16K等多源数据集进行训练，评估在多项指令遵循和Out-of-Domain任务上的表现。比较基线包括SFT、DPO、RL-AON和RL-CSR。指标涵盖指令级准确率、Rubric满足率。超参数设置包括α=1、β=0.5，模型包括Qwen和Llama系列。进行消融实验验证归一化策略效果，分析不同模型和任务的性能差异。实验结果显示，RTT在多个指标上均优于对比方法，验证其有效性和普适性。
Results
在指令遵循基准（如IFEval、IFBench）上，RTT-CSR和RTT-AON平均提升指令级准确率2.5%，Rubric满足率提升1.6%。在复杂多约束任务中，模型表现显著改善，尤其在多任务、多场景环境中展现出优越的泛化能力。 Ablation研究表明，Intra-sample归一化策略比Inter-sample更稳定、更有效，提升约1.2%。此外，RTT在Out-of-Domain任务中表现稳定，验证其鲁棒性。整体来看，细粒度Token奖励显著缓解了奖励稀疏和模糊问题，提升了训练效率和模型性能。
Applications
该方法适用于需要多维结构化评价的开放域任务，如智能问答、内容生成、自动摘要等。通过引入Token级奖励，模型能更准确理解和遵循复杂指令，提升用户体验。未来可结合多模态信息，扩展到视觉、语音等多模态场景，推动AI在多任务、多场景中的自主学习能力。
Limitations & Outlook
模型依赖高质量判别器训练，判别器性能不足会影响奖励质量。复杂任务中，结构化评分和Token判别难以准确反映行为，可能导致训练偏差。计算成本较高，限制大规模应用。未来需优化判别器泛化能力和归一化机制，降低成本，提升实用性。
Plain Language Accessible to non-experts
想象你在厨房里做菜，食谱上写着各种要求，比如“不要放太咸”、“用新鲜的蔬菜”。以前，厨师只看最后的菜好不好，评分要么全对，要么全错，很难知道哪一步出了问题。现在，RTT就像在每个步骤都贴标签，告诉你哪些动作让菜变得更好，哪些需要改进。这样，你可以逐步调整每个环节，而不是只靠最后的结果判断。它把复杂的菜谱拆成很多小部分，让厨师（模型）知道每个动作的好坏，做得更细、更合理。这个方法让厨师学得更快，也做出更符合要求的菜肴。
ELI14 Explained like you're 14
想象你在学校做科学实验，老师给你一个复杂的任务，比如“做一个环保的火箭”。以前，老师只会给你一个最终评分，要么你成功了，要么失败了。这样你不知道哪一步出了问题，也难以改进。现在，这个新方法就像老师在每个步骤都打分，告诉你哪些部分做得好，哪些需要改进。比如，火箭的设计、材料选择、点火方式，每个都被单独评价。这样，你可以针对每个部分改进，而不是只看最后的成败。它就像给你一份详细的反馈，让你变成更厉害的科学家。这个方法让学习变得更有方向，也能做出更环保、更安全的火箭！
Abstract
Rubric-based Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models (LLMs) with complex, open-domain instruction following tasks. However, existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems. To address these issues, we propose Rubrics to Tokens (RTT), a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained token-level credit assignment. RTT introduces a Token-Level Relevance Discriminator to predict which tokens in the response are responsible for a specific constraint, and optimizes the policy model via RTT-GRPO, which integrates response-level and token-level advantages within a unified framework. Furthermore, when transitioning from one-dimensional, outcome-level reward to three-dimensional reward space in the token-level rubric-based RL, we propose a novel group normalization method, called Intra-sample Token Group Normalization, to accommodate this shift. Extensive experiments and benchmarks demonstrate that RTT consistently outperforms other baselines in both instruction- and rubric-level accuracy across different models.