Fine-Grained Human Feedback Gives Better Rewards for Language Model Training

Recent advances in pre-trained language models like GPT-3 and T5 have dramatically improved natural language generation capabilities. Early efforts focused on supervised fine-tuning, but issues such as toxicity, misinformation, and incoherent outputs persisted. Reinforcement learning from human feedback (RLHF) emerged as a solution, where human preferences guide model optimization. However, most existing RLHF methods rely on sequence-level preference signals, which are sparse and lack error localization. As models are deployed in sensitive domains, the need for more precise control over generated errors has become urgent. Researchers have begun exploring multi-category feedback and dense reward signals, but systematic frameworks integrating these ideas remain limited. This paper builds on prior work by proposing a multi-category, multi-density reward framework, aiming to improve safety, factuality, and controllability in language generation.

The core challenge addressed is the inefficiency and limited granularity of traditional RLHF methods, which provide only a single scalar preference score at the sequence level. This hampers the model's ability to identify and correct specific errors such as falsehoods, irrelevance, or information gaps. Moreover, the sparse reward signals lead to slow learning and suboptimal correction of undesirable behaviors, especially in long-form text generation. The difficulty lies in designing reward mechanisms that can provide dense, category-specific feedback at appropriate granularities, enabling targeted error correction and more effective training. Overcoming these limitations is crucial for deploying language models in real-world applications requiring high safety and factual accuracy.

The paper introduces a multi-category, multi-density reward framework that segments generated text into sub-sentences, sentences, or entire sequences, training dedicated reward models for each error type. These models, based on Longformer encoders, predict category-specific rewards at different granularities, which are then combined during RL training via weighted sums (formula 1). This approach allows the model to receive detailed feedback about specific errors, facilitating targeted corrections. Additionally, the framework supports flexible reward weighting, enabling personalized behavior tuning. The integration of multiple reward models within PPO optimizes the policy towards safer, more accurate outputs, representing a significant step beyond prior single-reward, sequence-level methods.

• �� Define language generation as a Markov Decision Process (MDP), where each token is an action.
• �� Collect human feedback by annotating errors in generated outputs, categorizing into C1 (irrelevance/incoherence), C2 (factual errors), and C3 (incompleteness), at sub-sentence, sentence, and full sequence levels.
• �� Train separate reward models Rϕ1, Rϕ2, Rϕ3 using Longformer-base, each focusing on one error category and density.
• �� Segment generated text according to categories, compute category-specific rewards for each segment.
• �� Combine rewards using weighted sum (formula 1), with weights w1, w2, w3.
• �� Use PPO to optimize the language policy, maximizing the combined reward signal.
• �� Evaluate on detoxification and long-form QA tasks, comparing against holistic reward baselines.

The experiments involve training models on REALTOXICITYPROMPTS for toxicity filtering and QA-FEEDBACK for long-form QA. In toxicity tasks, dense sentence-level rewards led to toxicity scores of 0.081, outperforming holistic RLHF (0.130). Training efficiency improved by 30%. In QA, errors across relevance, factuality, and completeness were reduced by over 20%, validated through automatic metrics and human annotations. Multiple reward models were combined with adjustable weights, demonstrating controllability. Hyperparameters included PPO with top-k sampling (k=20), temperature 0.7, and reward weights w1=0.3, w2=0.5, w3=0.3. Results consistently showed superior performance over baseline methods.

Dense, multi-category rewards significantly improve safety and factuality, reducing toxicity scores and factual errors faster and more effectively than sequence-level rewards. The models trained with this framework achieve lower toxicity, higher relevance, and more complete information, validated across automatic metrics and human judgments. The ability to tune reward weights offers customizable behaviors, making the approach adaptable to different application needs. These findings demonstrate the potential of fine-grained reward signals to enhance the controllability and reliability of large language models.

This framework is applicable to content moderation, dialogue safety, and factual question answering, especially where precise error correction and safety are critical. It enables developers to incorporate domain-specific error categories and adjust model behavior according to user preferences. The approach can be extended to multi-modal generation and multi-task scenarios, supporting safer and more aligned AI systems in real-world deployments.

The increased computational cost from training multiple reward models poses scalability challenges. Human annotations for error categorization are subjective, potentially affecting consistency. The current validation is limited to toxicity and QA tasks; broader applicability needs further testing. Future work should focus on automating feedback collection, reducing annotation bias, and integrating multi-modal rewards for comprehensive control.