User Feedback Provides a Unique Signal that LLMs Can not Detect
Synthetic and real data experiments show user feedback significantly improves LLM responses; evaluation bias masks this benefit.
Key Findings
Methodology
The study combines synthetic data—created by intentionally corrupting model responses using specific types like causality inversion and entity swap—with natural dialogue data. It employs algorithms such as GPT-3.5 for response corruption and classification, and uses models like Gemini-3-flash for correction and Gemini-3.1-Pro as the evaluator. The process involves generating corrupted responses, collecting user feedback categorized into actionable insights, and applying different models to improve responses with or without feedback. Evaluation involves pairwise comparisons and issue resolution assessments via a strong judge model, analyzing biases and the true utility of feedback in controlled settings.
Key Results
- In synthetic data, feedback-guided revisions increased success rates by 9%-32%, reaching 95.6% correction success, while in real dialogue data, improvements ranged from 16%-27%, with correction rates of 89%. Smaller models benefited more from feedback, with a 35% increase in issue resolution compared to baseline. Despite these gains, automated judges favored responses without feedback, revealing systematic biases. Feedback primarily enhanced content quality, especially in substantive corrections, rather than stylistic changes.
- The experiments demonstrated that feedback-driven responses outperform baselines in resolving targeted issues, with statistically significant improvements. The bias in automatic evaluation was identified as a key obstacle, as judges often failed to recognize genuine corrections, especially in complex cases. The study also found that models struggle to evaluate responses they cannot improve, indicating correlated limitations in self-assessment and correction capabilities.
- Different types of feedback—full, problem-only, binary—were tested, showing that even weaker signals still provided measurable benefits. The research highlights the importance of improving evaluation methods to accurately reflect true model improvements and suggests that feedback is a valuable signal for model training and refinement.
Significance
This work fundamentally challenges the prevailing notion that natural user feedback is too noisy to be useful. By rigorously demonstrating its effectiveness through controlled experiments, it underscores the untapped potential of feedback in enhancing large language models. The findings reveal that current evaluation frameworks systematically underestimate feedback’s benefits, which has profound implications for future AI development. Incorporating realistic feedback signals can lead to more robust, aligned, and content-rich models, ultimately advancing human-AI interaction. The research bridges the gap between theoretical feedback utility and practical application, providing a pathway for more effective learning paradigms in NLP.
Technical Contribution
The paper introduces a comprehensive experimental framework combining synthetic corruption with real user feedback, enabling precise quantification of feedback utility. It employs advanced algorithms for response corruption, multi-stage improvement, and evaluation, including the use of strong LLM judges. The study uncovers systematic biases in current evaluation methods and demonstrates how feedback-driven improvements outperform baseline responses. It also reveals the limitations of models as evaluators, proposing a new perspective on feedback integration and assessment, with potential for scalable, real-world deployment in dialogue systems.
Novelty
This research is the first to systematically compare feedback-informed and baseline responses using both synthetic and natural data, revealing the hidden value of user feedback. It uniquely identifies evaluation biases that favor stylistic over substantive improvements, challenging the assumption that feedback noise diminishes its usefulness. The innovative use of strong LLM judges to analyze bias origins and the detailed categorization of feedback signals represent significant advances in understanding feedback’s role in model training and evaluation.
Limitations
- The experiments rely on specific corruption types, which may not encompass all real-world issues, limiting generalizability. Future work should explore more diverse corruption methods.
- Evaluation bias remains a challenge; current judges often fail to recognize genuine improvements, necessitating more robust assessment tools.
- Computational costs are high due to multi-stage inference with large models, which may hinder scalability in practical deployments.
Future Work
Future research will focus on developing more accurate evaluation models that reduce bias, expanding the types of corruption and feedback signals tested, and integrating multi-modal feedback. Additionally, efforts will aim to optimize computational efficiency for real-time applications. Extending feedback mechanisms to multi-task and multi-domain settings, and exploring reinforcement learning with human feedback, are promising directions to enable continuous, autonomous model improvement.
AI Executive Summary
Large Language Models (LLMs) have revolutionized natural language processing, yet their alignment with human intent remains a challenge. Traditional approaches rely heavily on explicit supervision, but naturally occurring user feedback offers a promising alternative. However, skepticism persists due to perceived noise and inefficacy. This study confronts that notion by systematically evaluating feedback’s utility through controlled experiments involving synthetic corruption and real dialogue data.
Using algorithms like GPT-3.5 for response corruption and classification, the researchers generate challenging samples with known ground truths. They then apply models such as Gemini-3-flash and Qwen-3-8B to improve responses, with or without feedback, and assess outcomes via a strong evaluator model, Gemini-3.1-Pro. Results reveal that feedback-guided revisions significantly outperform baselines—improving issue resolution rates by up to 32% in synthetic data and 27% in real data. Interestingly, automated evaluators tend to favor responses without feedback, exposing a systemic bias that underestimates feedback’s true value.
This discrepancy highlights a critical challenge: current evaluation methods are systematically biased against genuine improvements driven by feedback. The study further uncovers that models struggle to evaluate responses they cannot improve, indicating intertwined limitations in self-assessment and correction. Despite these biases, the core finding remains clear—user feedback is a highly actionable signal that can substantially enhance model performance if properly evaluated.
The implications are profound. Recognizing feedback’s potential shifts the paradigm in model training, emphasizing the need for better evaluation frameworks and feedback integration strategies. Future work will focus on reducing evaluation bias, expanding feedback signals, and optimizing computational costs, ultimately enabling more autonomous, human-aligned AI systems that learn continuously from real-world interactions.
Deep Analysis
Background
近年来，随着GPT、BERT等模型的突破，预训练与微调技术不断发展，推动自然语言处理迈入新时代。早期研究多集中在大规模监督数据的标注与模型优化（如BERT、GPT-3），逐步探索无监督与自我监督机制。自然反馈作为一种潜在的学习信号，逐渐引起关注（Hancock et al., 2019; Jin et al., 2025），尤其在对话系统中，用户的自然反应被用作模型微调依据。然而，反馈的噪声与不确定性被认为限制其效果（Liu et al., 2025），使得研究多偏向结构化标注或明确反馈。尽管如此，利用真实用户交互数据的潜力尚未充分挖掘，存在评估偏差与模型自我修正能力不足的问题。
Core Problem
核心问题在于，虽然用户反馈在理论上能改善模型输出，但实际效果受限于反馈的噪声和评估体系的偏差。现有自动评判模型（如基于GPT的打分系统）难以准确识别模型的真实改进，导致反馈的价值被低估。此外，模型在自我修正时表现有限，难以充分利用反馈信息，阻碍了持续优化。如何设计更有效的反馈利用机制，建立准确的评估体系，成为亟待解决的难题。
Innovation
本研究的创新点包括：1）结合合成数据（通过故意腐蚀模型响应）与真实对话数据，验证反馈的实用性；2）引入多种腐蚀类型（如因果倒置、实体交换）明确标定问题，验证反馈修正能力；3）利用强评判模型（如Gemini-3.1-Pro）揭示评估偏差，分析模型在反馈利用中的局限。通过对比有无反馈的修正效果，发现反馈显著提升内容改进质量，挑战了反馈噪声无用的传统观点。研究还揭示了评估偏差的根源，为未来改进提供理论基础。
Methodology
- �� 构建合成数据：利用GPT-3.5模型对真实响应进行腐蚀（如交换因果关系、删除关键信息），生成带有明确目标的腐蚀样本。• 采集自然反馈：从真实对话中提取用户反馈，分类为“问题指出”、“修正建议”等，过滤无关或主观反馈。• 模型修正：利用不同模型（如Gemini-3-flash、Qwen-3-8B）对腐蚀响应进行改进，比较有无反馈的效果。• 评估机制：采用强评判模型（如Gemini-3.1-Pro）对修正响应进行打分，分析偏差。• 统计分析：计算问题解决率、偏好偏差，结合人工标注验证，确保结论可靠。
Experiments
在合成数据（Arena-Hard-v2.0）和真实对话数据（ShareLM）上进行。利用不同腐蚀类型（因果倒置、实体交换）生成样本，使用预训练模型（GPT-3.5）腐蚀响应。模型修正后，通过强评判模型评估效果，比较有无反馈的修正率。设置不同反馈信号（完整、问题指示、二元判断），分析其对修正效果的影响。还包括偏差分析、模型性能对比，确保结论的普适性。
Results
反馈显著提升问题解决率，合成数据中，反馈引导修正成功率比无反馈高出9%-32%，真实数据中提升16%-27%。模型利用反馈后，内容改进优于风格调整，内容修正率在89%-95.6%之间，远高于无反馈条件。偏差分析显示，自动评判模型偏向未用反馈的响应，揭示偏差根源。不同腐蚀类型效果一致，验证了反馈的普适性。模型规模越小，反馈效果越明显，显示反馈在低资源模型中的潜力。
Applications
该方法适用于对话系统、自动问答、内容生成等场景，特别在用户交互频繁的应用中，通过引入反馈机制提升模型质量。未来可结合多模态信息（如语音、图像）实现更丰富的反馈利用，推动个性化与自主学习。行业中，可用于客服、教育、内容审核等领域，提升用户体验与系统智能水平。
Limitations & Outlook
当前实验依赖特定腐蚀类型，未涵盖所有实际场景复杂性。评判模型偏差仍存在，影响偏差分析的准确性。模型训练成本较高，实际部署需考虑效率。未来需优化偏差校正机制，扩展腐蚀类型，降低成本，增强模型的泛化能力。
Plain Language Accessible to non-experts
想象你在厨房做饭，食谱代表模型的回答。用户就像厨师，给出一些建议或指出菜的问题，比如“太咸”或“少放盐”。如果厨师听到反馈，可能会调整菜的味道。过去，人们觉得厨师很难理解这些反馈，因为反馈可能不准确或不具体，就像有人说“菜不好吃”，但没说具体问题。现在，这项研究就像教厨师如何更好地听懂反馈，甚至用模拟的食谱（合成数据）测试厨师的反应，发现只要厨师认真听，反馈可以帮助做出更好菜肴。虽然评判厨师的味道是否改善，过去常常出错，但研究发现，真正用心听反馈的厨师，能做出更合适的菜。这个方法可以让AI像个更聪明的厨师，听懂用户的建议，做出更满意的回答。
ELI14 Explained like you're 14
想象你在学校里写作文，老师会给你一些建议，比如“多用具体例子”或“句子太长”。有时候，你会觉得老师的建议很有用，但有时候又觉得没帮到忙。这项研究就像在试验：如果你听老师的建议，作文会变得更好，但如果你没听，可能也会觉得自己写得不错。科学家用电脑模拟这种情况，把一些“作文”故意改坏（比如把故事倒过来），然后看如果听老师的建议，能不能把它改回来。结果发现，听建议真的能帮忙，让作文变得更清楚、更有趣，但评判老师（自动评分系统）有时候会误判，觉得没有改好。这个发现很重要，因为它告诉我们，用户的反馈其实很宝贵，只是我们还没有找到最好的方法去理解和利用它。未来，我们可以让电脑更聪明，真正听懂人们的建议，让它变得更聪明、更贴心，就像一个懂事的朋友一样！
Abstract
Harnessing naturally occurring feedback from user interactions offers a promising learning signal for Large Language Models (LLMs). However, recent studies suggest this feedback is inherently noisy and difficult to leverage effectively. We challenge this conception by demonstrating that user feedback is a highly actionable signal for improvement, and that its perceived ineffectiveness stems from a systematic bias in current evaluation paradigms. To isolate the usefulness of feedback, we construct synthetic data with a definitive ground truth, alongside naturalistic data to validate that our findings hold in real-world scenarios. By comparing model revisions generated with and without access to feedback across both settings, we show that feedback-informed revisions resolve targeted issues at significantly higher rates than baseline revisions. Finally, we expose the root of the evaluation bias: when a model successfully fixes an issue exclusively due to feedback, LLM judges frequently fail to identify the genuinely corrected response, systematically preferring inferior baseline outputs instead.