Detecting RLVR Training Data via Structural Convergence of Reasoning
Introduces Min-kNN distance for black-box detection of RLVR training data via structural convergence, achieving 70% AUC.
Key Findings
Methodology
This study analyzes how RLVR training induces structural convergence in reasoning trajectories. It finds that models trained with RLVR produce more rigid, similar outputs for seen prompts, while unseen prompts maintain diversity. The proposed Min-kNN distance measures the average of the smallest k nearest-neighbor edit distances among multiple generated outputs, capturing the collapse in structural diversity. This black-box metric requires only sampling access, not internal probabilities. Experiments across multiple models (e.g., SimpleRL-32B, DAPO-Qwen-32B) show high detection accuracy (average AUC 0.70), outperforming baseline methods like perplexity and likelihood shift, demonstrating robustness across RL algorithms and model scales.
Key Results
- Across diverse RLVR models, Min-kNN distance achieved an average AUC of 0.70, outperforming baselines by 17%, with consistent performance from 1.5B to 32B parameters.
- The method remains effective under prompt paraphrasing, with only slight AUC drops, indicating strong robustness to input variations.
- Detection performance holds across different RL algorithms (GRPO, PPO) and training setups, including distillation and code/math tasks, confirming broad applicability.
Significance
This work reveals that RLVR training causes a structural collapse in reasoning diversity, which can be exploited for data exposure detection. It addresses the challenge of monitoring training data leaks in RL-based models, where traditional likelihood-based methods fail. By providing a simple, model-agnostic tool, it enhances transparency, security, and trustworthiness of large language models, crucial for safe deployment and regulatory compliance.
Technical Contribution
The core innovation is the Min-kNN distance metric, a purely structural, black-box indicator based on the collapse of reasoning modes. It leverages hierarchical clustering of generated outputs to identify limited reasoning patterns, enabling effective detection without internal model access. This approach generalizes across models, algorithms, and tasks, offering a scalable solution for RLVR data security.
Novelty
This is the first systematic study linking RLVR training to structural convergence in reasoning outputs. The proposed Min-kNN metric uniquely exploits the collapse in reasoning diversity, contrasting with prior work relying on likelihood or memorization features. Its black-box nature and broad applicability mark a significant advancement in RL model security detection.
Limitations
- The method relies on structural similarity, which may be less effective for highly complex or noisy reasoning tasks. Its performance could degrade with adversarial manipulation designed to disrupt structural patterns.
- Computational costs grow with sampling size and output length, posing challenges for very large models or real-time detection scenarios.
- Current focus is on symbolic and algebraic reasoning; extending to multimodal or highly contextual tasks requires further research.
Future Work
Future directions include integrating multi-modal features, developing adaptive thresholds, and exploring defenses against adversarial structural attacks. Extending detection to other RL scenarios and tasks, as well as optimizing computational efficiency, will further enhance practical deployment.
AI Executive Summary
Recent advances in large language models (LLMs) have significantly improved reasoning capabilities, especially with the advent of reinforcement learning with verifiable rewards (RLVR). This technique fine-tunes models to produce correct answers on tasks like mathematics and coding by reinforcing reasoning trajectories that lead to verifiable outcomes. Despite impressive benchmarks, concerns about training data leakage and contamination have grown, especially as many models are released without access to training datasets or checkpoints. Traditional detection methods, which rely on likelihood or memorization signals, falter in RLVR settings because the training objective emphasizes reward-driven, self-generated reasoning paths, leading to a collapse in output diversity for seen prompts.
To address this challenge, the authors propose Min-kNN distance, a simple yet effective black-box metric that quantifies the structural convergence of reasoning outputs. By sampling multiple completions for a prompt and computing the average of the smallest k edit distances among them, the method captures the collapse in reasoning diversity induced by RLVR. Extensive experiments across various models, including SimpleRL-32B and DAPO-Qwen-32B, demonstrate that Min-kNN achieves an average AUC of 0.70, outperforming existing baselines by 17%. The approach is robust across different RL algorithms, model sizes, and prompt paraphrasing, making it practical for real-world deployment.
This work uncovers a fundamental behavioral signature of RLVR training—structural convergence in reasoning trajectories—and leverages it for effective detection. It offers a scalable, model-agnostic tool that enhances transparency and security in RL-based language models. Looking ahead, integrating multi-modal features and defending against adversarial structural attacks are promising directions to further strengthen model safety and trustworthiness, supporting responsible AI deployment in sensitive applications.
Deep Analysis
Background
随着大规模预训练模型的普及，推理能力成为衡量模型性能的关键指标。早期工作如GPT系列、BERT等通过大规模无监督预训练，显著提升了模型的泛化能力。近年来，强化学习（如RLHF、RLVR）被引入优化推理路径，尤其在数学、编码和符号推理任务中表现出色。代表性研究包括Guo等（2025）、Zeng等（2025），推动模型在可验证任务中的应用。然而，训练数据的泄露和污染问题逐渐凸显，尤其在开源模型中，缺乏有效检测工具，导致模型可能“记忆”训练样本，影响泛化。传统检测方法依赖模型内部概率或记忆特征，但在RLVR中，模型通过奖励机制优化推理轨迹，导致输出结构趋于一致，难以用概率指标区分训练样本。近年来，结构分析和邻近距离技术逐渐兴起，但在RL环境中的应用仍处于探索阶段。
Core Problem
核心问题是如何在RLVR训练中识别模型是否暴露于特定训练样本。由于RLVR优化目标不同于最大似然训练，模型输出的推理轨迹表现出明显的结构收敛特征，传统的概率或记忆检测手段失效。现有方法难以捕捉RLVR引起的推理结构变化，亟需一种基于输出结构的检测技术，以保障模型安全和数据隐私。这对于防止训练数据泄露、提升模型可信度具有重要意义。解决方案应能在不访问模型内部参数的前提下，有效识别训练样本的暴露。
Innovation
本研究的创新在于：1）系统性分析RLVR训练引起的推理轨迹结构收敛现象，揭示符号和逻辑部分的收敛特征；2）提出基于邻近编辑距离的黑箱检测指标Min-kNN距离，利用生成内容的结构相似性实现训练数据暴露检测；3）无需访问模型内部概率信息，操作简洁，适用范围广。结合层次聚类分析推理轨迹的有限结构，突破了传统依赖概率的检测限制，为RL环境下模型安全检测提供新思路。
Methodology
- �� 采样：对给定提示多次采样生成内容（如32次），形成输出集合。• 结构分析：计算每对生成内容的归一化Levenshtein编辑距离，衡量结构相似性。• 邻近距离：对每个输出，找到其最近邻的编辑距离，形成邻近距离序列。• 统计指标：取最小k个邻近距离的平均值，作为Min-kNN距离。• 结构聚类：对输出进行层次聚类，识别有限的推理结构模式。• 训练检测：通过阈值判断Min-kNN距离，区分训练样本与未见样本。• 实验验证：在多个RLVR模型上测试，比较AUC性能，验证鲁棒性和泛化能力。
Experiments
采用公开RLVR模型（如SimpleRL-32B、DAPO-Qwen-32B）和控制场景（RL污染检测基准），采集训练样本和未见样本，计算AUC指标。设置采样次数（如32次）、邻近距离参数（k=10），评估不同模型规模和算法的检测效果。进行提示改写、蒸馏等变体测试，验证鲁棒性。对比概率、编辑距离等基线方法，分析检测性能和稳定性。
Results
Min-kNN距离在所有模型中表现优异，平均AUC达0.70，比传统方法提升17%。在不同RL算法（GRPO、PPO）和模型规模（1.5B-32B）中均表现稳定。提示改写后性能仅略有下降，显示鲁棒性强。对多种污染场景（如蒸馏、代码推理）均能有效识别训练样本，验证其广泛适用性。
Applications
该方法适用于模型训练监控、数据泄露检测和模型安全评估。可在开源模型发布前进行安全审查，也可结合部署中的实时检测，防止敏感信息泄露。未来可扩展到多模态任务和对抗攻击场景，具有广阔应用前景。
Limitations & Outlook
目前主要依赖生成内容的结构相似性，对于复杂推理或高变异任务效果有限。对抗性扰动可能削弱检测效果，计算成本在大模型和长推理链中较高。未来需优化采样策略和结合多模态信息，提升鲁棒性和效率。
Plain Language Accessible to non-experts
想象你在一家工厂里，工人们每天都按照固定的流程生产产品。经过一段时间后，他们的工作变得越来越标准化，几乎每次都用一样的方法完成任务。这就像模型在RLVR训练中，经过反复优化后，推理路径变得越来越一致，输出的内容也越来越像模板。这种变化让我们可以通过观察输出的结构，判断它是否经过训练。就像工厂的生产线变得越来越机械化一样，模型的推理也变得越来越刻板。我们用一种叫Min-kNN距离的方法，像工厂检查员一样，测量每次生产的产品有多相似，从而判断它是否来自这个工厂。这个方法简单、快速，不需要知道工厂的内部机器，只看产品的外观。它帮助我们确保模型没有泄露训练数据，也能检测到模型是否被训练过。
ELI14 Explained like you're 14
想象你在学校里，有一台特别的机器人老师。刚开始，它会用很多不同的方法解题，像你平时做作业一样多样化。可是，经过一段时间的训练，这个机器人变得越来越像一个“套路大师”，每次解题都用一样的步骤，甚至用一样的句子。你可以通过观察它的解题步骤，知道它是不是经过特别的训练。我们的方法就像是用放大镜看这个机器人，测量它每次解题的相似程度。如果它的解题都很像，说明它可能经过了特别的训练。这个放大镜不用知道机器内部的秘密，只看它的解答内容就能判断。这样，我们就可以知道这个机器人是不是“学过”那些题，也可以防止它泄露秘密。
Abstract
Reinforcement learning with verifiable rewards (RLVR) is central to training modern reasoning models, but the undisclosed training data raises concerns about benchmark contamination. Unlike pretraining methods, which optimize models using token-level probabilities, RLVR fine-tunes models based on reward feedback from self-generated reasoning trajectories, making conventional likelihood-based detection methods less effective. We show that RLVR induces a distinctive behavioral signature: prompts encountered during RLVR training result in more rigid and similar generations, while unseen prompts retain greater diversity. We introduce Min-$k$NN Distance, a simple black-box detector that quantifies this collapse by sampling multiple completions for a given prompt and computing the average of the $k$ smallest nearest-neighbor edit distances. Min-$k$NN Distance requires no access to the reference model or token probabilities. Experiments across multiple RLVR-trained reasoning models show that Min-$k$NN Distance reliably distinguishes RL-seen examples from unseen ones and outperforms existing membership inference and RL contamination detection baselines.