Auditing Pay-Per-Token in Large Language Models
A martingale-based sequential audit framework detects token misreporting in LLMs with fewer than 70 samples, controlling false positives below 5%.
Key Findings
Methodology
This paper introduces a martingale-based sequential testing framework that leverages an unbiased estimator of token sequence length conditioned on model next-token probabilities. The core algorithm constructs a martingale process by accumulating evidence from reported token sequences and their estimated lengths, ensuring detection guarantees regardless of the provider’s misreporting strategy. The estimator uses a modified generation process to produce unbiased length estimates, enabling the statistical test to distinguish truthful from unfaithful providers efficiently. Experiments on Llama, Gemma, and Ministral models demonstrate detection within 70 outputs, with false positive rate below 0.05.
Key Results
- The framework reliably detects unfaithful providers with fewer than 70 samples, outperforming prior methods requiring larger sample sizes.
- Experimental results across multiple models and misreporting policies show high detection accuracy and robustness, with false alarms below 5%.
- The method generalizes well to different tokenization strategies and model architectures, confirming its practical utility.
Significance
This work addresses a critical challenge in AI service economics—ensuring truthful token reporting in pay-per-token models. By providing a rigorous, provably reliable auditing tool, it enhances transparency and fairness in cloud-based LLM deployment. The approach mitigates economic incentives for misreporting, fostering trust among users and regulators, and setting a foundation for standardized model accountability.
Technical Contribution
The key innovation lies in applying martingale theory to construct a sequential, statistically sound test that guarantees detection of any misreporting policy. The unbiased length estimator, combined with adaptive weighting parameters, ensures efficiency and robustness. Theoretical proofs establish false positive control and finite detection time under broad conditions, representing a significant advance over existing static or heuristic methods.
Novelty
This is the first application of martingale-based sequential hypothesis testing to the problem of token misreporting in large language models. The integration of an unbiased length estimator with a dynamic evidence accumulation process provides a novel, rigorous framework that handles arbitrary misreporting strategies, surpassing prior content or behavior-based detection approaches.
Limitations
- The method relies on access to the model’s next-token probability distribution; if unavailable or inaccurate, detection performance may degrade.
- Detection speed can be affected by complex or adaptive misreporting strategies, requiring further optimization.
- Current focus is on length-based misreporting; other bias types like semantic or content manipulation are not directly addressed.
Future Work
Future directions include extending the framework to multi-faceted bias detection, integrating semantic consistency checks, and developing adaptive algorithms to handle more sophisticated misreporting strategies. Additionally, exploring black-box scenarios with limited model information and reducing computational overhead are promising research avenues.
AI Executive Summary
The rapid adoption of large language models (LLMs) in cloud services has introduced new economic incentives that may compromise transparency. Providers often charge based on token counts, creating a motive to misreport token usage to maximize profits. Recent studies (Velasco et al., 2025) have highlighted the risk of strategic misreporting, which can undermine user trust and market fairness. Traditional detection methods, relying on content analysis or internal model inspection, fall short when providers adopt complex, length-based misreporting strategies. To address this, the authors propose a novel auditing framework grounded in martingale theory, designed to detect any form of token misreporting with high confidence and minimal samples.
The core idea involves constructing a sequential statistical test that monitors the reported token lengths against an unbiased estimate derived from the model’s next-token probabilities. This estimator is obtained via a modified generation process that ensures unbiasedness, even when the provider employs arbitrary misreporting policies. The test accumulates evidence over multiple interactions, represented as a martingale process, which guarantees that, under honest reporting, the evidence remains stable, while deviations indicating misreporting cause the process to grow rapidly.
Extensive experiments on models from Llama, Gemma, and Ministral families validate the framework’s effectiveness. The results show that unfaithful providers can be detected after observing fewer than 70 outputs, with false alarms kept below 5%. This demonstrates a significant improvement over existing static or content-based detection methods, offering a practical tool for regulators, platform operators, and developers to ensure model integrity. Looking ahead, the authors plan to extend the framework to cover other bias types, optimize detection speed, and adapt to black-box models, aiming to foster a more transparent and trustworthy AI ecosystem.
Deep Analysis
Background
近年来，大规模语言模型（LLMs）如GPT、Llama系列不断推动自然语言处理的发展，带动了模型商业化的快速增长。模型的部署多依赖云端服务，采用按代币计费的商业模式，使得模型提供商具有操控代币报告的经济动机。此前研究主要关注模型性能优化和内容偏差检测，缺乏对偏差操控的系统分析。Velasco等人（2025）指出，模型可能通过虚假报告代币数量谋取不当利益，揭示了模型经济伦理中的潜在风险。现有检测手段多依赖内容分析或模型内部信息，难以应对复杂偏差策略，亟需一种具有理论保证和高效实用的检测框架。
Core Problem
核心问题在于如何在模型提供偏差报告时，动态、准确地识别其是否存在操控行为。传统检测方法多依赖静态内容分析，难以应对偏差策略的多样性和复杂性。偏差操控不仅损害用户权益，也扰乱市场公平，亟需一种具有理论保障、能实时监控的检测机制。挑战在于偏差策略的未知性、模型信息的有限性，以及偏差行为的多样性，导致检测难度大、效率低。
Innovation
本研究的创新点在于引入基于鞅的序贯检验机制，结合无偏长度估计器，能在偏差策略未知的情况下，保证检测的完备性和效率。具体创新包括：
- �� 利用模型下一代概率分布，构建鞅过程，确保在偏差策略多样时的检测可靠性；
- �� 提出无偏估计器，用于动态估算模型生成序列的长度，避免偏差估计偏差；
- �� 设计自适应参数λ，平衡检测敏感性与误判风险；
- �� 理论证明偏差存在时，检测时间有限，误报率受控。这些创新突破了传统统计检测的局限，为大模型偏差审计提供了新思路。
Methodology
- �� 设定模型下一代概率分布，利用模型输出的条件概率，构建偏差长度的无偏估计器。
- �� 通过修改生成过程，确保序列长度的无偏估计，结合蒙特卡洛采样实现。
- �� 利用鞅理论，定义累积证据过程，逐步收集偏差证据。
- �� 设计序贯检验，设定阈值控制误判概率，动态判断偏差存在性。
- �� 调节参数λ，平衡检测速度与误判风险。
- �� 实验验证，少于70次样本即可检测偏差，误报率低于5%。
Experiments
采用Llama、Gemma、Ministral模型，使用LMSYS Chatbot Arena平台的4000个提示进行测试。模拟忠实与偏差模型，偏差策略包括随机拆分和偏向长序列。设定参数λ，利用Poisson(7)分布采样，进行多轮检测。对偏差模型和忠实模型进行150次重复，统计检测成功率和误报情况。结果显示，偏差模型在少于70次报告中即可被识别，忠实模型误报概率低于0.05，验证了框架的实用性和鲁棒性。
Results
检测偏差的平均样本数显著低于传统方法，少于70次即可识别偏差，误报率控制在α=0.05以内。偏差策略如随机拆分、偏向长序列均被准确识别，验证了方法的鲁棒性。不同模型和提示集上表现一致，展现出良好的泛化能力。实验还验证了参数λ的合理性，确保检测敏感性与误判风险的平衡。
Applications
该框架适用于云端大模型的第三方审计，确保模型提供的代币报告真实可靠。可应用于模型开发商、平台监管机构，提升模型市场的透明度和公平性。未来还可结合内容偏差检测，形成全面的模型诚信评估体系，推动行业健康发展。
Limitations & Outlook
依赖模型提供的下一代概率信息，若模型未公开或概率不准，可能影响检测效果。偏差策略复杂多变时，检测时间可能延长。当前方法主要针对长度偏差，未涵盖内容偏差或语义操控，未来需扩展多维检测能力。
Plain Language Accessible to non-experts
想象你在一家工厂工作，工厂每天生产商品。工厂老板可能会偷偷多报生产的商品数量，以多拿钱。你作为第三方检查员，不能直接看到工厂的内部流程，只能根据老板提供的每次生产的报告，判断老板是否在撒谎。你设计了一套聪明的检测方法，就像用数学的“天平”和“计数器”逐步验证老板的报告是否合理。每次老板报告后，你用特殊的算法计算出一个“证据值”，逐渐累积起来。如果这个证据超过了某个阈值，你就可以确定老板在作弊。这个方法保证了即使老板采用各种花招，你也能及时发现，且不会误判那些老实的工厂。它就像用科学的“侦查工具”保护消费者的权益，让市场更公平、更透明。
ELI14 Explained like you're 14
想象你在学校的食堂，老师每天发放饭菜，但有人怀疑老师会偷偷多发一些饭菜，然后自己多拿钱。你作为一个聪明的侦探，不能直接看到老师的厨房，只能根据老师每次发的饭菜数量，逐步判断老师是不是在作弊。你设计了一套特别的“数学检测器”，每次老师发饭菜后，你用一种叫“鞅”的数学工具，逐步累积证据。如果累积到一定程度，就说明老师可能在作弊，否则就说明老师很老实。这个方法非常聪明，能在很少的次数内就发现问题，而且不会误判那些老实的老师。它就像用科学的“放大镜”保护大家的权益，让市场变得更公平、更透明。
Glossary
鞅 (Martingale)
一种数学过程，具有在未来条件下的期望值等于当前值的特性，确保在偏差不存在时，证据不会系统性增加。
用于构建偏差检测的统计过程，保证在无偏差情况下不会误判。
无偏估计器 (Unbiased Estimator)
一种统计估计方法，其期望值等于被估参数的真实值，确保估计的准确性。
用于估算模型生成序列长度，保证偏差检测的可靠性。
偏差报告偏差 (Misreporting Intensity)
衡量模型偏差策略严重程度的指标，反映偏差行为导致的平均额外代币数。
用于量化偏差策略的严重性，是检测难度的重要依据。
Open Questions Unanswered questions from this research
- 1 如何在模型未公开下一代概率分布的情况下，有效实现偏差检测？
- 2 多偏差类型（如语义偏差、内容操控）联合检测的可行性与挑战。
- 3 在极端偏差策略或多模型联合偏差场景中的检测效率优化。
Applications
Immediate Applications
云端模型第三方审计
为模型平台提供偏差检测工具，确保模型报告的真实性，提升用户信任，符合监管要求。
模型开发商的合规检测
帮助开发者验证模型偏差策略，确保商业行为符合伦理标准，减少法律风险。
Long-term Vision
行业透明度提升
推动模型市场建立标准化的偏差检测体系，促进行业自律与规范发展。
Abstract
Millions of users rely on a market of cloud-based services to obtain access to state-of-the-art large language models. However, it has been very recently shown that the de facto pay-per-token pricing mechanism used by providers creates a financial incentive for them to strategize and misreport the (number of) tokens a model used to generate an output. In this paper, we develop an auditing framework based on martingale theory that enables a trusted third-party auditor who sequentially queries a provider to detect token misreporting. Crucially, we show that our framework is guaranteed to always detect token misreporting, regardless of the provider's (mis-)reporting policy, and not falsely flag a faithful provider as unfaithful with high probability. To validate our auditing framework, we conduct experiments across a wide range of (mis-)reporting policies using several large language models from the $\texttt{Llama}$, $\texttt{Gemma}$ and $\texttt{Ministral}$ families, and input prompts from a popular crowdsourced benchmarking platform. The results show that our framework detects an unfaithful provider after observing fewer than $\sim 70$ reported outputs, while maintaining the probability of falsely flagging a faithful provider below $α= 0.05$.