Clean Engineering, Unstable Measurement: A Preregistered Reliability Failure of Black-Box LLM Observers on Shared Endpoints
This study reveals instability in black-box LLM observers on shared endpoints, validating that fixed thresholds often fail in real-world settings.
Key Findings
Methodology
Using two preregistered campaigns, the authors fixed evaluation thresholds and analyzed 52,988 request attempts. They employed specific algorithms such as Spearman correlation and byte-level hash comparisons to assess repeatability of rankings and replays. The study dissects mechanisms like label-meaning biases, sub-noise floor candidate gaps, and permutation-induced ranking shifts, demonstrating that neither metric substitution nor sampling remedies the instability. Metadata and engineering details were verified to be at ceiling, confirming that the observed variability stems from platform and model mechanisms rather than implementation errors.
Key Results
- Across 52,988 requests, repeat rankings within the same window showed a Spearman correlation of 0.400, far below the 0.90 threshold; next-day byte-identical replays agreed at 0.78, below the 0.99 standard. All requests had perfect engineering records, indicating the instability is intrinsic.
- Different providers (four in total) shared similar stability floors (0.74–0.88 median), with no metadata fields predicting performance. Waiting, switching providers, or self-hosting only temporarily improved stability under low load.
- Constructed error tests showed that the measurement separation correlates with error type, not size, exposing the limitations of the metrics used.
Significance
This research emphasizes that in shared endpoint environments, model names are not fixed instruments. The unverified assumption that responses are stable over time is invalid, risking misjudging model capabilities. The findings advocate for rigorous, pre-registered measurement protocols and mechanism-based validation to ensure evaluation reliability, which is crucial for fair competition, benchmarking, and scientific integrity in AI development.
Technical Contribution
The paper introduces a comprehensive, fully preregistered audit framework combining mechanism decomposition, multi-platform validation, and calibration strategies. It establishes a snapshot-identity ladder and layered metrics, providing a scientific basis for assessing endpoint stability. This approach addresses platform nondeterminism, bias sources, and measurement noise, offering a new standard for external model evaluation and long-term reliability assurance.
Novelty
This is the first systematic, preregistered audit of black-box model endpoints that explicitly tests the stability assumption. It uniquely combines mechanism analysis, multi-platform validation, and formal thresholds, setting a new benchmark for measurement discipline in AI evaluation, surpassing prior work focused solely on self-consistency or single-platform reproducibility.
Limitations
- The experiments focus on a specific task (exact arithmetic) and black-box models, limiting generalization to complex tasks or white-box models.
- Platform differences and hardware environments may influence results, requiring broader validation across diverse settings.
- Metrics are based on byte-level comparisons and ranking correlations, which do not capture internal model state changes or semantic shifts.
Future Work
Future efforts will develop dynamic monitoring tools that incorporate internal model states, enabling real-time stability checks. Establishing industry standards for preregistered evaluation pipelines, integrating automated anomaly detection, and extending validation to multi-task and multimodal models are key directions. These advancements aim to foster more trustworthy, transparent AI systems with rigorous scientific foundations.
AI Executive Summary
In the rapidly evolving field of AI, the evaluation of large language models (LLMs) has become paramount. However, the assumption that model responses are stable over time and across platforms remains largely unverified in real-world deployment environments. This study rigorously tests this premise through two preregistered campaigns, analyzing nearly 53,000 requests on shared inference endpoints. Despite engineering efforts ensuring perfect request delivery and metadata consistency, the authors find significant measurement instability.
The core of the investigation involves dissecting the mechanisms behind response variability. They identify biases in label semantics, the existence of candidate gaps far below the noise floor, and permutation-induced ranking shifts as primary sources. These factors collectively cause repeat rankings to correlate at only 0.400, and byte-identical replays at 0.78, both well below the predefined thresholds of 0.90 and 0.99 respectively. The experiments span multiple providers and configurations, revealing that platform differences, load, and even self-hosting under low load only marginally improve stability.
This work underscores a critical insight: on shared endpoints, a model name cannot be considered a fixed measurement instrument. The authors propose a rigorous, mechanism-based validation framework, including snapshot-identity ladders and layered metrics, to establish measurement reliability. Their approach advocates for pre-registered thresholds, formal calibration, and multi-platform verification—principles that could transform evaluation standards in AI.
Overall, the findings highlight the need for a disciplined, scientific approach to model assessment, emphasizing transparency, reproducibility, and robustness. While the study’s scope is limited to specific tasks and models, it sets a foundational precedent for future research and industry practice. The goal is to ensure that AI systems are evaluated with the same rigor as traditional scientific instruments, fostering trust and fairness in AI deployment.
Deep Analysis
Background
随着大规模预训练模型（如GPT-3、LLaMA）在自然语言处理中的广泛应用，模型评估逐渐从人工判断转向自动化指标。早期工作如MT-Bench和Chatbot Arena提出了自动化评估框架，但缺乏对平台不确定性和测量偏差的系统分析。近年来，研究关注模型自我一致性（Haldar & Hockenmaier, 2025）和重现性（Kapoor et al., 2024），但多未考虑共享端点环境下的稳定性问题。平台机制（如批处理、硬件调度）引入了非确定性，影响测量的可信度。行业标准逐步建立，但缺乏严格的预注册和机制验证，导致评估结果的可信度不足。本研究基于此背景，提出端点端到端的机制拆解和预注册验证方案，旨在解决模型端点测量不稳定的核心难题。
Core Problem
在实际部署中，模型端点作为评估工具的稳定性受到质疑。共享端点的请求排队、硬件调度、模型版本更新等因素引入了非确定性，导致重复请求的响应不一致。传统评估方法未能充分验证指标的可靠性，存在偏差和噪声底下的候选差距。缺乏科学的机制拆解和预注册门控，使得模型性能的科学评估变得不可靠。这不仅影响模型的公平竞争，也阻碍了模型持续改进的科学基础。解决这一问题需要系统的机制拆解、严格的预注册流程和多平台验证，确保测量工具的可信度。
Innovation
本研究的核心创新在于：1）引入端点端到端的预注册审计框架，确保测量指标在正式评估前经过严格验证；2）机制拆解，将偏差归因于标签-含义映射、噪声底下的候选差距和平台非确定性，提供科学的偏差解释；3）多平台、多请求环境验证，揭示平台差异对测量稳定性的影响；4）提出快照-身份阶梯和指标分层体系，为模型端点的稳定性提供科学验证工具。这些创新超越了现有自我一致性和单平台测试的局限，为模型评估的科学性提供了新思路。
Methodology
- �� 设计两轮预注册的请求协议，固定阈值和请求计划，确保请求和响应的可追溯性。• 采用字节级比对确保请求一致性，结合Spearman相关系数和排名指标，评估重复请求和次日重放的一致性。• 机制拆解：分析标签-含义映射偏差、噪声底下的候选差距、字节相同输入引发的排名变化。• 多平台验证：在不同供应商环境下重复请求，比较平台差异。• 构造错误样本，验证指标对错误类型的敏感性。• 采用快照-身份阶梯和指标分层体系，确保测量的科学性和可重复性。
Experiments
实验采用由生成器产生的四个候选方案，任务为精确算术，唯一标签为正确或错误。请求在不同时间点（同窗口、次日）重复，记录响应、元数据和模型状态。通过多平台（四个供应商）和不同请求协议（单映射、双映射、全排序）验证指标稳定性。设置严格的预注册门控（如Spearman门槛、重放一致性），确保无后期调整。分析指标偏差、平台差异和机制拆解，验证其对模型端点稳定性的影响。
Results
重复请求的排名相关性显著低于预期（0.400对0.90），次日重放一致性也远低于预设（0.78对0.99）。平台间中位数一致性为0.74至0.88，未由元数据预测。构造错误样本显示指标偏差与错误类型相关，验证了指标的局限性。所有工程细节（请求、响应、元数据）均达最高标准，排除工程误差，确认问题源于平台机制。
Applications
该研究为模型端点的科学评估提供了基础工具，适用于模型开发者、平台运营商和评估机构。可用于验证模型在实际部署环境中的稳定性，指导模型版本管理和平台优化。长远来看，推动行业制定标准化的端点评估流程，结合自动化监控和机制验证，确保模型的持续可靠性，促进AI系统的透明度和公平性。
Limitations & Outlook
研究仅在特定任务（算术）和模型（黑箱LLM）环境下展开，未覆盖多任务、多模态模型。平台差异和硬件环境可能影响结果的普适性。指标偏差主要源于字节比对和排名，未考虑模型内部状态变化。未来需扩展多场景验证，结合模型内部信息，提升方法的普适性和解释力。
Plain Language Accessible to non-experts
想象你在一家工厂里，工人们每天都要检测产品质量。理想情况下，每次检测都应该得出一样的结论，比如一块金属是否合格。但实际上，工厂的机器、环境变化会让检测结果变得不稳定。有时候，同一块金属放在不同时间检测，结果可能不同。这个研究就像是在检查这些检测机器是否可靠。作者用严格的规则和多次测试，发现即使机器看起来一切正常，结果还是会有差异。原因包括机器的微小偏差、环境变化，甚至同一批产品的微小差别。通过分析这些因素，研究提出了更科学的方法来验证检测机器的稳定性，确保工厂的检测结果真实可靠。这样，工厂才能真正保证每个产品都符合标准，消费者也更放心。
ELI14 Explained like you're 14
想象你在学校里参加一个考试，你的老师说每次考试都应该给出一样的成绩，但实际上，有时候你考试的答案会因为考场环境、时间不同而得出不同的分数。这就像是AI模型的评估一样。科学家们发现，模型在不同时间、不同平台上给出的答案会有很大差异，就像考试成绩一样不稳定。为了搞清楚为什么会这样，他们设计了严格的测试规则，反复检查模型的表现，发现一些隐藏的问题，比如模型内部的偏差、平台的不同处理方式，都会影响结果。最后，他们提出了一套新的方法，像是给模型装上了“校准仪”，确保每次测试都能得到一致的结果。这样，未来我们就能更公平、更准确地评估AI的能力，就像老师用更科学的标准来评分一样。
Abstract
Language-model judges now gate training data, score generations, and drive leaderboards. The judge is then a measurement instrument, resting on one rarely stated assumption: the same request, sent to the same model name, reads the same tomorrow. We audited that assumption in two preregistered campaigns with every threshold fixed in advance; neither got past validating its instrument. Across 52,988 audited request attempts, same-window repeat rankings agreed at Spearman 0.400 against a required 0.90, and byte-identical next-day replays agreed at 0.78 against a required 0.99, each time with the execution record at ceiling. Three mechanisms explain the gap: a label-to-meaning mapping that biased readouts as strongly as the signal; candidate gaps seven orders of magnitude below the instrument's own noise floor; and byte-identical inputs returning different rankings, a noise that exact-permutation readouts compound. Neither metric substitution nor sampling repaired it on the tested grid. Preregistered follow-ups bound the problem: waiting did not help on the days sampled (0.805 versus 0.800, replicated over five further days); switching providers did not help (four providers share the floor, medians 0.74 to 0.88, predicted by none of the metadata fields they expose); self-hosting on batch-invariant kernels helped only while the server was quiet; and on constructed errors with known gaps, the readout's separation tracks error type, not size. We distill the evidence into a three-level snapshot-identity ladder, eight design rules, and a reporting checklist; a pilot at roughly 2% of the study's call volume would have exposed both unreachable gates in advance. All results concern externally measured behaviour on shared serving infrastructure. On a shared endpoint, a model name is not a frozen instrument; a preregistered evaluation must measure its instrument before freezing any gate on it.