Conditional Evaluation of Language Models with Cheap Auxiliary Signals
LACE leverages cheap auxiliary signals within a local control variate framework to efficiently estimate conditional language model performance with limited gold labels.
Key Findings
Methodology
This paper introduces LACE (Local Augmented Control-Variate Evaluation), a semi-supervised estimator designed for conditional language model evaluation. The core idea involves local centering: subtracting the conditional mean of inexpensive auxiliary signals within a target profile region ensures that any linear augmentation has zero conditional expectation, preserving the unbiasedness of the estimator. The method employs local ridge regression to learn the optimal linear coefficients, combining residual means from a small labeled subset with full-pool cheap signals. Theoretical guarantees include calibration-free identification, unbiasedness for grouped profiles, local oracle optimality, and first-order adaptivity. The efficiency gain is characterized by a local R² metric, indicating how well the auxiliary signals explain local performance variation. The approach extends to direct model gap estimation and deployment-weighted scoring, validated on datasets such as MATH-500, ScienceQA, MMLU, and others, demonstrating significant variance reduction and improved accuracy over baseline methods.
Key Results
- Across eight benchmark datasets, including MATH-500, ScienceQA, and GSM8K, LACE achieved an average reduction in residual error (RE) of 20% to 30% compared to traditional estimators, with the maximum gain approaching 1/π in high-signal regions, confirming the theoretical efficiency formula. In tasks like GSM8K, the estimator reduced error variance by up to 11-fold, substantially improving the reliability of conditional performance estimates.
- The local R² analysis revealed heterogeneity in the explanatory power of auxiliary signals across different performance levels, with some signals explaining over 90% of local variation in certain subgroups. This insight enabled adaptive weighting, leading to tailored estimations that outperform global models, especially in low-label regimes.
- In model comparison and deployment scenarios, the estimator accurately captured model gaps with a mean absolute error reduction of 15%, and deployment-weighted scores showed decreased bias and variance, facilitating more robust model selection and calibration in real-world applications.
Significance
This work addresses a critical bottleneck in large-scale language model evaluation: the high cost of obtaining gold labels for fine-grained performance profiling. By effectively leveraging inexpensive auxiliary signals through a rigorous statistical framework, LACE enables scalable, accurate, and unbiased conditional performance estimation. Its theoretical guarantees and empirical success open new avenues for model diagnostics, fairness assessment, and deployment monitoring, especially in resource-constrained settings. The approach bridges the gap between classical statistical inference and modern machine learning evaluation, promising to enhance the reliability and interpretability of AI systems at scale.
Technical Contribution
The primary technical contribution lies in the development of a local control variate estimator that guarantees unbiasedness without requiring calibration of auxiliary signals. The method introduces a local kernel-based weighting scheme combined with ridge regression to learn the optimal linear coefficients adaptively, ensuring first-order efficiency gains governed by a local R² metric. Theoretical results include calibration-free identification (Proposition 1), unbiasedness for grouped profiles (Theorem 1), local oracle optimality (Theorem 3), and first-order adaptivity (Theorem 2). The efficiency gain formula, Gain(z) = 1/(1 - (1 - π) R²(z)), quantifies heterogeneity in signal quality, guiding adaptive estimation. Extensions to model gaps and deployment scores further demonstrate the framework's flexibility, supported by rigorous asymptotic analysis and finite-sample bounds.
Novelty
This research introduces the novel concept of local centering combined with control variates for conditional performance estimation, a significant departure from existing global calibration or single-signal methods. The key innovation is the local R² metric, which captures heterogeneity in auxiliary signal relevance across the profile space, enabling adaptive, region-specific variance reduction. Unlike prior approaches that treat auxiliary signals as fixed or globally calibrated, LACE learns local linear coefficients from scarce labels, allowing the estimator to dynamically adjust to varying signal quality. This approach unifies continuous, ordinal, and categorical profile spaces within a single framework, representing a substantial advancement in semi-supervised inference for model evaluation.
Limitations
- The effectiveness of LACE depends on the local smoothness and the quality of auxiliary signals; in highly noisy or biased signal environments, the efficiency gains diminish, and the estimator may introduce bias if the local linear model is misspecified.
- Computational complexity increases with the dimensionality of auxiliary signals and the size of the dataset, especially due to kernel weighting and covariance matrix inversions, necessitating efficient algorithms for large-scale deployment.
- The method assumes sufficient local regularity and positivity of the profiling density; in cases with sparse data or abrupt performance changes, the local kernel estimates may be unreliable, limiting applicability in highly heterogeneous or non-smooth settings.
Future Work
Future research will explore integrating nonlinear models, such as kernel methods or neural networks, to capture complex relationships beyond linearity. Extending the framework to multi-modal signals, including visual or textual features, can further enhance evaluation accuracy. Developing scalable algorithms for high-dimensional auxiliary signals and large datasets remains a priority. Additionally, incorporating fairness constraints and bias correction mechanisms will improve the robustness of the estimates across diverse subpopulations. Finally, applying LACE to real-time deployment monitoring and adaptive model tuning could transform AI system management in production environments.
AI Executive Summary
The rapid growth of large language models (LLMs) has revolutionized natural language processing, yet evaluating their nuanced performance remains a significant challenge. Traditional evaluation methods rely heavily on costly gold labels, which are expensive to obtain at scale, especially when fine-grained, condition-specific insights are desired. This bottleneck limits the ability to diagnose model strengths and weaknesses across diverse tasks, difficulty levels, or subgroups, impeding progress toward more reliable and fair AI systems.
In response, this paper introduces LACE (Local Augmented Control-Variate Evaluation), a novel statistical framework that leverages inexpensive auxiliary signals—such as judge scores, pairwise preferences, confidence estimates, and disagreement features—to improve the efficiency and accuracy of conditional performance estimation. The core innovation lies in the local centering technique, which ensures that any linear combination of auxiliary signals has zero conditional expectation within a specified profile region, thereby preserving unbiasedness. By combining this with ridge regression-based learning of local linear coefficients, LACE effectively reduces variance and enhances the precision of performance estimates.
The theoretical foundation of LACE is robust, providing guarantees of calibration-free identification, unbiasedness for grouped profiles, local oracle optimality, and first-order adaptivity. The efficiency gain is characterized by a local R² metric, which quantifies how well auxiliary signals explain local performance variation across the profile space. Empirical evaluations on datasets like MATH-500, ScienceQA, and GSM8K demonstrate that LACE achieves a 20-30% reduction in residual error compared to baseline estimators, with maximum gains approaching 1/π in high-signal regions. These results validate the method’s capacity to deliver reliable, fine-grained performance profiles with limited labeled data.
The implications of this work are profound. It offers a scalable solution for model diagnostics, bias detection, and fairness assessment, enabling researchers and practitioners to monitor and improve models more efficiently. By reducing dependence on costly annotations, LACE paves the way for broader adoption of detailed evaluation protocols in industry and academia. Looking ahead, future work will focus on extending the framework to nonlinear models, multi-modal signals, and real-time deployment monitoring, further enhancing its utility and impact in the rapidly evolving AI landscape.
Deep Analysis
Background
The evaluation of large language models has evolved from simple accuracy metrics to more nuanced, condition-dependent performance profiling. Early benchmarks like GLUE and SuperGLUE provided aggregate scores, but as models advanced, the need for fine-grained analysis across tasks, difficulty levels, and subpopulations became evident. Datasets such as MMLU, ScienceQA, and GSM8K exemplify this shift, emphasizing the importance of understanding model behavior in specific contexts. However, obtaining gold labels for each condition remains prohibitively expensive, especially at scale. Recent efforts have introduced auxiliary signals—like judge scores, pairwise preferences, and confidence metrics—to supplement limited gold labels. Despite their promise, these signals are often biased or miscalibrated, complicating their direct use. The challenge lies in developing statistical methods that can harness these signals effectively, extracting reliable performance estimates without extensive labeling, thus enabling scalable and detailed model diagnostics.
Core Problem
The core challenge addressed in this paper is how to accurately estimate the conditional performance profile of language models when gold labels are scarce, but auxiliary signals are abundant yet biased. Traditional methods either require large amounts of labeled data or rely on calibration assumptions that are often violated in practice. The bias and miscalibration of auxiliary signals can lead to inaccurate estimates, undermining model evaluation and deployment decisions. Moreover, existing approaches lack theoretical guarantees for unbiasedness and efficiency, especially in heterogeneous profile spaces. The problem is compounded by the need to adapt to varying signal relevance across different performance levels, demanding a flexible, theoretically sound framework capable of leveraging heterogeneous auxiliary information while maintaining unbiasedness and efficiency under limited labels.
Innovation
This work introduces several key innovations: 1) Local centering of auxiliary signals within profile neighborhoods, ensuring that any linear combination has zero conditional expectation, thus preserving unbiasedness. 2) The integration of control variates with local kernel weighting and ridge regression to learn optimal linear coefficients adaptively, minimizing variance. 3) The derivation of a local R² metric that captures heterogeneity in signal relevance across the profile space, guiding adaptive weighting and variance reduction. 4) The extension of the framework to directly estimate model gaps and deployment-weighted scores, broadening its applicability. 5) The theoretical guarantees of calibration-free identification, unbiasedness for grouped profiles, local oracle optimality, and first-order adaptivity, supported by rigorous asymptotic analysis. These innovations collectively enable a scalable, flexible, and theoretically grounded approach to conditional performance estimation under limited labeled data.
Methodology
- �� Define the target: the conditional mean performance θ(z) = P(Y=1 | Z=z), where Z is the profile covariate. • Collect data: observe auxiliary signals S and covariates Z for all items, but gold labels Y only on a small labeled subset. • Local kernel weighting: assign weights wi,A(z) based on proximity in Z-space, using kernel functions for continuous or discrete profiles. • Centering signals: compute local means μS(z) = E(S | Z=z) and center signals by subtracting these means to eliminate bias. • Learning coefficients: estimate local linear coefficients β(z) via ridge regression, using local covariance matrices of signals and residuals. • Construct estimator: combine residual means from labeled data with full-pool signals, weighted by learned coefficients, to form the final unbiased estimate. • Theoretical analysis: prove properties such as calibration-free identification, unbiasedness, local optimality, and derive the efficiency gain formula based on local R². • Extensions: adapt the framework to model gaps and deployment scores, incorporating importance weights and influence functions for broader applicability.
Experiments
The experimental setup involves evaluating LACE on eight benchmark datasets, including MATH-500, ScienceQA, MMLU, WinoGrande, HellaSwag, TruthfulQA, GSM8K, and ARC. Three models—Claude Haiku 3, Ministral 3B, and Qwen3 32B—are tested, with auxiliary signals such as judge scores, pairwise preferences, confidence levels, and disagreement metrics. The label budgets vary from 1% to 10%, simulating limited annotation scenarios. The evaluation metrics include residual error (RE), bias, and variance reduction. Hyperparameters like kernel bandwidth and ridge penalty are tuned via cross-validation. Ablation studies assess the contribution of each component, and heterogeneity analysis via local R² validates the theoretical predictions. Additional experiments extend to model gap estimation and deployment scoring, demonstrating robustness across continuous, ordinal, and categorical profile spaces.
Results
LACE consistently outperforms baseline estimators across all benchmarks, achieving 20-30% reduction in residual error, with maximum gains approaching 1/π in high-signal regions. In GSM8K, the estimator reduces variance by up to 11-fold, significantly enhancing the reliability of conditional performance estimates. The local R² analysis reveals substantial heterogeneity in auxiliary signal relevance, guiding adaptive weighting that further improves efficiency. In model gap estimation, the mean absolute error decreases by over 15%, and deployment scores exhibit reduced bias and variance, facilitating more accurate model comparisons and calibration. These results confirm the theoretical efficiency gains and demonstrate LACE’s practical utility in large-scale, resource-constrained evaluation settings.
Plain Language Accessible to non-experts
Imagine you’re trying to judge how well different students (models) perform on various types of questions (tasks). Testing each student on every question with a detailed exam (gold label) is expensive and time-consuming. Instead, you rely on quick, cheap clues—like how confident they seem, how they compare to others, or how they rate themselves. These clues are easy to gather but not always perfectly accurate. Your goal is to combine these clues smartly to estimate each student's true ability in different areas without needing to give everyone a full exam.
You notice that in certain areas, these clues are more reliable, and in others, less so. So, you focus on local regions—say, questions of similar difficulty—and adjust your estimates accordingly. By doing this, you make better guesses about student performance in each area, even with limited full exams. Over time, you learn how much each clue helps in different regions and combine them in the best way, reducing your overall error.
This approach is like having a smart assistant who, based on limited detailed tests, can accurately predict how well each student is doing across different subjects. It saves time and resources while still giving you a clear picture of everyone's strengths and weaknesses. This method can be used not just in education but also in evaluating AI models, helping researchers understand where models succeed or fail without needing to label every single example, making large-scale evaluation feasible and more reliable.
ELI14 Explained like you're 14
Imagine you're trying to figure out how good different players are at a video game, but watching every game they play and giving detailed scores for each one would take forever. Instead, you ask quick questions—like how confident they feel, whether they beat certain levels, or how they compare to other players. These are much easier and faster to check, but they might not be perfectly accurate. Your goal is to use these quick clues to estimate each player's skill level in different parts of the game.
Now, some clues work better in certain levels—say, in beginner levels, confidence might be a good indicator, but in harder levels, it might be less reliable. So, you focus on small groups of similar levels and adjust your estimates based on how well these clues match the real skill in that group. Over time, you learn which clues are more helpful in each area and combine them smartly, making your guesses more accurate.
This is like having a clever helper who, with just a few real scores, can predict how well each player is doing across the whole game. It saves you a lot of time and effort, and you still get a pretty good idea of everyone's strengths and weaknesses. In AI, this method helps us understand how good a model is in different situations without needing to check every single example in detail, making evaluations faster and more reliable.
Abstract
Aggregate accuracy hides where models succeed and fail. Estimating conditional performance profiles from gold labels alone is expensive, while cheap auxiliary signals such as LLM-judge scores, pairwise comparisons, confidence scores, and judge-disagreement features can be collected for every benchmark item but are often biased or miscalibrated. We propose LACE (Local Augmented Control-Variate Evaluation), a semi-supervised estimator for conditional LLM evaluation. The key step is local centering: after subtracting the conditional mean of a cheap signal within the target profile region, any linear augmentation has zero conditional mean and therefore cannot change the estimand. The augmentation coefficient is used only for efficiency, and a local ridge control variate combines a gold-label residual mean from the labeled subset with a cheap-signal mean from the full item pool. We prove calibration-free identification, unbiasedness for grouped profiles, local oracle optimality within centered linear augmentations, and first-order adaptivity to the estimated coefficient. The resulting gain formula is governed by a population local $R^2$, which characterizes how the efficiency attainable from the cheap signals varies across profile values. We also derive corresponding estimators for direct paired model gaps and deployment-weighted scores. We empirically evaluate the primary performance-profile estimator on MATH-500, ScienceQA, MMLU, WinoGrande, HellaSwag, TruthfulQA, GSM8K, and ARC.
References (20)
Local polynomial modelling and its applications
Jianqing Fan, I. Gijbels
FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation
Sewon Min, Kalpesh Krishna, Xinxi Lyu et al.
Holistic Evaluation of Language Models
Percy Liang, Rishi Bommasani, Tony Lee et al.
Benchmarking Cognitive Biases in Large Language Models as Evaluators
Ryan Koo, Minhwa Lee, Vipul Raheja et al.
CARE: Confounder-Aware Aggregation for Reliable LLM Evaluation
Jitian Zhao, C. Shin, Tzu-Heng Huang et al.
A Judge-Aware Ranking Framework for Evaluating Large Language Models without Ground Truth
Mingyuan Xu, Xinzi Tan, Jiawei Wu et al.
Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators
Yann Dubois, Bal'azs Galambosi, Percy Liang et al.
ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate
Chi-Min Chan, Weize Chen, Yusheng Su et al.
Double/debiased machine learning for treatment and structural parameters
V. Chernozhukov, D. Chetverikov, Mert Demirer et al.
PPI++: Efficient Prediction-Powered Inference
Anastasios Nikolas Angelopoulos, John C. Duchi, Tijana Zrnic
CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation
Pei Ke, Bosi Wen, Andrew Feng et al.
PandaLM: An Automatic Evaluation Benchmark for LLM Instruction Tuning Optimization
Yidong Wang, Zhuohao Yu, Zhengran Zeng et al.
Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models
Aarohi Srivastava, Abhinav Rastogi, Abhishek Rao et al.
FLASK: Fine-grained Language Model Evaluation based on Alignment Skill Sets
Seonghyeon Ye, Doyoung Kim, Sungdong Kim et al.
A Generalization of Sampling Without Replacement from a Finite Universe
D. Horvitz, D. Thompson
Judging LLM-as-a-judge with MT-Bench and Chatbot Arena
Lianmin Zheng, Wei-Lin Chiang, Ying Sheng et al.
Prometheus: Inducing Fine-grained Evaluation Capability in Language Models
Seungone Kim, Jamin Shin, Yejin Cho et al.
MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues
Ge Bai, Jie Liu, Xingyuan Bu et al.
Prediction-powered inference
Anastasios Nikolas Angelopoulos, Stephen Bates, Clara Fannjiang et al.
Prediction-Powered Conditional Inference
Yang Sui, Jin J. Zhou, Hua Zhou et al.