Evaluating LLMs Without Oracle Feedback: Agentic Annotation Evaluation Through Unsupervised Consistency Signals
Proposes CAI ratio for unsupervised LLM evaluation, effectively guiding model selection in dynamic environments.
Key Findings
Methodology
This paper introduces an agentic evaluation framework where a student model collaborates with a noisy LLM teacher, assessing annotation quality via model agreement. The student employs a user-preference-based majority voting strategy, leveraging cosine similarity in embedding space to determine sample consistency. The CAI ratio quantifies the proportion of consistent versus inconsistent samples, serving as an unsupervised reliability metric. The approach operates without external labels, suitable for real-world, dynamic scenarios. Experiments utilize MINILM as the student, with GPT-3.5, GPT-4, Google Gemini, and Llama-8B across ten NLP datasets, demonstrating high correlation (over 0.9) between CAI ratio and LLM accuracy, enabling effective model selection.
Key Results
- Across ten datasets, the Pearson correlation between CAI ratio and LLM accuracy exceeds 0.85, outperforming traditional metrics. Using CAI for model selection correctly identified top models (e.g., GPT-3.5 Turbo, Google Gemini) with accuracy reaching 87.24%. The method proved stable across tasks like intent classification, sentiment analysis, and relation extraction. Consistent samples achieved over 90% accuracy, validating the reliability of agreement-based evaluation. The CAI ratio effectively distinguished high-performing models, confirming its robustness as an unsupervised indicator.
- The experimental results show that CAI ratio maintains a strong positive correlation with actual accuracy (average ρ > 0.9), making it a reliable heuristic for model selection. The approach reduces reliance on costly human annotations, accelerates model deployment, and offers a scalable solution for large-scale NLP applications. The analysis of sample consistency further supports the theoretical premise that agreement signals reflect true model performance, especially in noisy, real-world data.
- Additional ablation studies reveal that the combination of cosine similarity and preference-based voting enhances robustness, especially in low-data regimes. The method's effectiveness persists across different LLM architectures and datasets, indicating broad applicability. Overall, CAI ratio emerges as a practical, theoretically grounded metric for unsupervised model evaluation and selection in NLP.
Significance
This work addresses a critical bottleneck in deploying large language models—reliable, cost-effective evaluation without labeled data. By leveraging internal consistency signals, the proposed CAI ratio enables automatic, scalable assessment of model quality, facilitating rapid iteration and deployment in real-world applications. It bridges the gap between theoretical unsupervised metrics and practical needs, offering a new paradigm for model selection that reduces dependence on expensive annotations. The high correlation with actual accuracy underscores its potential to transform NLP workflows, especially in dynamic, data-rich environments where traditional evaluation is infeasible.
Technical Contribution
The core technical innovation lies in the integration of a student-teacher collaborative mechanism with a cosine similarity-based agreement measure, forming the CAI ratio. This metric captures the internal consistency of model outputs, serving as a proxy for accuracy. The framework employs preference-based majority voting within the embedding space, enhancing robustness against noise. Theoretical analysis demonstrates the asymptotic behavior of the CAI ratio, establishing its reliability as a performance indicator. The approach extends existing unsupervised evaluation methods by explicitly modeling agreement patterns, providing a scalable, low-cost alternative to traditional metrics.
Novelty
This study is the first to formalize the CAI ratio as an unsupervised, agreement-based metric for LLM annotation quality assessment. Unlike prior work relying on external labels or heuristic proxies, it exploits the intrinsic consistency signals between models. The combination of a student model with a preference-driven voting strategy and a cosine similarity agreement measure offers a novel, effective way to evaluate and select models without labeled data. This approach fundamentally shifts the paradigm from external supervision to internal model agreement, opening new avenues for scalable, autonomous evaluation.
Limitations
- The effectiveness depends on the assumption that model agreement correlates with accuracy; in cases of overconfidence or correlated errors, the CAI ratio may mislead.
- Performance on highly imbalanced or domain-specific datasets remains to be validated, as agreement signals may be less reliable under skewed distributions.
- Current validation is limited to classification tasks; extending to generative or multi-modal tasks requires further research.
Future Work
Future directions include extending the CAI framework to multi-modal tasks, integrating reinforcement learning for adaptive agreement thresholds, and exploring active learning strategies to refine preference signals. Additionally, investigating the theoretical bounds of agreement-based metrics in diverse data regimes and developing more sophisticated models for dynamic environment adaptation are promising avenues. The goal is to build a comprehensive, autonomous evaluation system capable of guiding large-scale NLP model deployment with minimal human intervention.
AI Executive Summary
In the rapidly evolving field of natural language processing, evaluating the quality of large language models (LLMs) remains a significant challenge, especially in real-world, unsupervised settings. Traditional metrics like accuracy and F1-score rely heavily on labeled datasets, which are costly and time-consuming to produce. As LLMs are increasingly used for annotation, fine-tuning, and deployment, there is a pressing need for reliable, scalable evaluation methods that do not depend on external labels.
This paper introduces a novel framework based on agentic reasoning, where a student model collaborates with a noisy LLM teacher to assess annotation quality through internal consistency signals. The key innovation is the CAI ratio, which quantifies the proportion of samples where the student and teacher agree, serving as an unsupervised proxy for model accuracy. The approach leverages cosine similarity in embedding space and a preference-based majority voting mechanism, enabling robust evaluation without external supervision.
Extensive experiments across ten NLP datasets, including intent classification and sentiment analysis, demonstrate that the CAI ratio correlates strongly with true model accuracy (correlation coefficients exceeding 0.9). The method successfully identifies the best-performing models, such as GPT-3.5 Turbo and Google Gemini, with high precision. The results highlight the potential of agreement-based metrics to facilitate model selection, reduce costs, and accelerate deployment.
Overall, this work offers a significant step toward autonomous, scalable evaluation of large models, addressing a critical bottleneck in NLP research and industry. Its theoretical grounding and empirical validation suggest broad applicability, paving the way for future enhancements in multi-modal, multi-task, and real-time model assessment systems.
Deep Dive
Key Concepts
Agreement Signal
An internal indicator derived from model output consistency, used to estimate the reliability of annotations.
Unsupervised Evaluation
Assessing model performance without relying on labeled ground truth, based on internal signals like agreement.
Model Robustness
The ability of a model to maintain performance across different data distributions and noise levels, assessed via agreement metrics.
Open Questions Unanswered questions from this research
- 1 How well does the CAI ratio generalize to generative tasks beyond classification, such as summarization or dialogue generation?
- 2 What are the theoretical limits of agreement-based metrics in highly noisy or adversarial environments?
- 3 Can the framework incorporate active learning to iteratively improve model reliability assessment?
Applications
Immediate Applications
Model Selection in Deployment
Automatically identify high-performing models in real-time without labeled data, reducing costs and speeding up deployment.
Unsupervised Fine-tuning
Guide fine-tuning processes by selecting models with high CAI ratios, ensuring better annotation quality.
Long-term Vision
Autonomous Model Evaluation Systems
Develop self-assessing AI systems capable of continuous performance monitoring and adaptation in dynamic environments.
Abstract
Large Language Models (LLMs), when paired with prompt-based tasks, have significantly reduced data annotation costs and reliance on human annotators. However, evaluating the quality of their annotations remains challenging in dynamic, unsupervised environments where oracle feedback is scarce and conventional methods fail. To address this challenge, we propose a novel agentic annotation paradigm, where a student model collaborates with a noisy teacher (the LLM) to assess and refine annotation quality without relying on oracle feedback. The student model, acting as an unsupervised feedback mechanism, employs a user preference-based majority voting strategy to evaluate the consistency of the LLM outputs. To systematically measure the reliability of LLM-generated annotations, we introduce the Consistent and Inconsistent (CAI) Ratio, a novel unsupervised evaluation metric. The CAI Ratio not only quantifies the annotation quality of the noisy teacher under limited user preferences but also plays a critical role in model selection, enabling the identification of robust LLMs in dynamic, unsupervised environments. Applied to ten open-domain NLP datasets across four LLMs, the CAI Ratio demonstrates a strong positive correlation with LLM accuracy, establishing it as an essential tool for unsupervised evaluation and model selection in real-world settings.