Codifying the Judge: Scalable Evaluation via Program Distillation
PAJAMA distills LLM evaluation into programs, matching 13B model accuracy while cutting API costs by 50×.
Key Findings
Methodology
PAJAMA synthesizes Python programs from LLM evaluation logic, calibrates their outputs, aggregates decisions, and uses a fallback mechanism for low-confidence cases.
Key Results
- Result 1: Programmatic judges achieve 88.78% accuracy on Prometheus, matching OLMO-2-7B-INSTRUCT, while running 47.25× faster.
- Result 2: Hybrid evaluation improves OLMO-2-7B-INSTRUCT accuracy by 5% and throughput by 2.9×.
- Result 3: On RewardBench, reward models trained on program labels outperform GPT-4 labels at 50× lower API cost.
Significance
This work addresses LLM evaluation's cost, latency, and transparency issues, offering a scalable solution for automated evaluation with significant cost savings and interpretability.
Technical Contribution
Introduces program distillation for LLM evaluation logic, combining weak supervision and hybrid routing to enhance efficiency and reliability.
Novelty
First to distill LLM evaluation logic into executable programs, introducing the PAJAMA system to overcome cost and transparency bottlenecks in LLM evaluation.
Limitations
- Limitation 1: Program quality depends on the initial LLM, potentially introducing biases.
- Limitation 2: Complex or uncovered cases still require costly LLM fallback.
- Limitation 3: Diversity and coverage of programs are limited by the initial rubric design.
Future Work
Future work could explore advanced program synthesis techniques, more efficient routing strategies, and applications to more complex tasks.
AI Executive Summary
The current paradigm of using LLMs as judges for automated evaluation faces challenges such as high costs, latency, and opaque decision-making. To address these issues, the authors propose PAJAMA, a system that distills LLM evaluation logic into executable programs. These programmatic judges are transparent, reusable, and significantly reduce API costs.
PAJAMA comprises three components: program synthesis from diverse evaluation rubrics, calibration and aggregation of program outputs into joint decisions, and a fallback mechanism for low-confidence cases. Experiments across five datasets and four model families demonstrate that programmatic judges match the accuracy of a 13B LLM while running 47× faster. Hybrid evaluation further improves accuracy and throughput, advancing the Pareto frontier.
Additionally, programmatic judges provide cost-effective reward signals. On RewardBench, reward models trained on program labels outperform those trained on GPT-4 labels at 50× lower cost. While challenges remain, such as program diversity and reliance on initial LLM quality, PAJAMA offers a scalable, transparent, and efficient solution for automated evaluation and reward modeling.
Deep Analysis
Background
LLMs have become critical for tasks like preference labeling, reward modeling, and reinforcement learning feedback. However, LLM-based evaluation suffers from high costs, latency, and systemic biases, limiting scalability.
Core Problem
LLM evaluation is expensive, slow, and opaque. Revising evaluation criteria requires re-running inference on entire datasets, leading to inefficiency. These issues hinder large-scale deployment.
Innovation
PAJAMA's innovations include: 1) distilling LLM evaluation logic into executable programs for transparency and cost reduction; 2) calibrating and aggregating program outputs for reliability; 3) introducing a fallback mechanism for hybrid evaluation.
Methodology
- �� Use LLMs to generate Python programs based on evaluation rubrics.
- �� Normalize and calibrate program outputs to ensure consistent scoring.
- �� Select top-k programs based on validation accuracy and aggregate their verdicts using weak supervision.
- �� Route low-confidence cases to LLMs using program-derived signals like vote variance.
Experiments
Experiments were conducted on five datasets (e.g., Prometheus, JudgeLM) and four model families (e.g., OLMO-2, QWEN2.5). Metrics included accuracy, throughput, and cost. RewardBench was used to evaluate reward model performance.
Results
Programmatic judges achieved 88.78% accuracy on Prometheus, matching OLMO-2-7B-INSTRUCT, and ran 47× faster. Hybrid evaluation improved accuracy by 5% and throughput by 2.9× on OLMO-2-7B-INSTRUCT.
Applications
PAJAMA is ideal for large-scale automated evaluation, reward model training, and cost-effective model performance monitoring, especially in resource-constrained settings.
Limitations & Outlook
Program quality depends on the initial LLM, which may introduce biases. Complex cases require LLM fallback. Program diversity is limited by the initial rubric design.
Plain Language Accessible to non-experts
Imagine a master chef who can cook any dish but charges a lot for every meal. PAJAMA is like writing down the chef's recipes so you can cook the meals yourself. These recipes are easy to follow, cost less, and can be adjusted as needed. For really tricky dishes, you can still call the chef, but most of the time, the recipes are enough.
ELI14 Explained like you're 14
Imagine you have a super-smart friend who always knows the right answers, but asking them costs a lot of money. PAJAMA is like getting your friend to teach you how to figure things out on your own. Now you can save money and still get good answers! Cool, right?
Glossary
Program Distillation
The process of converting LLM evaluation logic into executable programs to reduce API costs.
Used to create programmatic judges.
Weak Supervision
Combining noisy labels from multiple sources to produce high-quality labels.
Used to aggregate programmatic judges' decisions.
Hybrid Evaluation
Combining programmatic and LLM judges for efficient evaluation.
Used to handle low-confidence cases.
Pareto Frontier
A set of optimal solutions where improving one metric compromises another.
Used to evaluate accuracy-throughput trade-offs.
Reward Model
A model trained on preference data to guide generative models' behavior.
Tested on RewardBench using programmatic labels.
Open Questions Unanswered questions from this research
- 1 How can program diversity and coverage be further improved?
- 2 What are more efficient routing strategies to reduce LLM fallback costs?
- 3 How can PAJAMA be extended to more complex tasks?
Applications
Immediate Applications
Low-cost Model Evaluation
Quickly evaluate model performance using programmatic judges, suitable for resource-constrained teams.
Reward Model Training
Train reward models using program-generated labels, reducing API costs.
Long-term Vision
Automated AI Evaluation Systems
Develop fully automated evaluation frameworks adaptable to various tasks and domains.
Abstract
LLM-as-a-judge has become the standard for automated evaluation, but it suffers from high cost, significant latency, and opaque decisions -- limitations that undermine its scalability and reliability. We address these with a simple, efficient alternative: program distillation. Instead of prompting an LLM at the evaluation time, we distill its decision logic into a committee of programs that score candidates directly. These programmatic judges offer transparency, are easily inspected or edited, and eliminate per-sample API costs. Building on this notion, we introduce PAJAMA, a system that synthesizes programs as judges, aggregates their decisions into a joint verdict, and incorporates a fallback mechanism to selectively escalate low-confidence cases to an LLM. Across five datasets and four model families, we show that programmatic judges can match the performance of a 13B-size LLM judge. When using program outputs as routing signals, PAJAMA improves both accuracy and throughput and advances the Pareto frontier. Beyond evaluation, programmatic judges produce cheap and effective reward signals: on RewardBench, a reward model distilled from programs' verdicts outperforms one trained on a proprietary LLM's labels at two orders of magnitude lower API cost.