Rethinking Math Reasoning Evaluation: A Robust LLM-as-a-Judge Framework Beyond Symbolic Rigidity

In recent years, large language models (LLMs) have made significant advances in natural language processing and reasoning tasks. Mathematical reasoning, as one of the fundamental tasks for evaluating models' logical reasoning and problem-solving abilities, has been a focus of research. However, traditional mathematical reasoning evaluation methods primarily rely on symbolic mathematics tools, such as SymPy, which have limitations in handling diverse mathematical representations and answer formats. Especially when the model-generated answer format differs from expectations, symbolic comparison methods may lead to inaccurate evaluation results. To address this issue, researchers have begun exploring LLM-based evaluation methods to improve the accuracy and robustness of mathematical reasoning evaluation.

The core problem in mathematical reasoning evaluation is accurately verifying model-generated answers. Traditional symbolic math comparison methods have limitations in handling diverse mathematical representations and answer formats, struggling to generalize to different mathematical expressions and solution formats. This can lead to models being underestimated due to different answer formats, even when the answer is mathematically correct. Additionally, symbolic verification systems assume specific notations and formatting styles as the ground truth answer, increasing uncertainty in evaluation.

The core innovation of this paper is the introduction of an LLM-based evaluation framework called LLM-as-a-judge. • This framework does not rely on traditional symbolic math comparison but leverages the generalization capabilities and prior knowledge of LLMs to evaluate model-generated answers. • It ensures accuracy and consistency through two stages: independent question answering and dataset answer validation. • Finally, an LLM acts as a judge to evaluate model predictions, enhancing robustness through multiple assessments and majority voting. • The introduction of the pass@k metric to assess the diversity and reliability of model outputs provides a new evaluation perspective.

The paper proposes an LLM-based evaluation framework called LLM-as-a-judge. • First, in the independent question answering stage, the LLM generates candidate answers for each question without providing the dataset's ground truth answer to reduce bias towards the dataset answer. • Then, in the dataset answer validation stage, the LLM evaluates the correctness of the generated answer against the dataset ground truth answer and synthesizes a final validated answer. • Finally, an LLM acts as a judge to evaluate model predictions, enhancing robustness through multiple assessments and majority voting. • Random sampling and shuffling of responses are used to reduce input response position bias.

The experimental design includes evaluating the performance of the LLM-as-a-judge method on multiple datasets, such as GSM8K and Minerva. • Experiments are conducted using the Qwen2.5 model series (including 7B, 14B, and 32B parameters) and comparing two popular evaluation frameworks: Lighteval and SimpleRL. • The pass@k metric is used to assess the diversity and reliability of model outputs, and a meta-evaluation is conducted to verify the accuracy of the evaluation method. • On the meta-evaluation dataset, the correctness of model responses is manually annotated for numerical evaluation and quantifying contributions.

Experimental results show that the LLM-as-a-judge method improved accuracy by approximately 2.7% over traditional symbolic evaluation methods on the Qwen2.5-7B model, particularly on GSM8K and Minerva datasets. • Comparing SimpleRL and Lighteval frameworks, the LLM-as-a-judge method showed consistent evaluation results across different frameworks, whereas symbolic evaluation methods exhibited significant discrepancies. • On a meta-evaluation dataset, the LLM-as-a-judge method achieved an F1 score of 0.969, significantly outperforming the symbolic evaluation method's 0.741.

This method has broad application prospects in academia and industry. • In academia, the LLM-as-a-judge method provides a more reliable means of assessing mathematical reasoning, enabling more accurate performance monitoring and advancing intelligent systems. • In industry, it helps improve the accuracy of solving mathematical problems, especially in applications requiring complex mathematical expressions, such as scientific computing and engineering design.

Despite the excellent performance of the LLM-as-a-judge method in evaluation accuracy and robustness, there are still some limitations. • First, the LLM-as-a-judge method may be limited by the LLM's capabilities when handling certain complex mathematical problems, leading to less accurate evaluation results. • Second, due to the generative nature of LLMs, evaluation results may be influenced by input response position bias, although random sampling and shuffling can partially mitigate this issue. • Additionally, errors or inconsistencies within the dataset itself may affect evaluation accuracy, although filtering out inapplicable samples can improve reliability. Future research directions include further optimizing the evaluation framework to enhance accuracy in handling complex mathematical problems and exploring applications in other domains.