From Benchmarking to Reasoning: A Dual-Aspect, Large-Scale Evaluation of LLMs on Vietnamese Legal Text

In Vietnam, the complexity of legal texts and the use of specialized terminology make it difficult for ordinary citizens to understand and access legal information. This situation is particularly pronounced in civil law systems, where legal provisions are often expressed in complex legal language and structures, hindering public understanding of their fundamental rights and obligations. In recent years, the advent of Large Language Models (LLMs) has offered new possibilities for simplifying legal texts. By translating complex legal provisions into more understandable language, LLMs have the potential to lower the barrier to public access to legal information. However, this potential is accompanied by the risk of generating fluent but inaccurate legal simplifications. Therefore, evaluating the capabilities of LLMs in legal text processing has become crucial. Existing research primarily focuses on surface performance metrics such as legal accuracy, user-perceived readability, and output consistency, but these metrics fail to explain the reasons behind model performance.

The complexity of Vietnam's legal texts presents a significant barrier to public access to justice. While LLMs offer a promising solution for legal text simplification, evaluating their true capabilities requires a multifaceted approach that goes beyond surface-level metrics. Existing research primarily focuses on surface performance metrics such as legal accuracy, user-perceived readability, and output consistency, but these metrics fail to explain the reasons behind model performance. A model might achieve a high accuracy score by correctly summarizing the general rule for inheritance, yet completely miss a critical exception for a specific circumstance, a subtle but catastrophic reasoning error that superficial scores would mask.

This paper introduces a dual-aspect evaluation framework combining quantitative benchmarking and qualitative error analysis. This approach not only reveals performance differences in legal text processing but also provides an in-depth understanding of model failure modes. By introducing an expert-validated error typology, the research offers a new perspective for evaluating and improving legal AI. Unlike previous studies that focus primarily on surface performance, this paper delves into the reasoning errors of models, revealing their systemic weaknesses in legal reasoning.

• �� Establish performance benchmark: Evaluate four state-of-the-art large language models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) across three dimensions: Accuracy, Readability, and Consistency.

• �� Conduct large-scale error analysis: Analyze 60 complex Vietnamese legal articles using a novel, expert-validated error typology.

• �� Dataset selection: Choose 20 articles each from the Penal Code 2015, Civil Code 2015, and Land Law 2024 to ensure representativeness and challenge.

• �� Task design: Use zero-shot prompt asking models to act as legal assistants, explaining legal articles and providing practical examples for laypersons.

• �� Evaluation metrics: Include Legal Accuracy, Readability, and Consistency, rated by law students and non-expert participants.

The experimental design involves selecting four state-of-the-art large language models (GPT-4o, Claude 3 Opus, Gemini 1.5 Pro, and Grok-1) and evaluating them using 60 complex legal articles selected from the Penal Code 2015, Civil Code 2015, and Land Law 2024. Each article is generated twice by the models, creating a corpus of 480 outputs. Evaluation metrics include Legal Accuracy, Readability, and Consistency, rated by law students and non-expert participants. Additionally, a novel expert-validated error typology is used for detailed error analysis of model outputs.

The experimental results show that Grok-1 excels in Readability and Consistency but compromises on fine-grained legal Accuracy. Claude 3 Opus achieves high Accuracy scores that mask a significant number of subtle but critical reasoning errors. Error analysis identifies Incorrect Example and Misinterpretation as the most prevalent failures, indicating that the primary challenge for current LLMs is controlled, accurate legal reasoning rather than summarization. By integrating a quantitative benchmark with a qualitative deep dive, the study provides a holistic and actionable assessment of LLMs for legal applications.

The findings of this study can be directly applied to the simplification of legal texts and public access to legal information. By identifying and categorizing error types, the study offers concrete suggestions for future model development. Additionally, the results can be used to evaluate and improve the safety and reliability of existing legal AI applications, particularly in civil law systems like Vietnam.

Despite the dual-aspect evaluation framework providing deep insights into the capabilities of LLMs in legal text processing, there are some limitations. First, the dataset size is relatively small, consisting of only 60 legal articles, which may not fully reflect model performance across other legal domains. Second, the reliance on law students for error annotation, despite rigorous training, may lack the practical experience of professional experts. Additionally, the experimental design is limited to a zero-shot setting, not considering other techniques that might improve performance. Future research could expand the dataset size, cover more legal domains, and explore techniques like few-shot learning or chain-of-thought prompting.