Ready Jurist One: Benchmarking Language Agents for Legal Intelligence in Dynamic Environments

Legal AI has evolved from basic information extraction and statute memorization, exemplified by datasets like LexGLUE and LawBench, towards more complex reasoning tasks. Early systems focused on static NLP tasks, but real legal practice involves multi-turn dialogues, procedural compliance, and multi-stakeholder interactions. Recent advances in large language models (LLMs) such as GPT-4 and specialized legal models like LawLLM have demonstrated impressive knowledge capabilities. However, these models often lack the ability to perform in dynamic, procedural environments, limiting their practical deployment. Existing benchmarks do not adequately evaluate models' abilities in realistic legal workflows, especially in multi-agent, multi-step scenarios. This gap hampers progress towards general legal intelligence, which requires models to understand, execute, and coordinate complex procedures akin to human legal practitioners.

Current legal AI models excel at factual recall but struggle with procedural adherence, multi-turn interactions, and role-based collaboration in dynamic environments. Static benchmarks fail to reflect real-world legal tasks, which demand step-by-step reasoning, strict procedural compliance, and multi-agent coordination. This disconnect results in models that perform well in isolated tasks but falter in practical settings, such as court proceedings or legal consultations. The core challenge is designing an evaluation framework that captures these complexities, enabling meaningful assessment and targeted improvements. Without such tools, progress toward deployable legal AI remains limited, and trust in AI-assisted legal processes is hindered.

This work introduces several key innovations: 1) J1-ENVS, a hierarchical, multi-role environment based on authentic legal data, supporting multi-turn interactions across consultation, drafting, and adjudication. 2) J1-EVAL, a dual-metric evaluation system combining outcome accuracy and procedural compliance, evaluated via rule-based and LLM-based methods. 3) Integration of real-world legal case data, including Chinese judgment documents and legal articles, to ensure realism. 4) Emphasis on procedural and interactional metrics, addressing gaps in existing static benchmarks. These innovations enable a comprehensive, realistic assessment of models’ legal reasoning, procedural adherence, and multi-agent collaboration, guiding future model development and deployment.

• �� Data collection from Chinese judgment documents and legal articles, extracting factual and legal elements. • Role modeling using personality theories (Big Five) to generate diverse, realistic agents. • Environment design with three hierarchical levels: consultation, document drafting, and court adjudication, each with specific roles and interaction protocols. • Multi-turn dialogue systems enabling agents to ask questions, gather evidence, and perform procedural steps. • Implementation of rule-based and LLM-based scoring mechanisms for outcome and process metrics, ensuring objective evaluation. • Integration of real case data into environment instances, totaling 508 scenarios covering civil and criminal cases. • Extensive testing across 17 models, analyzing performance in knowledge, procedural adherence, and interaction fidelity.

The evaluation involved 17 models, including GPT-4o, LawLLM, and open-source models like Deepseek-v3. The dataset comprised 508 instances across three levels, with metrics measuring accuracy, structure conformity, procedural completeness, and reasoning quality. Experiments tested models’ abilities in knowledge recall, procedural execution, multi-role interaction, and long-horizon reasoning. Fine-tuning and prompt engineering were employed to optimize performance. Results showed knowledge competence but highlighted significant procedural gaps, especially in complex court scenarios. Ablation studies confirmed the importance of procedural adherence metrics, and cross-model comparisons revealed size and training data influence performance. The environment’s realism was validated through expert review and detailed error analysis.

Models achieved over 70% accuracy in factual knowledge questions but scored below 60% in procedural tasks like court judgments. Procedural adherence in multi-turn interactions was often below 50%, with errors mainly in step sequencing and role coordination. Larger models like GPT-4o outperformed smaller ones but still exhibited major procedural lapses. Results demonstrated knowledge alone is insufficient; procedural and interaction skills are critical for practical legal AI. The dual-metric evaluation revealed that models with higher process adherence consistently produced more valid outcomes, emphasizing the importance of procedural learning. These insights guide future efforts to improve AI’s legal reasoning and operational fidelity.

The platform can serve as a benchmark for developing and evaluating legal AI systems, supporting applications such as automated legal consultation, document drafting, and judicial assistance. It provides a realistic testing ground for models to learn procedural norms and multi-agent collaboration. Additionally, it can be used for training legal practitioners, simulating complex scenarios for skill enhancement. In the long term, the framework aims to facilitate AI systems capable of autonomous legal reasoning, reducing judicial workload, and improving access to justice by providing reliable, transparent, and compliant legal support.

The environment is tailored to Chinese legal procedures, limiting immediate applicability to other jurisdictions. Model performance in multi-agent, multi-stage scenarios remains imperfect, especially in long-horizon tasks. The evaluation metrics focus on accuracy and procedural compliance but lack interpretability and reasoning depth assessment. Heavy reliance on GPT-4o as the driver model may introduce bias; broader model comparisons are needed. Future work should address environment complexity, cross-jurisdictional adaptation, and multi-modal data integration to enhance robustness and generalization.