Improving Argument Saliency Coverage in Small LLMs for Long Legal Opinion Summarization via Sequence-Level Distillation
Sequence-level distillation from Qwen3-14B raises small-model legal argument coverage, reaching ARCscore 0.781 on CANLII.
Key Findings
Methodology
The paper applies Kim and Rush’s sequence-level knowledge distillation through supervised fine-tuning. Given prompt P and legal opinion L, a teacher samples a reasoning trace τ and summary S; the student minimizes −(1/|G|)Σlog Pm(gt|g<t,P,L). Three targets are compared: summary-only G=S, reasoning-only G=τ, and joint G=τ⊕S.
Key Results
- On 1,049 expert-annotated CANLII test opinions, Qwen3-14B summary distillation reaches ARCscore 0.621, 0.717, 0.744, and 0.781 for 0.6B, 1.7B, 4B, and 8B students, versus zero-shot scores of 0.408, 0.522, 0.631, and 0.658.
- Teacher choice matters: GPT-5-mini distillation yields 0.553, 0.652, 0.712, and 0.741. Yet standalone teacher ARCscore is nearly identical—0.742±0.166 for GPT-5-mini and 0.738±0.172 for Qwen3-14B—suggesting student-teacher distribution alignment matters.
- Coverage gains mainly reduce Missing Errors: for the 0.6B student, ME falls from 55.87% to 31.31% while FE rises from 10.20% to 12.22%; for 8B, ME falls 30.01% to 17.40% and FE 11.86% to 9.05%.
Significance
The work addresses a central failure in long legal summarization: salient Issues, Reasons, and Conclusions are sparse across thousands of tokens. It shows that unlabeled in-domain opinions plus teacher-generated summaries can transfer content-selection behavior without costly expert annotation. This is valuable for legal retrieval, case understanding, and research support, where omission is often more damaging than stylistic weakness. The method improves assistance, not autonomous legal judgment.
Technical Contribution
The study extends sequence-level distillation beyond news benchmarks to long legal opinions and evaluates argument coverage directly with ARCscore and FE/ME diagnostics. It isolates summary supervision from native reasoning-trace supervision across a common Qwen3 student family. The finding that summaries alone suffice, together with strong gains from roughly 10 documents, provides an efficient engineering recipe for domain adaptation under limited annotation and compute.
Novelty
The novelty is the targeted empirical study of argument-saliency transfer in long legal opinions, not a new loss function. Unlike rationale-oriented methods such as TriSum, it systematically tests teacher identity, student scale, reasoning supervision, and training-set size, showing that joint reasoning-plus-summary training adds only +0.001 to +0.011 ARCscore over summaries alone.
Limitations
- Evidence is restricted to CANLII, ARCscore, the Qwen3 family, and two teachers; transfer across jurisdictions, legal subdomains, metrics, and architectures remains unknown.
- Experiments use LoRA and standard SFT rather than full-parameter or preference optimization. Teacher biases may be inherited, and ARCscore treats atomic arguments as equally salient.
Future Work
The authors propose contrastive or preference-based objectives, replacing manually annotated ARCscore facts with LLM-extracted facts, and broader cross-domain validation. A curriculum could first distill on-distribution teacher summaries and then adapt to harder expert data. Learned saliency weights and stronger faithfulness evaluation are also needed.
AI Executive Summary
Long legal opinions contain thousands of tokens, while their decisive Issues, Reasons, and Conclusions may appear far apart. Small language models often produce fluent summaries that omit precisely these arguments. Expert-summary tuning is expensive, and structured inference-time planning such as Chain-of-Arguments does not reliably solve the selection problem.
The authors use sequence-level knowledge distillation. GPT-5-mini or Qwen3-14B generates roughly 250-word summaries for 1,000 unlabeled opinions; Qwen3 students from 0.6B to 8B parameters then learn those complete sequences with SFT. On 1,049 annotated CANLII test opinions, Qwen3-14B distillation reaches ARCscore 0.621, 0.717, 0.744, and 0.781 across student sizes, exceeding zero-shot scores of 0.408, 0.522, 0.631, and 0.658, as well as expert tuning and CoA.
The improvement is chiefly coverage: the 0.6B student’s Missing Errors fall from 55.87% to 31.31%, while the 8B model falls from 30.01% to 17.40%. Significant gains appear with about 10 examples and largely saturate after 100. Reasoning-only distillation is competitive, but adding reasoning to summaries yields only marginal gains. The study therefore presents an economical adaptation recipe, while cautioning that it covers one dataset, metric, and model family and cannot replace professional legal review.
Deep Analysis
Background
Legal summarization has moved from generic fluency toward faithfulness and argument coverage. ArgLegalSumm, argument-aware reranking, and ARCscore address argument identification, generation selection, and evaluation. Yet news benchmarks such as CNN/DailyMail and XSum do not capture the sparse, dispersed evidence structure of legal opinions, where omission of one reasoning step can change practical usefulness.
Core Problem
The task is to preserve salient legal arguments while avoiding contradictions, not merely to shorten text. Bottlenecks include long-context selection, sparse argumentative evidence, costly expert summaries, limited deployment budgets, and a trade-off between coverage and factual consistency. Small models are especially vulnerable to omission because they must compress both case structure and legal reasoning.
Innovation
- �� Applies sequence-level distillation to long legal argument coverage.
- �� Compares GPT-5-mini and Qwen3-14B teachers, separating output quality from distributional compatibility.
- �� Tests summary-only, reasoning-only, and joint supervision.
- �� Measures scaling at 10, 100, and 1,000 documents, revealing unusually high data efficiency.
Methodology
- �� Input: prompt P and opinion L, with training contexts truncated to 4,096 tokens.
- �� Teachers: temperature 0.7 and nucleus sampling p=0.95; summaries average 250 words. Qwen3-14B also supplies native trace τ.
- �� Objectives: SFT cross-entropy over S, τ, or τ⊕S.
- �� Students: Qwen3-0.6B, 1.7B, 4B, and 8B, tuned with LoRA and evaluated with thinking/no-thinking inference.
- �� Evaluation: ARCscore uses an LLM atomic verifier to estimate supported expert arguments and separates Factual Errors from Missing Errors.
Experiments
CANLII contains about 28,000 opinion-summary pairs. The test set has 1,049 opinions with sentence-level IRC annotations; 1,000 non-overlapping unlabeled opinions train the models. Baselines are zero-shot B, expert-summary tuning E, and Chain-of-Arguments prompting. Teachers generate summaries matched to the 270-word expert-summary average. Teacher ARCscore is 0.742±0.166 for GPT-5-mini and 0.738±0.172 for Qwen3-14B.
Results
Qwen3-14B summary distillation reaches 0.621–0.781 across students and consistently beats B, E, and CoA. GPT-5-mini also helps but peaks at 0.741. ME decreases at every size; FE improves mainly at 4B and above. Around 10 examples produce significant gains, while performance saturates near 100–1,000 documents. A second seed changes scores by at most 0.031 ARCscore.
Applications
Potential uses include case previews, judgment-point extraction, lawyer search assistance, and internal legal knowledge bases. Deployment requires jurisdictional adaptation, source citations, human review, and separate factuality auditing. Teacher-generated data reduces annotation cost but does not eliminate governance, privacy, or bias requirements.
Limitations & Outlook
The scope is narrow: one dataset, metric, student family, and two teachers. LoRA and standard SFT may not represent stronger optimization. Reasoning traces may contain redundant or non-transferable content. ARCscore gives atomic facts equal weight and does not fully measure fluency, factual reliability, or operational legal risk.
Plain Language Accessible to non-experts
Imagine a huge warehouse filled with case documents. A useful summary is not a pile of random boxes; it must retrieve three special shipments: what legal question was asked, why the court reasoned as it did, and what decision was reached. A small model is like a new warehouse worker: it can write a neat delivery note, but may accidentally leave the most important boxes behind.
The paper first asks a highly experienced worker—the teacher model—to prepare a concise delivery note. The smaller worker then studies the original warehouse map together with many such notes. No expert has to label every box by hand. The smaller worker learns the teacher’s habit of deciding what deserves space.
With the Qwen3-14B teacher, the smallest student’s coverage score rises from 0.408 to 0.621, and the 8B student reaches 0.781 instead of 0.658. Only about 10 practice documents produce most of the improvement. However, the worker can still misread a label, so every summary needs checking before it supports a real legal task.
ELI14 Explained like you're 14
Suppose you must explain a 500-page school mystery novel to a friend before class. You need the main question, the clues that prove the answer, and the ending—not every character’s snack order. A small AI can sound confident while remembering the boring details and forgetting the crucial clue. That is exactly the danger in long legal opinions!
The researchers use a “cool older student” model as a coach. The coach reads an entire legal case and writes a short, roughly 250-word explanation. A smaller Qwen3 model reads the case and many coach examples, then practices making similar choices. It is like learning not just how to write, but what is worth putting in the study guide.
The results are impressive: with a Qwen3-14B coach, a 0.6B student’s ARCscore jumps from 0.408 to 0.621, while an 8B student reaches 0.781 from 0.658. Even around 10 examples help a lot—wow! More examples still help, but much less.
The researchers also tested the coach’s hidden thinking notes. Those notes were not very useful once the final summaries were available. And remember: better highlighting is not the same as being a lawyer. The AI can still miss or distort facts, so humans must check its work.
Glossary
Sequence-level knowledge distillation
A small model learns complete sequences generated by a stronger model rather than only token probabilities. This transfers generation behavior, including content selection.
The paper trains students on teacher summaries, reasoning traces, or both.
ARCscore
An LLM-verifier metric estimating the proportion of expert-annotated arguments supported by a generated summary. It distinguishes Missing Errors from Factual Errors.
It is the primary evaluation measure.
IRC scheme
A legal argument-role scheme consisting of Issues, Reasons, and Conclusions. It structures sentence-level annotations for coverage evaluation.
The annotated CANLII test split uses IRC labels.
Missing Error
Failure to include a salient argument that should appear in the summary. It differs from contradicting an argument, which is a Factual Error.
Distillation primarily reduces this error type.
LoRA
A parameter-efficient fine-tuning method that trains low-rank updates while freezing most base-model weights. It lowers memory and compute requirements.
All student experiments use LoRA.
Open Questions Unanswered questions from this research
- 1 Can the approach transfer to U.S. Supreme Court opinions, other jurisdictions, legal subfields, or non-legal long documents? The paper provides no cross-domain evidence, so controlled replication is needed.
- 2 ARCscore treats atomic arguments as equally important, although legal users may value some arguments far more than others. Expert-derived saliency weighting remains unresolved.
- 3 Why does teacher-student distribution alignment outperform similar standalone teacher quality? Causal studies controlling prompts, architecture, and data distribution are needed.
Applications
Immediate Applications
Case preview generation
Legal information platforms can distill a small local model from teacher summaries of unlabeled opinions, producing fast previews of Issues, Reasons, and Conclusions. Outputs should retain source citations and undergo human sampling audits.
Lawyer search assistance
Firms can use a distilled model to triage internal opinions and surface likely relevant arguments. The system should link every claim to source passages and function as a retrieval aid, not legal advice.
Long-term Vision
Auditable legal summarization infrastructure
Future systems could combine argument graphs, factual verification, and learned saliency weights to show which arguments a summary covers or omits. Such transparency could support large-scale judicial knowledge management.
Abstract
We show that sequence-level distillation from a capable long-context teacher model is a simple, annotation-free, and data-efficient strategy for improving argument saliency coverage in long legal opinion summarization, where small LLMs often struggle to retain the most salient argumentative content. Across student model sizes, distillation consistently surpasses tuning on expert-written summaries in our legal-opinion setting. We further demonstrate that most gains are achieved with as few as ~10 training summaries, highlighting the strong data efficiency of teacher-generated supervision. Finally, we find that summary distillation is sufficient for improvements: reasoning-chain distillation remains competitive with summary-only distillation, but provides marginal benefit when combined with summary supervision.