Long SKILL Compliance as Logical Reasoning: Closure-Grounded Detection with Scaling-Guided On-Policy Distillation
SkillCDG employs a two-layer graph model for long SKILL compliance detection, achieving 12.8% F1 improvement and reducing token usage by 64.3%.
Key Findings
Methodology
This work introduces SkillCDG, a framework converting enterprise long SKILL policies into a two-layer constraint dependency graph: the upper layer indexes scenario routing, while the lower layer captures atomic rule dependencies. During inference, two-stage retrieval narrows the context, followed by dependency closure to form a complete rule subgraph. An LLM then performs compliance judgment based on this structured graph, with source spans retained for traceability. The approach enhances logical completeness and reduces inference overhead. Extensive evaluation on three enterprise datasets and two public benchmarks shows detection F1 scores improve by up to 12.8%, with token savings up to 64.3%. Furthermore, the study models the relationship between model scale, policy complexity (measured via rule density and dependency coupling), and detection accuracy, revealing a saturation trend. Leveraging this, adaptive sample selection and on-policy distillation are proposed to boost small model performance efficiently.
Key Results
- On three enterprise datasets, SkillCDG outperforms baselines with up to 12.8 percentage points higher F1 scores, and token consumption drops by 64.3%, demonstrating both effectiveness and efficiency.
- Across model scales from 4B to 27B parameters, detection accuracy exhibits a saturation pattern, with the policy complexity metric effectively predicting instance difficulty and potential performance gains.
- Using complexity-based instance prioritization and on-policy distillation, small models significantly improve in long-policy compliance detection, validating the scalability trend and practical utility.
Significance
This research advances automated enterprise compliance auditing by integrating structured logical reasoning into large language models. The two-layer graph approach ensures logical completeness and source traceability, addressing limitations of flat rule collections. The discovery of a model-scale and policy-complexity scaling law guides efficient model deployment, reducing costs while maintaining high accuracy. Such innovations are crucial for industries like finance, insurance, and supply chain management, where regulatory adherence is critical. The framework also opens avenues for future research in structured policy modeling, scalable inference, and adaptive training strategies, fostering the development of more reliable and cost-effective AI auditing tools.
Technical Contribution
The paper introduces a novel two-layer constraint dependency graph (CDG) that explicitly models logical dependencies among atomic rules, enabling precise and complete compliance reasoning. It integrates graph traversal and transitive closure to form policy-specific contexts, reducing inference complexity. The study further develops a quantitative complexity metric combining rule density and dependency coupling, which correlates with detection difficulty. By modeling the relationship between model size, policy complexity, and detection accuracy, the authors propose a scaling law that predicts performance saturation. They also design an adaptive sampling and on-policy distillation strategy based on this law, improving small model capabilities without increasing inference costs significantly. These contributions collectively push forward the state-of-the-art in structured policy reasoning and efficient model scaling.
Novelty
This work is the first to formalize enterprise long SKILL policies as a hierarchical dependency graph, explicitly capturing logical prerequisites and enabling complete rule closure. Unlike prior semantic or entity-centric graphs, this approach emphasizes logical completeness and source traceability. The integration of a quantitative complexity metric with empirical scaling laws offers a new perspective on model performance prediction and resource allocation. The combination of structured graph reasoning with adaptive training based on policy complexity represents a significant innovation, bridging logical formalism with scalable deep learning methods.
Limitations
- The approach relies on well-structured policy documents; ambiguous or poorly formatted policies may impair graph construction and reasoning accuracy.
- Graph traversal and closure computations introduce additional computational overhead, potentially limiting real-time deployment in very large-scale settings.
- The empirical scaling law is validated primarily within the studied datasets and model families; its generalization to other domains or architectures remains to be tested.
Future Work
Future research will explore unsupervised or semi-supervised methods for automatic policy graph construction, reducing reliance on manual annotations. Extending the framework to multi-modal policies, incorporating visual or tabular data, could enhance robustness. Additionally, developing more efficient graph algorithms and incremental updating mechanisms will improve scalability. Further, integrating reinforcement learning to dynamically adapt sampling and distillation strategies based on real-time performance metrics is a promising direction to optimize resource allocation and model accuracy in diverse enterprise scenarios.
AI Executive Summary
In the era of increasingly complex enterprise operations, long policy documents—such as SKILLs—serve as critical standards for compliance and auditing. Traditional methods relying on keyword matching or flat rule collections struggle with the sheer length and logical intricacy of these policies, leading to inefficiencies and inaccuracies. This challenge is compounded by the need for source traceability and logical completeness, especially when violations hinge on intricate prerequisite dependencies.
To address these issues, the authors propose SkillCDG, a novel framework that models long SKILL policies as a hierarchical, two-layer constraint dependency graph. The upper layer indexes policies for rapid scenario routing, while the lower layer captures atomic rule dependencies, explicitly modeling logical prerequisites. During inference, a two-stage retrieval process narrows the context, followed by dependency closure to generate a complete rule subgraph. An LLM then performs compliance judgment based on this structured context, with source spans retained for audit traceability.
Extensive experiments across multiple enterprise datasets and public benchmarks demonstrate that SkillCDG significantly outperforms baseline methods, achieving up to 12.8% higher F1 scores while reducing token consumption by over 60%. The study also uncovers a saturation trend in detection accuracy relative to model scale, modulated by policy complexity—quantified via rule density and dependency coupling. Leveraging this insight, the authors develop an adaptive sampling and on-policy distillation strategy, which notably enhances the performance of small models.
This research marks a substantial step forward in automated compliance auditing, combining structured logical reasoning with scalable deep learning. It offers a practical blueprint for deploying efficient, accurate, and traceable policy compliance systems in diverse industries, paving the way for more reliable and cost-effective enterprise AI solutions.
Deep Analysis
Background
Enterprise长政策文件在合规审查中的重要性不断提升，传统基于规则或关键词匹配的方法难以应对长文本中的复杂逻辑关系。近年来，知识图谱和语义关系抽取等图结构技术逐渐成为研究热点，代表工作包括知识图谱构建、语义关系抽取等。尽管如此，长文本中的依赖关系复杂，如何高效准确地提取和利用这些关系，仍是行业难题。现有方法如RAG和语义图虽能改善信息访问，但在逻辑完整性和推理效率方面仍有不足。企业政策的多样性和复杂性要求新型结构化建模技术，特别是在保证逻辑严密的同时，兼顾推理速度和模型规模的平衡。
Core Problem
长SKILL政策的合规检测面临多重挑战：一是政策文本庞大，信息分散，导致检索成本高；二是政策规则之间存在复杂的逻辑依赖，单纯检索难以保证完整性；三是现有模型在处理长文本时，推理效率低、准确率不足，限制了实际应用。如何在保证逻辑完整性的基础上，提高检测效率和模型适应性，成为行业亟需解决的问题。这不仅关系到企业合规的自动化水平，也影响到法规遵从的效率和准确性。
Innovation
本研究的核心创新在于提出两层约束依赖图（CDG）模型，将长政策文本转化为结构化的逻辑图，明确规则间的依赖关系。上层索引场景，支持快速定位相关政策；下层捕获原子规则及其依赖关系，支持逻辑闭包。结合图遍历和闭包机制，有效保证逻辑完整性。引入策略复杂度指标，量化政策难度，结合模型缩放规律，设计自适应样本筛选和蒸馏策略，提升小模型性能。这一方法区别于传统的语义匹配或知识图谱，强调逻辑严密和推理可追溯，具有理论创新和工程实用价值。
Methodology
- �� 将长SKILL政策转化为两层约束依赖图（CDG），上层索引场景，下层存储原子规则及其require关系。
- �� 离线阶段：将长文本拆分成语义块，提取原子规则，构建局部子图，融合成完整规则图，定义require关系。
- �� 在线阶段：
- 通过关键词检索快速定位相关场景和规则。
- 从种子规则出发，遍历require边，生成完整依赖闭包。
- 利用LLM对闭包进行合规判断，源跨度提供证据支持。
- �� 结合策略复杂度指标（规则密度、依赖耦合）量化策略难度，结合模型缩放规律，设计样本优先级和蒸馏策略，优化模型性能。
Experiments
采用三家企业的订单履约、售后和商户支持等数据集，以及公开的CompliBench和Long变体，评估检测准确率和F1。比较RawSkill、LightRAG和SkillCDG在不同模型（Qwen3.5、DeepSeek-V4、GPT-5）上的表现。重点分析模型规模与策略复杂度的关系，验证缩放规律的有效性。实验设置包括不同模型参数、不同策略复杂度阈值，采用交叉验证和置信区间估计，确保结果的稳健性。
Results
SkillCDG在三企业数据集上，F1最高提升12.8个百分点，令牌消耗最大降低64.3%，优于对比方法。模型规模从4B到27B，检测正确率表现出饱和趋势，策略复杂度指标能有效预测实例难度。基于复杂度的样本优先级排序，结合蒸馏策略，显著提升小模型性能，验证了模型缩放和策略复杂度的关系，为实际部署提供理论指导。
Applications
该方法适用于企业自动审计、法规遵从、合同合规检测等场景，依赖企业政策文本的结构化表示和逻辑依赖关系。可实现自动化合规检测、风险识别和源头追溯，降低人工成本，提高审查效率。未来还可结合多模态信息，扩展到更复杂的场景，实现全流程自动化合规管理。
Limitations & Outlook
当前模型对政策格式和标注依赖较强，面对模糊或极端复杂政策时表现不足。图遍历和闭包计算存在一定成本，可能影响大规模实时应用。模型缩放规律在特定数据集验证，泛化到其他行业仍需验证。未来需优化图结构构建和推理效率，增强模型的鲁棒性和泛化能力。
Plain Language Accessible to non-experts
想象你在管理一个复杂的工厂，工厂里有许多工序和规则，每个工序都依赖前面的步骤。要确保整个生产流程顺利，必须知道每个步骤的前置条件和依赖关系。传统方法就像只看每个工序的说明书，容易遗漏关键步骤或误判。本文提出的方法像是用一张详细的流程图，把每个工序和它的依赖关系都画出来。这样一来，工厂管理者可以快速找到问题所在，确保每个环节都符合规定。通过这种结构化的方式，不仅提高了效率，还能更准确地检测出潜在的违规行为。就像在厨房做菜，知道每个调料的用量和顺序，才能做出美味佳肴。这个方法让企业的政策变得像一份清晰的操作手册，自动检测是否遵守了所有规则。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，每个拼图块都代表一个规则或条件。要拼出完整的图案，你需要知道每个拼图块的前置条件，比如哪个拼图必须先放，哪个可以后放。有时候，拼图太多，容易搞错顺序或者漏掉重要的部分。这个研究就像是设计了一张超级聪明的拼图指南，把所有拼图块的关系都画出来，告诉你哪个必须先放，哪个可以后放。这样一来，你就可以用这张图快速找到拼图的正确顺序，确保拼出来的图案没有漏洞。它还会告诉你哪些拼图最难拼，帮你优先处理那些关键部分。最终，这个方法让拼图变得更简单、更快，也更容易拼出完美的图案。就像在游戏中找到最强的策略一样，帮助企业自动检测政策是否被正确遵守，避免出错。
Abstract
The increasing complexity of enterprise business scenarios has promoted the widespread adoption of long SKILL documents in agent systems, posing new challenges for compliance detection: large models incur substantial inference costs, while small models may fail to maintain detection accuracy. To address this gap, we propose SkillCDG, a graph-based framework for long SKILL compliance detection. SkillCDG represents complex business policies as a two-layer constraint dependency graph, where the upper layer indexes SKILL descriptions for scenario routing and the lower layer captures dependencies among atomic constraints within each SKILL. During inference, two-level retrieval followed by dependency closure supports compliance judgment and source traceability. We comprehensively evaluate the framework on three enterprise datasets and two controlled public benchmark variants. Experimental results demonstrate that SkillCDG outperforms baseline methods by up to 12.8 percentage points in detection F1 score, while reducing token consumption by a maximum 64.3\%. Moreover, we further investigate the inherent relationships among policy-graph complexity, model scale, and detection performance. Comparative experiments conducted on four checkpoints from a single model family validate a concise and effective scaling trend: end-to-end detection correctness exhibits a complexity-differentiated scaling pattern, and the complexity metric derived from the constraint dependency graph can effectively quantify instance difficulty and the performance improvement potential of models. Leveraging this insightful scaling trend, we conduct adaptive training sample selection and adopt on-policy distillation to efficiently enhance the compliance detection capability of small-scale models.