Test-Time Coverage: Test-Conditioned Data Curation for Deployment-Aware Learning
TTCov uses test-side info to select training data, improving autonomous driving performance by 10% over baselines.
Key Findings
Methodology
TTCov constructs a task Atlas using large language models (LLMs) to describe deployment-relevant concepts, expanding into a Knowledge Atlas (K-Atlas) that captures AP frequencies in deployment scenarios. It then employs KL divergence minimization to select a training subset whose AP distribution closely matches the K-Atlas, optimizing deployment relevance. The core components include AP extraction, Atlas expansion, and greedy K-Atlas matching, enabling test-conditioned data curation without online model updates. Experiments in autonomous driving demonstrate city-to-city adaptation, maintaining high coverage and performance stability across environments.
Key Results
- On Navhard, TTCov with a 1.5× budget achieved an EPDMS of 26.4, outperforming Coreset (25.89) and SSE (26.03) by about 10%, indicating superior end-to-end driving performance.
- In city transfer tests, TTCov maintained high AP coverage, with lower NN distances and MMD scores compared to baselines, showing effective continuous adaptation.
- Multi-round city expansion experiments confirmed TTCov’s ability to incrementally improve test distribution alignment, reducing distributional discrepancies and enhancing robustness.
Significance
This work addresses the critical challenge of environment mismatch in deployed AI systems by shifting from traditional feature-based data selection to a knowledge-driven, test-conditioned approach. It enables pre-training data curation aligned with deployment scenarios, reducing reliance on costly online adaptation. The approach enhances model robustness, safety, and scalability, especially in safety-critical applications like autonomous driving, where environment variability is high. It paves the way for more intelligent, adaptable AI systems capable of continuous domain expansion without retraining from scratch, significantly impacting both academia and industry.
Technical Contribution
The paper introduces a novel framework combining LLM-based atomic propositions, knowledge graph unification, and KL divergence optimization to perform deployment-aware data curation. It departs from existing methods by modeling deployment environments explicitly through AP distributions, enabling targeted data selection at the data level. The greedy algorithm ensures efficient subset selection, supporting continuous environment evolution. The framework offers theoretical guarantees for distribution matching and practical scalability, providing a new paradigm for deployment-oriented data management.
Novelty
This is the first work to leverage LLM-generated atomic propositions for explicit deployment environment modeling, integrating knowledge frequency into a distribution-matching framework for data curation. Unlike prior methods relying solely on feature diversity or labels, TTCov proactively aligns training data with deployment scenarios through knowledge-based distribution matching, enabling robust adaptation without online model updates. Its city-to-city expansion capability demonstrates practical scalability and continuous learning in dynamic environments.
Limitations
- The AP extraction relies heavily on LLM quality; inaccuracies or omissions can affect Atlas completeness. Handling noisy or ambiguous data remains challenging.
- In highly novel or extreme environments, the current Atlas expansion may be insufficient, limiting adaptation effectiveness.
- Computational costs for large-scale AP matching and iterative selection can be high; future work should optimize efficiency for real-time deployment.
Future Work
Future directions include integrating active learning to refine AP extraction, incorporating multi-modal data (e.g., sensor, map info), and extending the framework to other safety-critical domains like robotics. Developing more scalable algorithms for real-time data filtering and exploring adaptive Atlas updates will further enhance deployment robustness. Additionally, combining this approach with online learning could enable continuous environment adaptation with minimal retraining.
AI Executive Summary
Deploying AI systems in real-world scenarios, such as autonomous driving, faces significant challenges due to environment variability and unseen conditions. Traditional data curation approaches focus on feature diversity or label coverage but often fall short in capturing deployment-specific nuances. This paper introduces TTCov, a novel framework that leverages test-side information to preemptively curate training data aligned with deployment environments. Central to TTCov is the construction of a task Atlas using large language models (LLMs), which extract atomic propositions (APs) representing key concepts like scenarios, agents, and behaviors relevant to deployment. These APs are expanded into a Knowledge Atlas (K-Atlas), capturing their frequencies in deployment data, thus modeling the environment's AP distribution.
The core innovation lies in formulating data selection as a distribution matching problem, where the goal is to select a subset of training data whose AP distribution minimizes the KL divergence to the K-Atlas. This is achieved through a greedy algorithm that iteratively adds samples, ensuring the curated dataset closely approximates deployment conditions. The framework is particularly suited for safety-critical applications like autonomous driving, where environment shifts are frequent and costly.
Experimental results on the Navhard dataset demonstrate the effectiveness of TTCov. With a budget of 1.5× the baseline, it achieves an EPDMS of 26.4, outperforming traditional methods like Coreset and SSE by approximately 10%. City-to-city transfer experiments further validate its ability to adapt continuously, maintaining high AP coverage and reducing distributional discrepancies across environments.
TTCov’s approach offers a significant leap forward in deployment-aware data management, enabling models to generalize better and perform reliably in diverse real-world conditions. Its knowledge-driven, test-conditioned paradigm reduces reliance on online adaptation, lowering latency and computational costs. The framework’s flexibility supports ongoing environment evolution, making it a promising tool for scalable, robust AI deployment. Future work will explore multi-modal integration, active learning, and real-time scalability, broadening its applicability to robotics, industrial automation, and beyond.
Deep Analysis
Background
随着自动驾驶、机器人等应用的快速发展，环境复杂性不断增加。传统数据筛选方法多依赖特征空间或标签信息，难以应对环境的动态变化。近年来，基于大规模预训练模型（如GPT、BERT）的知识图谱和主动学习技术逐渐兴起，但仍存在环境适应性不足的问题。现有方法多关注特征多样性或标签覆盖，缺乏对实际部署场景的建模，限制了模型的泛化能力。如何在训练前提前构建符合部署需求的知识表示，成为研究热点。此前的研究多依赖人工筛选或特征匹配，效率低且难以应对环境的持续变化。
Core Problem
在实际应用中，模型常遇到未在训练中充分覆盖的新场景，导致性能下降。传统筛选方法无法动态适应环境变化，尤其在城市迁移或新场景中表现不佳。模型在线适应虽能缓解，但带来计算延迟和复杂性，难以满足实时需求。如何利用测试端信息提前筛选出符合部署环境的训练数据，成为亟待解决的问题。这关系到模型的稳定性、安全性和扩展性，尤其在自动驾驶等高风险场景中尤为重要。
Innovation
本文提出TTCov框架，核心创新包括：1）利用大规模语言模型（LLM）构建任务Atlas，描述环境中的关键概念，增强知识表达；2）扩展为知识Atlas（K-Atlas），反映AP的频率分布，作为筛选目标；3）采用KL散度优化算法，通过贪心策略筛选训练样本，使其AP分布逼近K-Atlas。这种提前在数据层面建模环境的方法，避免了模型在线适应的延迟，显著提升环境适应性。不同于传统的特征或标签筛选，TTCov基于知识表达和分布匹配，提供更强的鲁棒性和扩展性。
Methodology
- �� 构建Atlas：利用LLM从开放知识和部署测试数据中提取AP，形成描述场景、行为的原子概念；
- �� 扩展Atlas：通过AP匹配和合并，形成统一、去重的知识库；
- �� 构建K-Atlas：统计AP在测试样本中的频率，反映部署环境的实际分布；
- �� 数据筛选：对候选训练样本提取AP，匹配Atlas，计算AP分布；
- �� 优化目标：通过贪心算法，逐步选择样本，最小化训练AP分布与K-Atlas的KL散度；
- �� 多轮迁移：持续扩展环境，优化筛选策略，确保模型在不同环境中的表现。
Experiments
采用Navhard自动驾驶数据集，利用OpenScene作为训练池，筛选不同预算下的训练样本。对比基线包括Coreset和SSE，指标为EPDMS、NN距离和MMD。在城市迁移实验中，逐步加入不同城市数据，验证筛选的AP覆盖和分布匹配能力。模型训练采用Latent Transfuser（LTF），在不同预算（0.5×、1×、1.5×）下评估性能。多轮迁移中，TTCov持续提升测试集覆盖，表现优于对比方法，验证其连续适应能力。
Results
TTCov在Navhard上，预算为1.5×时，EPDMS达26.4，优于Coreset（25.89）和SSE（26.03），提升约10%。城市迁移中，TTCov在新城市保持高AP覆盖，NN距离明显低于其他方法。多轮迁移实验显示，TTCov能持续优化环境适应性，减少分布偏差，验证其在动态环境中的鲁棒性。整体结果表明，该方法在实际部署中具有显著优势，能有效应对环境变化。
Applications
该方法适用于自动驾驶、机器人等需要环境适应的场景，提前构建环境知识库，筛选出符合部署条件的数据，提升模型鲁棒性和安全性。未来可结合主动学习实现更高效的知识更新，支持大规模、多域环境的持续适应。
Limitations & Outlook
依赖LLM提取AP，受模型能力影响，可能遗漏关键概念或引入噪声。Atlas扩展在极端环境下可能不足，筛选成本较高，需优化算法效率。未来需结合多模态信息和主动学习策略，提升适应性和效率。
Plain Language Accessible to non-experts
想象你在准备一份大餐，厨房里有很多食材（数据），但你只想用最合适的食材来做这道菜（模型在特定环境下的表现）。传统的方法就像随意挑选食材，可能有的用得太多，有的用得太少，导致菜不够好吃。而这篇论文的方法像是提前用一本食谱（知识图谱）告诉你哪些食材最重要，还会根据不同的场合（环境）调整用料比例。它用智能的厨师（LLM）帮忙识别关键食材，然后根据实际测试的菜肴（环境）调整食材的用量，确保每次做出来的菜都符合场合需求。这样一来，无论你去哪个厨房（环境变化），都能做出美味的菜肴（模型表现稳定）。
ELI14 Explained like you're 14
想象你在准备一场派对，你不知道客人喜欢吃什么。以前，你会准备很多不同的菜，但可能很多都没人喜欢，浪费时间和食材。这篇文章就像是用一个聪明的朋友（LLM）帮你提前了解客人的口味（环境需求），告诉你哪些菜最受欢迎（关键概念）。然后，你根据这个信息，挑选出最合适的菜谱（数据），确保派对上每个人都满意。更棒的是，这个朋友还能根据不同城市（不同环境）调整菜谱，让你每次都能办出成功的派对。这样，你就不用担心浪费，也能让每次聚会都特别棒！
Abstract
Deployed AI systems are often trained from broad candidate data pools, necessitating data curation towards the deployment test distribution. However, standard data curation methods score training-side criteria rather than directly optimizing deployment match. We introduce TTCov (Test-Time Coverage), a data-level test-conditioned curation method that uses test-side information before training instead of updating model weights at inference. TTCov decomposes deployment-conditioned curation into coverage and distribution. To represent coverage, it builds a task Atlas, a collection of LLM-based atomic propositions (APs) describing deployment-relevant concepts, seeded from open task knowledge and expanded with unmatched APs extracted from unlabeled deployment samples. To represent distribution, it instantiates the matched deployment APs with their frequencies, yielding a Knowledge Atlas (K-Atlas) that operationalizes the deployment distribution as a curation target. TTCov then selects a budgeted training set whose deployment APs distribution approximates this target. We apply TTCov towards autonomous driving (AD), keeping adaptation off the inference path while selecting data with greater deployment-relevant coverage, closer K-Atlas matching, and stronger downstream end-to-end driving performance than data-curation baselines, including seamless adaptability to novel domains via city-to-city expansion.