Post-Launch Capability Expansion of Vision-Language Models via Prompting for On-Orbit Spacecraft Inspection
Using SAM3 with structured prompts enables zero-shot spacecraft component segmentation without model fine-tuning, achieving 0.385 [email protected] on unseen satellites.
Key Findings
Methodology
This study employs a pretrained SAM3 model in a strictly frozen state, utilizing carefully designed structured textual prompts that incorporate spatial and geometric descriptions. The model performs single-pass inference on images of unseen satellites, producing instance segmentation masks conditioned solely on prompts. The prompts guide the model to localize large spacecraft structures reliably, with performance evaluated via AP metrics at IoU thresholds 0.5 and 0.5:0.95. The approach emphasizes prompt design over model adaptation, demonstrating practical deployment feasibility on embedded GPU hardware.
Key Results
- SAM3 achieves 0.385 [email protected] and 0.267 [email protected]:0.95 on the test set of 129 images. Large structures like spacecraft bodies ([email protected]=0.639) and solar arrays ([email protected]=0.598) are localized accurately, while small components such as antennas ([email protected]=0.221) and thrusters ([email protected]=0.081) pose challenges. Structured prompts with spatial and geometric descriptors improve detection by up to 82%. The model operates within embedded GPU constraints, validating its practical applicability.
Significance
This work addresses the critical challenge of post-launch semantic extension for spaceborne perception systems constrained by fixed model weights. By leveraging prompt-driven grounding, it enables dynamic recognition of new spacecraft components without retraining, significantly reducing operational costs and complexity. The approach aligns with the operational realities of space missions, where bandwidth, safety, and computational resources limit frequent model updates. It paves the way for more flexible, autonomous space inspection systems, advancing both scientific research and practical space operations.
Technical Contribution
The paper introduces a novel application of promptable vision-language models, specifically SAM3, for zero-shot instance segmentation in space environments. It highlights the importance of prompt formulation—combining spatial and geometric cues—to enhance large-structure detection. The methodology demonstrates that model parameters can remain frozen, with detection capabilities extended solely through prompt engineering. The work also validates the deployment feasibility on embedded GPU hardware, emphasizing the practical significance of lightweight, prompt-based semantic extension in resource-constrained settings. This represents a shift from traditional retraining-based paradigms towards prompt-centric adaptability.
Novelty
This research is pioneering in applying structured prompt-driven zero-shot segmentation to space domain imagery, a setting characterized by significant domain shift and scale variation. Unlike prior natural-image-focused open-vocabulary detection, it emphasizes the effectiveness of prompt design in a highly constrained, operational environment. It demonstrates that large structural components can be reliably localized without model fine-tuning, marking a significant step forward in space AI's ability to adapt post-deployment. The integration of spatial and geometric descriptors into prompts for space imagery is a novel contribution that broadens the scope of open-vocabulary grounding.
Limitations
- Detection of small and visually ambiguous components like antennas and thrusters remains unreliable due to limited pixel support and appearance variability. The approach is sensitive to prompt wording, requiring careful crafting and validation of prompts. The current evaluation is limited to a single dataset and model, raising questions about generalization across different spacecraft and environments. High-precision detection of micro-components may necessitate additional modalities or domain-specific training, which are not addressed in this study.
Future Work
Future research will explore automatic prompt optimization and multi-modal data integration (e.g., infrared, LiDAR) to improve micro-component detection. Domain adaptation techniques could enhance robustness across diverse spacecraft. Extending the framework to include active learning or few-shot fine-tuning may further boost accuracy. Additionally, deploying the system in real space missions will validate its operational effectiveness and inform iterative improvements, ultimately enabling fully autonomous spacecraft inspection and maintenance.
AI Executive Summary
Spacecraft inspection and maintenance are critical yet challenging tasks, especially given the constraints of onboard computing resources and the difficulty of updating models post-launch. Traditional approaches rely heavily on supervised training and fixed label sets, which are impractical for in-orbit updates. This study introduces a novel solution: leveraging a pretrained, promptable vision-language model, SAM3, to achieve zero-shot instance segmentation of spacecraft components. By designing structured prompts that encode spatial and geometric information, the model can reliably detect large structures like satellite bodies and solar arrays, even on previously unseen satellites. The experiments demonstrate that, under a strict deployment scenario with frozen weights and single-pass inference, the model attains an average precision of 0.385 at IoU 0.5, with performance strongly dependent on component scale. Larger structures are localized with high accuracy, while smaller parts like antennas and thrusters remain challenging. The key insight is that prompt formulation significantly influences detection performance, with structured prompts outperforming simple category names by up to 82%. This approach operates within the computational limits of embedded GPUs, making it practical for space applications. The findings suggest that prompt-driven semantic extension offers a lightweight, flexible mechanism for post-launch model adaptation, reducing operational costs and enabling more autonomous space systems. Future directions include multi-modal data fusion, prompt optimization, and domain adaptation to improve detection of micro-components, ultimately advancing the goal of fully autonomous spacecraft inspection and maintenance.
Deep Analysis
Background
随着空间任务的复杂化，自主检测成为关键技术。早期方法如Faster R-CNN、Mask R-CNN依赖大量标注数据和离线训练，难以快速适应新目标。近年来，SAM等预训练模型引入开放词汇检测，提升了泛化能力，但在空间环境中仍面临尺度变化、光照变化和域偏移的挑战。空间场景特殊，模型参数难以频繁更新，限制了其灵活性。为应对这一难题，研究者开始探索提示驱动的零样本检测技术，旨在利用预训练模型的通用性，通过设计提示实现新目标的识别。
Core Problem
空间站在轨检测面临模型参数固定、环境复杂、微小目标识别困难等问题。传统微调方案成本高、效率低，难以满足动态任务需求。如何在模型参数不可变的情况下，通过输入提示实现新结构识别，成为核心难题。这不仅关系到检测的实时性，也影响空间站的自主能力。现有方法在微尺度目标检测上表现不足，亟需一种低成本、高效的后发能力扩展方案。
Innovation
本研究的创新在于：1）采用SAM3模型实现空间环境中的零样本检测，参数完全冻结；2）设计结构化提示，结合空间位置和几何特征，显著提升大结构检测效果；3）验证模型在嵌入式GPU上的单次推理能力，确保实际部署可行。不同于传统微调方案，强调提示工程在模型扩展中的作用，为空间AI提供新思路。该方法突破了模型参数不可变的限制，展示了提示驱动的潜力。
Methodology
- �� 采用预训练的SAM3模型，确保模型参数冻结，避免在空间环境中的微调需求。• 设计结构化文本提示，结合空间关系（如‘延伸自卫星’）和几何描述（如‘方形或圆柱’），提升大结构的检测能力。• 通过单次前向推理实现实例分割，避免多轮交互或微调。• 提示内容经过优化，包含空间位置、形状和结构特征，增强语义表达。• 采用[email protected]和[email protected]:0.95指标，评估不同类别的检测性能。• 利用Web Satellite Dataset（WSD）增强空间场景标注，构建测试集。• 对不同提示变体进行对比，验证提示设计的有效性。
Experiments
- �� 使用129张未见过的卫星图像作为测试集，进行零样本检测评估。• 设定固定阈值，无微调或后处理，确保部署一致性。• 以AP指标评估天线、推进器、太阳能阵列、卫星主体等类别的检测效果。• 比较不同提示变体，验证空间和几何描述的影响。• 分析不同尺度目标的检测表现，揭示模型尺度依赖性。• 结果显示大结构检测效果优异，小结构检测仍有提升空间。
Results
- �� SAM3在固定提示下实现0.385 [email protected]，表现出良好的大结构检测能力。• 大型结构如卫星主体（[email protected]=0.639）和太阳能阵列（[email protected]=0.598）检测效果稳定。• 小型附件如天线（[email protected]=0.221）和推进器（[email protected]=0.081）检测困难，反映尺度限制。• 结构化提示比短类别名提升多达82%的AP，验证提示设计的重要性。• 模型在嵌入式GPU上实现单次推理，满足实际部署需求。
Applications
- �� 适用于空间站自主检测，快速识别新结构，减少模型维护成本。• 结合提示工程，实现模型在发射后对新目标的动态扩展。• 未来结合多模态数据，提升微小目标检测能力，推动自主空间监测。• 该技术也适用于其他有限资源环境中的目标识别与监控。
Limitations & Outlook
- �� 小型结构检测性能不足，受限于空间纹理、尺度和视角变化。• 提示设计对语义表达敏感，需细致调优。• 当前仅在单一空间场景和有限类别验证，泛化能力有限。• 未来结合多模态信息和域适应技术，增强鲁棒性。
Plain Language Accessible to non-experts
想象你在厨房做饭，每次只用一句话（提示）告诉厨师（模型）你要做什么菜，比如‘炒鸡蛋’或‘煮面’。厨师不用重新学做菜，只要听到这句话，就能用厨房里的食材和工具帮你完成。空间站里的检测模型就像这个厨师，只要给它一句描述，比如‘大大的方形结构’，它就能帮你找到对应的空间结构，不需要重新训练。不同的是，描述越详细，模型越能准确找到目标。这样，即使模型不能改变参数，只要你换个提示，它就能帮你识别不同的空间结构。这让空间站的检测变得更灵活、更智能，就像用一句话就能告诉机器人你想找什么一样。
ELI14 Explained like you're 14
想象你有个超级厉害的机器人助手，它可以帮你找房间里的东西，比如沙发、电视或灯泡。平时，你得教它每样东西的样子（训练模型），但这很麻烦，也不方便每次都教。现在，有一种神奇的方法，只要你用一句话告诉它，比如“大大的白色沙发在窗边”，它就能马上帮你找到沙发，不用重新教它。这就像你用一句描述让机器人知道你要找的东西，它就能在房间里找到对应的东西。对于空间站来说，这个方法特别有用，因为它不能经常重新“学习”新东西，只能用事先准备好的模型。只要你用不同的提示描述新结构，模型就能找到它们。虽然对一些很小的东西，比如天线或推进器，还不够完美，但对大结构的识别已经很靠谱了。这种用提示让机器人变得更聪明、更灵活的方法，将来可以让空间站更自主、更智能，帮你完成各种复杂任务！
Abstract
Spaceborne inspection systems often deploy perception models prior to launch, after which updating model weights or expanding fixed label sets becomes operationally impractical. While supervised models can be integrated pre-flight, adding new semantic capabilities in orbit requires retraining and re-uploading parameters. We investigate whether prompt-driven vision--language models can enable post-launch semantic expansion, allowing new spacecraft components to be specified via natural-language prompts without modifying onboard weights. We evaluate zero-shot instance segmentation of spacecraft components under a strictly frozen, single-pass inference protocol on a test set of $129$ images of previously unseen satellites. Under fixed global thresholds and no post-processing, SAM3 achieves $0.385$ mAP@$0.5$ and $0.267$ mAP@$0.5{:}0.95$. Performance is strongly scale-dependent: large structural elements like spacecraft bodies ($0.639$ AP@$0.50$) and solar arrays ($0.598$ AP@$0.5$) localize reliably, while relatively small appendages like antennas ($0.221$ AP@$0.5$) and thrusters ($0.081$ AP@$0.5$) remain difficult. Prompt formulation influences performance, with structured prompts incorporating spatial and geometric descriptors yielding up to $82%$ improvement over short category-name prompts. The model operates within the memory and compute envelope of contemporary embedded GPUs, suggesting prompt-driven grounding can provide a practical mechanism for post-launch semantic extension of dominant spacecraft structures while highlighting limitations of zero-shot localization for fine-scale components under orbital domain shift.