SG-AMP: Scene-Graph-Guided Active Perception and Semantics-Aware Motion Planning for Pepper Plants
SG-AMP combines robust depth completion, scene graph reasoning, and semantics-aware active view planning, achieving 55.27% semantic mIoU and 38.67% PQ on pepper data.
Key Findings
Methodology
The SG-AMP framework integrates input-conditioned uncertainty depth completion (using Robust Fast Fill, RFF), persistent panoptic mapping, and plant scene graph reasoning. RFF employs median/MAD filtering to reject inconsistent depth measurements, restoring supported data based on spatial, RGB, and depth consistency. The panoptic map fuses semantic, instance, and depth info, maintaining persistent identities for peppers, peduncles, and stems. Scene graph hypotheses generate missing structure predictions, such as unobserved peduncle attachments, guiding active sensing. View selection maximizes expected information gain while considering class-dependent motion costs, balancing exploration and protection of critical plant parts.
Key Results
- On pepper datasets, the perception network achieved 55.27% semantic mIoU, 38.67% PQ, and 40.62mm depth RMSE. RFF improved depth denoising from 149mm to 59.79mm RMSE under noisy conditions. Input-conditioned uncertainty enhanced NLL from -1.6518 to -1.6925 and AUSE from 0.0102 to 0.0087 on NYUv2, demonstrating better depth and semantic estimation reliability.
- Active view planning based on expected information gain and class-aware motion costs effectively protected sensitive structures like peppers and peduncles while enabling exploration of occluded regions, leading to higher detection accuracy and more complete plant models.
- Experimental results validate the system’s robustness in complex, occluded scenarios, outperforming static mapping and non-informative view selection baselines, with significant improvements in depth accuracy, semantic segmentation, and structural inference.
Significance
This work advances plant perception by integrating deep learning-based depth completion, structural reasoning, and active sensing. It addresses key challenges in dense foliage environments, enabling more accurate and complete plant models, which are critical for precision agriculture tasks like monitoring and harvesting. The approach bridges the gap between static perception and dynamic, structure-aware exploration, promising to enhance autonomous agricultural robots' capabilities and efficiency.
Technical Contribution
The paper introduces a novel combination of input-conditioned uncertainty depth completion with median/MAD filtering, persistent panoptic mapping with multi-resolution refinement, and scene graph-based structural hypotheses. The semantic-aware motion planning considers class-dependent costs, enabling safe and efficient exploration. These innovations collectively improve robustness, accuracy, and interpretability over existing methods like SemSegDepth, PanDepth, and EVPSNet, providing both theoretical and practical advancements.
Novelty
This is the first work to incorporate scene graph reasoning for actively inferring unobserved plant structures, such as peduncle attachments, within a perception pipeline that combines semantic segmentation, depth completion, and active view planning. Unlike prior static or purely geometric approaches, SG-AMP leverages structural hypotheses to guide sensing, significantly enhancing perception in occluded, cluttered environments.
Limitations
- The system’s performance depends on accurate scene graph hypotheses; severe occlusions or highly deformable foliage may lead to incorrect structural inferences. Computational complexity may hinder real-time deployment in large-scale scenarios. Generalization to diverse plant species and environmental conditions remains to be validated.
- Sensitivity to parameter tuning, especially for motion costs and view sampling, can affect robustness. Handling dynamic plants or moving camera platforms requires further development. Future work should focus on optimizing computational efficiency and broadening dataset diversity.
Future Work
Future directions include integrating reinforcement learning for adaptive view planning, expanding multi-modal sensing (e.g., multispectral, LiDAR), and deploying the system in real-field conditions. Enhancing structural inference robustness, reducing computational load, and generalizing across plant varieties will be key to translating this approach into commercial agricultural robotics.
AI Executive Summary
In modern precision agriculture, robotic perception faces significant challenges due to occlusion, complex plant structures, and incomplete data. Traditional RGB-D sensors often fail to capture the full extent of dense foliage, limiting the effectiveness of autonomous monitoring and harvesting. Existing methods like static semantic segmentation or geometric reconstruction cannot fully address these issues, especially in cluttered environments where critical structures such as peduncles and stems are occluded or partially observed.
To overcome these limitations, this study introduces SG-AMP, a comprehensive system that combines deep learning-based depth completion, persistent panoptic mapping, and structural reasoning through scene graphs. The depth completion module employs Robust Fast Fill (RFF), which filters out inconsistent measurements and restores missing data, providing reliable depth estimates even under noisy conditions. The persistent map maintains semantic and instance identities over time, enabling long-term plant monitoring.
A key innovation is the use of scene graph hypotheses to infer unobserved plant structures, such as peduncle attachments, by leveraging structural priors and spatial relationships. These hypotheses guide active sensing by directing the robot’s camera to regions with high expected information gain, while class-dependent motion costs protect sensitive plant parts. Experimental results demonstrate that SG-AMP achieves 55.27% semantic mIoU and 38.67% PQ on pepper datasets, with depth RMSE reduced to 40.62mm, outperforming baseline methods.
The system’s ability to infer missing structures and plan safe, informative viewpoints significantly enhances plant perception accuracy. This has profound implications for agricultural robotics, enabling more precise monitoring, targeted harvesting, and structural analysis, ultimately contributing to increased productivity and automation in farming. Future work aims to optimize computational efficiency, incorporate multi-modal sensors, and validate performance in real-world field deployments, paving the way for scalable, intelligent agricultural solutions.
Deep Analysis
Background
植物感知在智能农业中扮演核心角色，早期方法多依赖RGB图像或激光雷达，难以应对遮挡和复杂结构。深度学习推动了深度完成和语义分割的发展，如SemSegDepth、PanDepth等，为植物结构理解奠定基础。然而，静态重建无法应对植物的动态变化和遮挡，导致信息丢失。场景图推理逐渐成为研究热点，用于结构推断和目标关系建模，但在植物感知中的应用仍属新颖。整体而言，感知系统需要融合多模态信息、推理能力和主动视角策略，以实现高精度、鲁棒的植物结构理解。
Core Problem
现有植物感知系统在遮挡、稀疏数据和复杂结构环境中表现不足。传统深度完成算法对噪声敏感，难以提供可靠深度信息。静态映射无法应对植物生长变化和遮挡，导致信息丢失。主动视角规划缺乏对未观察结构的推理，限制了感知范围。如何结合深度补全、结构推理和主动感知，提升整体性能，是亟待解决的核心问题。
Innovation
本研究创新点包括：1) 结合Median/MAD滤波的鲁棒深度补全（RFF），增强噪声环境下的深度恢复能力；2) 引入场景图推理，将未观察到的植物结构（如穗柄）转化为目标，指导主动感知；3) 设计基于语义类别的运动成本，有效保护关键结构，避免损伤。系统实现了多层次的持续映射与结构推断，显著优于传统几何或静态方法，为植物感知提供了全新解决方案。
Methodology
- �� 输入RGB-D数据，经RFF算法进行快速深度补全，结合Median/MAD滤波抑制噪声，输出支持不确定性估计的深度图。• 将深度、语义和实例信息融合到持续的多分辨率全景语义映射中，保持植物各部分的持久性。• 构建植物场景图，识别已观察和未观察的结构，通过假设缺失连接（如穗柄）生成推断目标。• 采用信息增益为基础的视点采样，结合语义类别的运动成本，筛选最优视角，平衡探索与保护。• 通过运动规划确保路径安全，利用语义信息区分保护结构和可穿越叶片，优化感知效率。
Experiments
采用辣椒植物数据集进行验证，比较不同深度补全策略和感知模型性能。评估指标包括语义mIoU、PQ、深度RMSE，以及在NYUv2上的NLL和AUSE。通过消融实验验证RFF的鲁棒性和输入条件不确定性估计的贡献。主动视点策略在遮挡和复杂场景中的效果也被详细分析，确保系统在实际应用中的可靠性和效率。
Results
系统在辣椒数据集上达到了55.27%的语义mIoU和38.67%的PQ，深度RMSE为40.62mm。RFF深度补全在噪声环境下显著优于传统方法，RMSE由149mm降至59.79mm。输入条件不确定性提升了深度估计的可靠性，NYUv2上的NLL从-1.6518改善到-1.6925，AUSE从0.0102降至0.0087。主动视点选择有效保护关键结构，提升了未观察区域的感知质量，验证了系统的实用性。
Applications
该技术适用于农业机器人中的植物监测、果实采摘和结构分析，尤其在遮挡严重或结构复杂的环境中表现优越。系统可集成于自主导航平台，实现高效、精准的植物结构感知，为智能农业提供技术支撑。未来还可结合多模态传感器，拓展到更广泛的农业场景中。
Limitations & Outlook
模型在极端遮挡和复杂叶片结构下仍存在推断误差，计算复杂度较高影响实时性，泛化能力需在多样化环境中验证。未来需优化算法效率，增强模型鲁棒性，并扩展多样性数据集以提升泛用性。
Plain Language Accessible to non-experts
想象你在一个繁忙的厨房里，厨师需要找到所有的食材，但厨房里有很多遮挡，比如盖着布的食材或被其他东西挡住。传统的方法就像用手去摸，可能摸不到所有的食材，尤其是藏在角落的。这个系统就像一个聪明的助手，它不仅能用手摸，还能用眼睛观察，结合推理告诉厨师哪些食材可能藏在某个角落，还会主动去那里确认。它会用一种特殊的“眼镜”看清楚深度信息，判断哪些地方需要多看几眼，哪些地方可以跳过。这样，厨师就能更快找到所有食材，避免误拿或遗漏。这个助手还能记住厨房里每个食材的位置，随着时间推移不断更新信息，确保每次都能找到最新的食材位置。整体来说，这个系统就像一个聪明的厨房助手，能主动寻找隐藏的食材，帮厨师节省时间，提高效率。
ELI14 Explained like you're 14
想象你在玩一个超级复杂的拼图游戏，但拼图块都藏在盒子里，你只能看到一部分。普通的拼图方法就是一边拼一边猜，但有时候你会拼错或者遗漏。这个系统就像有个聪明的朋友，他不仅能帮你看清楚每块拼图，还能猜出藏在盒子底下的拼图块在哪里，然后告诉你下一步该拼到哪里去。他会用特殊的“眼睛”看深度，知道哪些地方有拼图，哪些地方空着，还会根据拼图的形状和颜色，判断下一块应该放在哪里。更厉害的是，他会主动去那些还没拼好的地方检查，确保每个拼图都放得稳稳当当。这样，你就能更快拼完整个图，而且拼得更漂亮。这就像一个聪明的助手，帮你解决难题，让拼图变得简单又有趣！
Abstract
We present SG-AMP, integrating robust depth completion with input-conditioned uncertainty, persistent panoptic mapping, plant scene-graph reasoning, and semantics-aware active view-motion planning. Beyond inspecting uncertain observed regions, the scene graph explicitly hypothesizes unobserved pepper--peduncle attachments and directs close-range sensing toward them. Candidate views are selected according to expected information gain, while class-dependent motion costs distinguish protected peppers, peduncles, and stems from conditionally traversable foliage. On pepper data, the perception network achieves $55.27\%$ semantic mIoU, $38.67\%$ PQ, and $40.62\,\mathrm{mm}$ depth RMSE, while input-conditioned uncertainty improves NYUv2 NLL from $-1.6518$ to $-1.6925$ and AUSE from $0.0102$ to $0.0087$.